Chinese opera sound source separation method based on domain adaptation and time domain information guidance

Abstract

The invention discloses a Chinese opera sound source separation method based on domain adaptation and time domain information guidance, and relates to the technical field of audio signal processing and artificial intelligence. The method comprises the following steps: constructing a field adaptation training data set MUSDB18 fused with Chinese traditional musical instruments, and superposing national musical instrument audio frequency bridged domain differences to the MUSDB18 data set; processing the mixed audio by adopting a pre-trained VAD model, generating a time domain confidence sequence, and applying Gaussian attenuation to obtain a time domain weight sequence; taking Mel-Band RoFormer as an infrastructure, constructing an MSS model with VAD weighted loss, and guiding the model to focus a human voice area through the weighted loss; and applying the model to real opera audio separation. The problems that traditional opera national musical instrument modeling is insufficient, and attention is excessively put into an area without effective human voice are solved, the SDR on a traditional opera test set reaches 10.81 dB, and the SIR on the traditional opera test set reaches 28.77 dB.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of audio signal processing and artificial intelligence technology, especially to the field of music source separation (MSS), and in particular to a drama music source separation method based on domain adaptation and time domain information guidance. BACKGROUND

[0002] Artificial intelligence technology is increasingly widely used in artistic creation, and singing voice synthesis (SVS) technology has made significant progress, which can automatically generate high-quality singing voice audio according to input music scores and lyrics. However, Chinese traditional opera, as a non-material cultural heritage with deep cultural connotations, its singing contains rich melody changes (such as glissando, long tone, vibrato), and integrates dialect pronunciation, character roles (sheng, dan, jing, chou) and specific vocal systems (xipi, erhuang), which puts higher requirements on singing voice synthesis technology.

[0003] To promote the development of opera singing synthesis, researchers have conducted a number of explorations. For example, Zhou et al. proposed the OperaSinger system for Peking Opera in "A High-Quality Melody-Aware Peking Opera Synthesizer Using Data Augmentation" published in the 2023 IEEE International Conference on Multimedia and Expo (ICME2023, pp. 1092-1097). The system introduces a melody-aware position-variable convolution module based on the FastSpeech2 architecture, and combines melody tempo variation and phoneme recombination to improve the naturalness of synthesized voices. Bai et al. proposed a local attention mechanism and a phoneme-level adaptive weighting loss function to address the local inconsistency problem of the Transformer model in "Improving Chinese Pop Song and Hokkien Gezi Opera Singing Voice Synthesis by Enhancing Local Modeling" published in the Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. The method improved the performance of Chinese pop song and Hokkien Gezi Opera singing voice synthesis. Zheng et al. proposed the FT-GAN acoustic model and constructed the GOAT dataset in "FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis" published in the Proc. AAAI Conf. Artif. Intell. (Vol. 38, No. 17, March 2024). The model improved the quality and generalization ability of opera singing synthesis through fine-grained tune modeling and a dedicated pitch encoder. However, the above progress still faces a fundamental bottleneck: the recording and fine labeling of high-quality unaccompanied opera a cappella audio, including phonemes, pitch, duration, roles, and plate types, are time-consuming and labor-intensive, resulting in a scarcity of publicly available training data.

[0004] The most representative work currently comes from Gong Rong et al. in "Creating an ACappella Singing Audio Dataset for Automatic Jingju Singing Evaluation Research" published in the 4th International Digital Libraries for Musicology workshop (DLfM 2017). They constructed a dataset of 120 pieces of a cappella singing of Jingju, containing 1265 melody phrases, and annotated key metadata such as character type, vocal style (such as Xipi, Erhuang), and board type, providing valuable resources for tasks such as pitch modeling and pronunciation similarity calculation. However, even so, the data size is still far from sufficient to support a high-expression, multi-style opera singing synthesis system.

[0005] The Internet has a large number of opera audio and video resources with accompaniment. If clean vocals are extracted through source separation (SS) technology, the data scarcity problem can be alleviated. Source separation aims to restore independent sound sources from mixed audio. Although existing mainstream models (such as Spleeter, Demucs, Mel-RoFormer, etc.) have advanced performance in general tasks, they are designed for Western popular music, and the accompaniment of the dataset (such as MUSDB18) is mainly composed of Western instruments such as guitar, drum, and bass. Chinese traditional opera uses national instruments such as Jinghu, Yueqin, and gong, which have significant differences in spectral characteristics and time-frequency structure from Western instruments, resulting in poor separation performance of existing models on opera audio, especially for voice extraction.

[0006] Chinese patent CN118737184A discloses an opera voice extraction method and system based on accompaniment separation and voice separation. It uses a conditional variational autoencoder (CVAE) to let the model learn the distribution difference between "opera voice" and "accompaniment instruments" (especially the easily confused instruments such as Erhu) in the hidden space. However, it can only separate single instruments and cannot be applied to real opera performance scenes with multi-instrument accompaniment, which has certain limitations.

[0007] Voice activity detection (VAD) is a binary classification task to determine whether there is human voice in a segment of audio signal (voiced segment), and its core role is to distinguish "human voice" from "non-human voice" (such as silence or background noise) in speech processing, so as to improve the robustness of the system and avoid invalid calculation on the unvoiced segment. It is mainly used in speech recognition and speech translation tasks, and in the broad sense of song and opera performance, there are also segments with only instruments and no human voice. For the model, the segment with both instrument sound and human voice is a more difficult area to separate than the segment with only instrument sound, and there is no research on applying VAD technology to the source separation task to optimize the attention allocation of the model to the human voice area.

[0008] In summary, the existing technology mainly has two problems: one is that the existing MSS model lacks training data containing national musical instruments, and the modeling of opera accompaniment is insufficient, and the separation effect is poor; the second is that the model over-invests attention in the area without effective human voice, reducing the purity of human voice separation. Therefore, there is an urgent need for a source separation method that adapts to the characteristics of the opera field and guides the time domain information to improve the quality of opera human voice separation. SUMMARY

[0009] The purpose of the present application is to solve the problems of poor adaptability of existing music source separation technology on opera audio and poor human voice separation effect, and to provide a source separation method based on field adaptation and time domain information guidance that combines data level field adaptation and time domain VAD information guided loss weighting, to significantly improve the separation quality of opera human voice and the generalization ability of the model.

[0010] To achieve the above-mentioned purpose of the application, the present application provides the following technical solutions:

[0011] The source separation method based on field adaptation and time domain information guidance for opera, comprising the following steps:

[0012] Step 1: Construct a field adaptation training data set that integrates Chinese traditional musical instruments: obtain pure instrument audio segments commonly used in Chinese traditional opera from public data sets, cut them, and add random gain to the mixed audio of the standard popular music source separation data set to generate new mixed audio, and keep the original human voice track as the target label;

[0013] Step 2: Generate a time domain weight sequence: use a pre-trained voice activity detection model to process the new mixed audio and output a time domain confidence sequence; for each continuous segment containing human voice, apply a Gaussian decay weight to the center of its time domain confidence sequence and perform normalization to obtain a time domain weight sequence;

[0014] Step 3: Construct and train the source separation model with VAD weighted loss: take the frequency domain music source separation (MSS) model as the backbone, input the mixed audio generated in step 1, and output the target voice estimation; weight the loss function with the aforementioned time domain weight sequence, define the VAD information weighted loss function, and train the model with the loss function as the optimization goal;

[0015] Step 4: Voice separation and testing: apply the trained source separation model to real opera audio, perform voice separation, and test the indicators on the test set to evaluate the separation quality.

[0016] In step 1, the specific steps of constructing the domain adaptation training data set that fuses Chinese traditional instruments can be:

[0017] 1.1 Obtain Chinese traditional opera commonly used pure instrument audio segments from public data sets, including erhu, pipa, sanxian, dizi, suona, qin, zhongruan, liuqin, guzheng, yangqin, and sheng;

[0018] 1.2 Use audio processing tools (such as ffmpeg) to cut the pure instrument audio into non-overlapping segments of 10-20s;

[0019] 1.3 Randomly select mixed audio from the standard popular music source separation data set MUSDB18;

[0020] 1.4 Use audio processing tools (such as ffmpeg or Python audio processing library) to superimpose the instrument segments in step 1.2 with a random gain of -6dB to +3dB on the mixed audio of MUSDB18 to generate new mixed audio;

[0021] 1.5 Ensure that the inserted instrument segments in the training set, validation set, and test set do not overlap to avoid data leakage;

[0022] 1.6 Keep the original voice track in MUSDB18 as the target label to form a domain adaptation data set, denoted as MUSDB18_mix.

[0023] In step 2, the specific steps of training the voice activity detection (VAD) model and generating the time domain confidence sequence can be:

[0024] 2.1 Use a pre-trained VAD model to process each mixed audio generated in step 1.4;

[0025] 2.2 Input the audio into the VAD model to detect the segments with voice activity, and output the confidence value of each segment with voice activity to form a time domain confidence sequence v(t) ∈ [0, 1] equal in length to the audio;

[0026] 2.3 For each continuous segment containing human voice, a Gaussian decay weight is applied with the center of its time-domain confidence sequence as the center of the Gaussian kernel:

[0027]

[0028] wherein, is the time-domain confidence sequence, is the current time point, is the center time of the segment, σ is a hyperparameter controlling the decay range, and exp is the exponential function, is the final confidence value after Gaussian decay weight processing; preferably, the value of the hyperparameter σ is 1 / 6 of the segment length;

[0029] 2.4 The final confidence value after Gaussian decay weight processing is normalized to the [0, 1] interval as the time-domain weight sequence for subsequent loss weighting.

[0030] In step 3, the specific steps of constructing and training the MSS model with VAD weighted loss can be:

[0031] 3.1 The frequency domain source separation (MSS) model adopts a Mel-Band RoFormer model, which is used as the basic architecture and includes a frequency band segmentation module, a rotary position embedding Transformer module, and a mask generation module. The frequency band segmentation module is used to divide the input frequency spectrum into multiple sub-bands. The rotary position embedding Transformer module is used to model hierarchical attention for the time and sub-band axes. The mask generation module is used to output complex ratio masks to separate the target human voice.

[0032] 3.2 The input generated in step 1.4 is mixed audio, and the output is target human voice estimation.

[0033] 3.3 Define the VAD information weighted loss function (VAD weighted loss):

[0034]

[0035] wherein, is the model output human voice, is the real human voice, is the weight, is the spectrum of the real human audio in the corresponding window value under, is the spectrum of the model output human audio in the corresponding window value under, is the weight extended to the weight matrix of the corresponding spectrum size; S is the total number of windows, the value range of​ ; represents a norm operation;

[0036] 3.4 Model training: using the MUSDB18_mix dataset constructed in step 1.6 as training data, using the VAD information weighted loss function defined in step 3.3 as the optimization objective, using the Adam or AdamW optimizer to train the MSS model until the separation index (such as SDR) on the validation set converges.

[0037] Further, in step 3.1, the execution flow of the rotation position embedding Transformer module includes:

[0038] applying the Transformer processing to the input tensor along the time axis;

[0039] applying the Transformer processing to the output tensor along the sub-band axis;

[0040] In each Transformer layer, the query and key are positionally encoded using the rotation position encoding.

[0041] In step 4, the application of the real opera audio voice separation, the specific steps of the index test on the test set can be:

[0042] 4.1 Convert the real opera audio with accompaniment into a single-channel audio file with a sampling rate of 44.1kHz, input the MSS model trained in step 3 with VAD weighted loss, and output the separated opera voice audio;

[0043] 4.2 Construct an opera test set GeZiXi-mix containing mixed audio and corresponding artificially annotated clean voice audio, input the test set mixed audio into the model, and calculate the separation index signal distortion ratio (SDR) and signal interference ratio (SIR).

[0044] Compared with the prior art, the application has the following outstanding advantages and technical effects:

[0045] 1. In view of the problem of insufficient modeling of opera accompaniment and poor separation effect of the existing MSS model due to the lack of relevant training data containing national musical instruments, the application does not need to re-label opera data, but only needs to inject Chinese traditional national musical instrument audio into a general dataset (MUSDB18) to construct a field-adapted dataset (MUSDB18_mix), thereby reducing the data acquisition cost and quickly improving the adaptability of the model to opera accompaniment; from the data level, the domain difference between western musical instruments and opera national musical instruments is bridged, and the modeling capability of the model to opera accompaniment is improved.

[0046] 2. In order to solve the problem that the model pays attention to the area without effective human voice and reduces the purity of human voice separation, the application generates a time domain confidence sequence by using a VAD model, guides the model to focus on the active area of human voice by using a weighted loss function, and combines Gaussian attenuation processing to alleviate the influence of boundary misjudgment of the VAD model and reduce the invalid attention of the model to the silent segment.

[0047] 3. The application innovatively applies the VAD technology to the MSS task, weights the loss function by using the VAD confidence, introduces Gaussian attenuation to alleviate the influence of boundary misjudgment, focuses the model on the active area of human voice, guides the attention distribution of the model, and optimizes the separation effect of the human voice area.

[0048] 4. Experiments show that on the MUSDB18 standard test set, the SDR of the method (Mel-RoFormer-VAD) reaches 11.56 dB, which is better than the mainstream models HDemucs (8.04 dB), BSRNN (10.01 dB) and the like; on the constructed opera test set GeZiXi-mix, the SDR of the method reaches 10.81 dB, which is more than 4.4 dB than the baseline Mel-RoFormer (6.32 dB), the SIR reaches 28.77 dB, and is significantly better than the existing methods. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 It is the overall architecture diagram of the model of the application.

[0050] Figure 2 It is a schematic diagram of a rotation position embedding Transformer module.

[0051] Figure 3 It is a schematic diagram of a single rotation position embedding Transformer module. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical scheme and advantages of the application more clear and understandable, the following embodiments will further illustrate the application with reference to the drawings. It should be understood that the specific embodiments described herein are only used to explain the application and do not limit the application.

[0053] The specific implementation method of the embodiment of the application is as follows:

[0054] Step 1: Construct a domain adaptation training data set that integrates Chinese traditional musical instruments;

[0055] 1.1 From Xia Gong et al. published in Proceedings of the 4 thThe dataset ChMusic downloaded from the International Conference on Big Data Technologies (ICBDT'21, 184-189 pages) contains audio clips of commonly used Chinese traditional musical instruments, including Erhu, Pipa, Sanxian, Dizi, Suona, Zhuchun, Zhongruan, Liusheng, Guzheng, Yangqin, and Sheng. The details of the audio duration are shown in Table 1.

[0056] Table 1

[0057]

[0058] Each music clip is saved in.wav file format. The file naming follows the "a.b.wav" format, where a represents the instrument number (range 1-11), and b represents the corresponding music clip number of the instrument (range 1-5). These audio files are recorded in two channels, with a sampling rate of 44100 Hz, and a duration between 25s and 280s.

[0059] 1.2 Use the ffmpeg tool to cut the pure instrument audio into non-overlapping segments of 10-20s, and the specific cutting process is as follows:

[0060] 1.2.1 Set the path and environment;

[0061] 1.2.2 Specify the list of audio files to be processed;

[0062] 1.2.3 Process each audio file;

[0063] 1.2.3.1 Use ffprobe (part of FFmpeg tool) to read the total duration of the audio file (unit: s).

[0064] 1.2.3.2 Calculate the number of segments, and set the target segment duration as L segment (For example, L segment ∈[10s,20s]). According to the total duration L total and the segment duration L segment , calculate the number of segments N segment =floor(L total / L segment ). The longer the total duration of the instrument, the larger the number of segments N segment . If the total duration cannot be divided by L segment , the last segment is the remaining part (may be less than L segmentFor example, as shown in Table 1, the total duration of the Erhu is 9 minutes and 31 seconds, so the segment duration of the Erhu is set to 20 seconds, and finally 29 segments of Erhu small segments can be cut out. The total duration of the SanXian is only 4 minutes and 13 seconds, so the segment duration of the SanXian is set to 10 seconds, and finally 26 segments of SanXian small segments can be cut out.

[0065] 1.2.3.3 Cut the audio segment by segment, use the-ss (start time) and-to (end time) parameters of ffmpeg for accurate cutting. Use stream copy (-c copy) mode: do not re-encode the audio, directly copy the original data, so the segmentation speed is fast and there is no loss of sound quality. Save the file as a.b.part_c.wav, a, b corresponds to the naming method of the ChMusic dataset mentioned above, and c represents the cth segment of a.b.wav.

[0066] 1.3 Separate the randomly selected mixture audio (mixture.wav) in the dataset from the standard popular music source; in this embodiment, the MUSDB18 dataset is used, which contains 150 pieces of mixture audio, each 3-6 minutes, with a sampling rate of 44.1 kHz. Randomly select 120 pieces from the MUSDB18 dataset as the source data of the training set, 15 pieces as the source data of the validation set, and 15 pieces as the source data of the test set.

[0067] 1.4 Use soundfile, pydub, and other python audio processing libraries to superimpose the instrument segments in step 1.2 onto the mixture audio of the MUSDB18 dataset to generate new mixture audio, the specific method is as follows:

[0068] Load all instrument segments: read all small segment wav files in step 1.2, save their audio data, sampling rate, and file name.

[0069] Balanced instrument segment assignment: Count the total number of songs in MUSDB18 (train+test+valid), calculate how many times each erhu segment should be used to ensure balanced usage (e.g.: assume there are 100 songs, 7 erhu segments → some segments are used 14 times, some segments are used 15 times). Generate a shuffled assignment list (instrument_assignment) to ensure that instrument segments are randomly and evenly distributed in the dataset. Process each song individually, select a MUSDB18HQ music file, select an instrument segment from the assignment list, resample the instrument segment audio to match the current song's sample rate (use pydub for resampling), then overlay the instrument segment audio at 70% volume into mixture.wav and other.wav. Finally, in most digital audio processing systems (including Python's soundfile, librosa, pydub libraries), audio signals in floating-point format (float32 or float64) are usually normalized to the range [-1.0, +1.0]: -1.0 represents the negative maximum amplitude (trough), +1.0 represents the positive maximum amplitude (peak), and values outside this range cannot be correctly represented when playing or saving. Therefore, clip the mixed audio to -1.0 to 1.0 to prevent audio overflow distortion.

[0070] 1.5 Ensure that the inserted instrument segments in the training set, validation set, and test set do not overlap, and divide the instrument audio segments according to the proportion of the training set, validation set, and test set: 80% of the instrument audio segments will be mixed into the training set, 10% of the instrument audio segments will be mixed into the test set, and the remaining 10% will be mixed into the validation set to avoid data leakage.

[0071] 1.6 Preserve the original vocal track in the original MUSDB18 as the target label, and pair it with the new mixed audio to form a new domain adaptation dataset, denoted as MUSDB18_mix.

[0072] Step 2: Train the voice activity detection (VAD) model and generate the time-domain confidence sequence;

[0073] 2.1 Use a pre-trained VAD model (such as a model based on WavLM or ECAPA-TDNN) to process each mixed audio generated in step 1.4; the VAD model is configured to use a sliding window with a frame length of 200ms and a step length of 100ms, outputting a frame-level voice activity probability sequence with a time resolution of 100ms.

[0074] 2.2 Input the new mixed audio in the MUSDB18_mix training set into the VAD model, detect the segments with human voice activity, and output the confidence value of each segment with voice activity, forming a time-domain confidence sequence with the same length as the audio wherein, ; t is the current time point.

[0075] 2.3 For each continuous segment containing human voice, take the center of its time-domain confidence sequence as the center of the Gaussian kernel, and apply a Gaussian decay weight:

[0076]

[0077] wherein, is the time-domain confidence sequence, t is the current time point, is the center time of the segment, σ is a hyperparameter controlling the decay range (preferably 1 / 6 of the segment length), exp is the exponential function, is the final confidence value after Gaussian decay weight processing;

[0078] 2.4 The final confidence value after Gaussian decay weight processing is normalized to the [0, 1] interval using Min-Max normalization as the time-domain weight sequence for subsequent loss weighting, and the Min-Max normalization is defined as .

[0079] wherein, is a value in a set of data, is the minimum value in the entire set of data, is the maximum value in the entire set of data, is the normalized value, ranging between [0, 1].

[0080] Step 3: Construct and train the MSS model with VAD weighted loss;

[0081] 3.1 Select a frequency domain MSS backbone model (Mel-Band RoFormer) as the basic architecture;

[0082] 3.2 The input is the mixed audio generated in step 1.4, and the output is the target voice estimation;

[0083] 3.3 Define the VAD information weighted loss function (VAD weighted loss):

[0084]

[0085] wherein, ŷ is the model output voice, y is the true voice, is the frequency spectrum of the true voice audio under the corresponding window value S, a spectrum of the model output human voice audio at a corresponding window value S, a weight a weight matrix extended to a corresponding spectrum size.

[0086] 3.4 Model training: the MUSDB18_mix dataset constructed in step 1.6 is used as training data, the VAD information weighted loss function defined in step 3.3 is used as the optimization objective, and the MSS model is trained using the Adam optimizer until the separation index (such as SDR) on the validation set converges. The training parameters are as follows: the frequency domain dimension is 1024, the time frame number is 256, the frame shift of short-time Fourier transform is 441, the window size of short-time Fourier transform is 2048, the number of channels is 2, the sampling rate is 44100, the hidden layer dimension of the model is 384, the number of layers of the rotation position embedding Transformer module is 6, the number of sub-bands is 60, the batch size is 4, the learning rate is 0.00001, the maximum number of training rounds is 500, the number of training steps per round is 1000, and the training device is 4 Nvidia L40 graphics cards.

[0087] 3.5 Model interpretation:

[0088] As shown in Figure 1 , let denote the audio waveform of the input mixed audio, where, and denote the number of channels and the number of audio samples, respectively, denote the real number domain; the frequency domain feature learning module of the model uses a complex spectrum as input, and converts into a time-frequency representation , where, and correspond to the number of frames and the number of frequency points, respectively; denote the number of channels; denote the complex number domain; the short-time Fourier transform formula is defined as:

[0089]

[0090] where, is the current time point of the waveform audio, x(t) is the waveform audio value at time t, is the frame shift size, is the number of frequency points. is a window function, is a complex spectrum at the th frame and frequency point. is the imaginary unit; is the base of the natural logarithm; A kernel function for the Fourier transform, used to extract specific frequencies in a signal.

[0091] Segmenting the frequency band based on the Mel frequency can help purify the learning representation on different frequency bands, thus enhancing the robustness to cross-frequency ambiguity. The method followed divides the complex spectrum into unequal-length, non-overlapping sub-bands, and applies an independent multi-layer perceptron (MLP) to each sub-band. The output of each sub-band is denoted as , where is the th sub-band, is the number of frequency points in the th sub-band, is the number of channels, and is the number of time frames. All sub-bands constitute the complete complex spectrum , and satisfy

[0092] . Each multi-layer perceptron (MLP) contains a root mean square normalization (RMSNorm) layer followed by a linear layer. The root mean square normalization (RMSNorm) layer regularizes the sum of inputs to a neuron in a layer according to the root mean square. The root mean square normalization (RMSNorm) is an efficient alternative to layer normalization (LayerNorm) normalization. The n-th linear layer contains a learnable matrix of shape and a learnable bias of shape , where is the number of channels, is the number of frequency points in the th sub-band, and is the number of features (i.e., latent dimension) output by the linear layer. The transformed output of the n-th sub-band is denoted as , which has a shape of . Here, represents the number of frames, represents the latent dimension. Stacking all along the sub-band axis results in a stacked tensor of shape , which is used as the input to the rotary position embedding Transformer module (RoPE Transformer), where

[0093] represents the total number of sub-bands. Figure 2 As shown in , the hierarchical processing flow of the rotary position embedding Transformer module, specifically as follows: the input is a three-dimensional tensor with dimensions (latent dimension), (total number of sub-bands).(frame number); first, along the 3rd dimension (time frame), the Transformer is processed to model the time sequence characteristics within each sub-band; the processed tensor dimension remains the latent dimension , the total number of sub-bands , the frame number unchanged, then along the 2nd dimension (sub-band), the Transformer is processed to globally model the spectral characteristics across sub-bands; finally, the processed three-dimensional tensor is output, through the hierarchical and interleaved Transformer operation, to enhance the model's ability to capture complex relationships in the time-frequency domain.

[0094] Assuming that there are blocks in the rotation position embedding Transformer module, the output of the block of the sub-band is denoted as , where represents the number of layers of the rotation position embedding Transformer module, n represents the output of the nth sub-band, n = 1,..., N, represents the real number field, represents the frame number, represents the latent dimension, and N is the total number of sub-bands; the frequency band segmentation output as the input of the first Transformer block. Unlike traditional Transformer encoders that only apply self-attention mechanisms to one-dimensional sequences (such as time), the Transformer structure of the present application is hierarchical: in each Transformer block, interleaved Transformer layers are applied to the time frame and the frequency axis in turn. The former is called time-Transformer, which is used to model the (local) time sequence within the sub-band; the latter is called subband-Transformer, which is used to process the (global) spectral sequence across frequency bands to ensure that information can be exchanged between frequency bands. The processing flow of the Transformer block is shown in Algorithm 1.

[0095] Algorithm 1 is an explanation of the running logic of the rotation position embedding Transformer module.

[0096]

[0097] For the time-Transformer, all mini-batches (batch size ) and subband are stacked; for subband-Transformer, all mini-batches and frames are stacked. All Transformer layers adopt the same architecture.

[0098] Figure 3 A Transformer layer is depicted, which contains an attention module (with rotated positional encoding) and a feed-forward module. In the attention module, the input signal is first processed by RMSNorm; the weight matrix 、 and generate the query ), key ), value ), respectively, where and need to be processed by rotated positional embedding; sequentially pass through the attention layer + random dropout, fully connected layer + random dropout; finally, through the residual connection (⊕) and the input signal processed by RMSNorm are added, the attention module is completed. In the feed-forward module, the output of the attention module is received, first processed by RMSNorm; then enter the fully connected layer + GELU activation function + random dropout unit; then pass through the attention layer + random dropout; finally, output through the residual connection (⊕), complete the operation of the entire Transformer module.

[0099] Specifically, the input is rearranged according to the processing requirements of the time axis or the subband axis (see Algorithm 1), and then RMSNorm is applied. Next, the query (Query), key (Key) and value (Value) projection layers are used to generate the corresponding 、 and : , , . Among them, is the input tensor of the layer, the shapes of the query , key and value are (when processing the time axis) or (when processing the subband axis), and the shapes of the learnable weight matrices 、 and are . Among them, represents the number of frames, represents the latent dimension, represents the number of subbands. Subsequently, is divided into multiple attention heads.

[0100] In terms of position encoding, a Rotary Position Embedding (RoPE) is adopted, which is applied as follows: , where Rot(·) is the RoPE encoder shared across the whole Transformer block. The time-Transformer and the subband-Transformer each have an independent RoPE encoder, which applies a rotation matrix according to the position of each embedding in its corresponding sequence.

[0101] Using the processed queries, keys and values, the attention operation is applied as follows:

[0102]

[0103] where N is the number of attention heads, Z is a parameter related to the feature dimension; is the query after applying RoPE; is the key after applying the Rotary Position Embedding (RoPE); is the value; is the transpose of the query multiplied by the key; is a scaling factor, is a normalization function, is a random dropout function. The attention matrix calculation is the most computationally intensive core part. To accelerate the calculation, the Flash Attention technique is used. After the attention module, a fully connected layer with random dropout (Dropout) is used. A residual connection is applied between the input and output of the attention module. The feed-forward module consists of a root mean square normalization (RMSNorm) layer, a fully connected layer with a Gaussian error linear unit (GeLU) activation function, and a random dropout (Dropout) layer. Subsequently, a fully connected layer with random dropout (Dropout) is applied. Similarly, a residual connection is applied between the input and output of the feed-forward module.

[0104] The output of the Rotary Position Embedding Transformer module (RoPE Transformer) is denoted as

[0105] where L denotes the number of layers of the RoPE Transformer, denotes the output of the i-th subband, ​​​​​N represents the total number of sub-bands. Similar to the frequency band segmentation module, for each sub-band... N independent multilayer perceptron (MLP) layers are applied. Each MLP layer consists of a root mean square normalization (RMSNorm) layer, a fully connected layer followed by a Tanh activation function, and a fully connected layer followed by a gated linear unit (GLU). The nth MLP layer outputs a subband mask. Its shape is ,in It is the number of vocal tracts. It's the frame rate. It is the first The number of frequency points in each sub-band. The output sub-band mask of the nth MLP layer. It includes both real and imaginary part masks. All outputs are concatenated along the frequency axis to obtain the Complex Ideal Ratio Mask (cIRM). .in, Represents the field of complex numbers. It is the number of vocal tracts. It's the frame rate. It refers to the number of frequency points.

[0106] Separated complex spectrum By using complex scaling masks With the complex spectrum of the input Multiplying them together gives: ;in, The spectrum of the separated complex numbers. Represents the complex field, where C is the number of channels, T is the number of frames, and F is the number of frequency points; X is the Complex Ideal Ratio Mask (cIRM); X is the complex spectrum; This indicates an element-wise multiplication operation.

[0107] Finally, a short-time inverse Fourier transform (iSTFT) is performed on the complex spectrum to obtain the model-predicted separated human voice audio. The short-time inverse Fourier transform is defined as follows:

[0108]

[0109] in, For the reconstructed waveform audio at time points The sampled value at that location. To predict the complex spectrum The values ​​at time frame m and frequency k. is a complex exponential term, representing the phase of a sine wave with frequency k at time point t. This represents the number of frequency points. is a window function at a time point , is a frame shift size.

[0110] In the audio source separation task, SDR (Signal-to-Distortion Ratio) and SIR (Signal-to-Interference Ratio) are the core indicators for evaluating the separation quality. SDR measures the ratio of the target signal to all distortion components (including interference, noise and artificial artifacts), reflecting the overall reconstruction quality; SIR only measures the ratio of the target signal to the remaining interference sources (i.e. other speakers or background sound sources), reflecting the separation ability of the model.

[0111] The calculation formula of the signal-to-noise ratio SDR and the interference ratio SIR is as follows:

[0112]

[0113]

[0114] In the formula, s represents the clean target source signal (ground truth); ŝ represents the estimated separation signal of the model; represents the part of the estimated signal that aligns with the target (obtained by projection or optimal filtering); represents the part of the estimated signal that comes from other interference sources; ||·||² represents the L² norm (i.e. signal energy); represents the logarithm operation with base 10.

[0115] The following verifies the effect of the VAD information guiding method on the MUSDB18 dataset:

[0116] The experimental results on the MUSDB18 standard test set show that the Mel-RoFormer-VAD method proposed by the present application exhibits significant performance advantages in the speech separation task. The index verification result of the effectiveness of VAD information guiding on the MUSDB18 dataset is shown in Table 2.

[0117] Table 2

[0118]

[0119] ​As shown in Table 2, the method of the present application achieves 11.56dB in SDR index, which is better than the existing mainstream models, including HDemucs (8.04dB), BSRNN (10.01dB), BS-RoFormer (10.78dB) and the basic version of Mel-RoFormer (11.21dB). This improvement verifies the effectiveness of introducing VAD (voice activity detection) mechanism combined with Mel spectrum features, not only enhancing the modeling ability of the model to the time-frequency structure of speech, but also improving the robustness to non-speech interference, so as to realize higher quality speech separation in complex mixed sound scenes.

[0120] The following verifies the effect of the VAD information guiding method and the domain adaptation method on the opera test set GeZiXi-mix:

[0121] The existing mainstream sound source separation models are mostly designed for western popular music, and their performance in the opera scene is generally low. In this experiment, representative models in the field and the general field are selected, and the experimental results on the opera test set GeZiXi-mix (containing mixed audio of various Chinese traditional instruments) constructed by the present application show that the Mel-RoFormer-VAD method proposed by the present application exhibits strong generalization and adaptation ability. The effectiveness index verification result of the domain adaptation method on the opera test set GeZiXi-mix is shown in Table 3.

[0122] Table 3

[0123]

[0124] As shown in Table 3, the basic version of Mel-RoFormer has achieved better performance than HDemucs, BSRNN and BS-RoFormer on this data set (SDR: 6.32dB, SIR: 16.54dB). After fine-tuning on the mixed instrument data set, the model performance is greatly improved to SDR 10.58dB, SIR 28.65dB, proving the effectiveness of domain adaptation training on non-western music scenes. After further introducing the VAD mechanism, the performance of the Mel-RoFormer-VAD model is further enhanced, finally reaching SDR 10.81dB, SIR 28.77dB, which is the highest level among all comparison methods. This result fully verifies that the VAD mechanism effectively optimizes attention allocation by guiding the model to focus on the active area of human voice, so as to extract purer human voice under complex accompaniment. Fine-tuning on target domain data + VAD guiding time series modeling is the key path to improve the quality of complex national instrument separation. The Mel-RoFormer-VAD (fine-tuned on mixed data set) model of the present application has the highest performance level among all comparison methods.

[0125] The above embodiments are only the preferred embodiments of the present application, and cannot be considered as limiting the scope of the present application. Any equivalent changes and improvements made according to the scope of the present application should still belong to the patent coverage of the present application.

Claims

1. A method for opera sound source separation based on domain adaptation and time domain information guidance, characterized in that The method comprises the following steps: Step 1: constructing a field adaptation training data set of fusing Chinese traditional musical instruments: obtaining audio segments of commonly used pure musical instruments of Chinese traditional operas from a public data set, cutting the audio segments, superimposing random gain to mixed audio of a standard popular music source separation data set to generate new mixed audio, and retaining original vocal tracks as target labels; Step 2: generating a time domain weight sequence: processing the new mixed audio by using a pre-trained voice activity detection model to output a time domain confidence sequence; for each continuous segment containing a voice, a Gaussian attenuation weight is applied to the center of the time domain confidence sequence of the segment, and normalization processing is performed to obtain a time domain weight sequence; Step 3: constructing and training a source separation model with a VAD weighted loss: taking a frequency domain music source separation model as a backbone, inputting the mixed audio generated in step 1, and outputting target voice estimation; weighting the loss function by using the aforementioned time domain weight sequence, defining a VAD information weighted loss function, and training the model by using the loss function as an optimization target; Step 4: voice separation and testing: applying the trained source separation model to real opera audio for voice separation, and testing indicators on a test set to evaluate separation quality.

2. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 1, the commonly used pure musical instruments of Chinese traditional operas include at least one of an erhu, a pipa, a sanxian, a dizi, a suona, a qin, a zhongruan, a liuqin, a guzheng, a yangqin, and a sheng; the cut segments are non-overlapping segments with a length of 10-20s; and the random gain range is-6dB to +3dB.

3. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 2, the calculation formula of the Gaussian attenuation weight is: wherein, is a time-domain confidence sequence, is a waveform audio current time point, is a segment center time, σ is a hyperparameter that controls the decay range, and exp is an exponential function, is the final confidence value after the Gaussian decay weight processing.

4. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 3, characterized in that The value of the superparameter σ controlling the attenuation range is 1 / 6 of the segment length.

5. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 3, the function of the VAD information weighted loss is: wherein, is the model output of the human voice, is the weight, is the spectrum of the real human voice audio in the corresponding window value , is the spectrum of the model output human voice audio in the corresponding window value , is the weight is extended to a weight matrix of the corresponding spectrum size; S is the total number of windows, the value range of ; denotes the norm operation.

6. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 3, the frequency domain music source separation model adopts a Mel-Band RoFormer model, which includes a frequency band segmentation module, a rotating position embedding Transformer module, and a mask generation module; the frequency band segmentation module is used to divide the input frequency spectrum into multiple sub-bands; the rotating position embedding Transformer module is used to model hierarchical attention for the time and sub-band axes; The mask generation module is used to output a complex proportion mask to separate the target voice.

7. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 6, characterized in that The execution process of the rotating position embedding Transformer module includes: Applying a Transformer to process the input tensor along the time axis; Applying a Transformer to process the output tensor along the sub-band axis; In each Transformer layer, using rotating position encoding to encode the query and key positions.

8. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, wherein In step 4, the indicators for evaluating the separation quality include a signal distortion ratio and a signal interference ratio.

Images