A Method for Separating Opera Sound Sources Based on Domain Adaptation and Temporal Information

Abstract

A method for separating opera audio sources based on domain adaptation and temporal information guidance is proposed, involving the fields of audio signal processing and artificial intelligence. The method includes: constructing a domain adaptation training dataset MUSDB18 that incorporates traditional Chinese musical instruments; bridging domain differences by overlaying ethnic musical instrument audio onto the MUSDB18 dataset; processing the mixed audio using a pre-trained VAD model to generate a temporal confidence sequence and applying Gaussian attenuation to obtain a temporal weight sequence; constructing an MSS model with VAD weighted loss based on the Mel-Band RoFormer architecture, guiding the model to focus on the vocal region through weighted loss; and applying the model to the separation of real opera audio. This method addresses the problems of insufficient modeling of traditional opera musical instruments and excessive attention being paid to regions lacking effective vocal representation. On the opera test set, the SDR reaches 10.81 dB and the SIR reaches 28.77 dB.

Description

Technical Field

[0001] This invention relates to the fields of audio signal processing and artificial intelligence, particularly to the field of music source separation (MSS), specifically to a method for separating opera sound sources based on domain adaptation and temporal information guidance. Background Technology

[0002] Artificial intelligence technology is increasingly being applied in the field of artistic creation. Among them, singing voice synthesis (SVS) technology has made significant progress, which can automatically generate high-quality singing audio based on input musical scores and lyrics. However, as an intangible cultural heritage with profound cultural connotations, traditional Chinese opera's singing style contains rich melodic variations (such as glissando, sustained notes, and vibrato), and integrates dialect pronunciation, role types (male, female, painted-face, and clown), and specific vocal systems (Xipi and Erhuang), which places higher demands on singing voice synthesis technology.

[0003] To advance the development of Peking Opera voice synthesis, researchers have undertaken numerous explorations. For example, Zhou Xun et al.'s paper, "A High-Quality Melody-Aware Peking Opera Synthesizer Using DataAugmentation," published in *2023 IEEE International Conference on Multimedia and Expo* (ICME2023, pp. 1092-1097), proposes an OperaSinger system for Peking Opera. This system introduces a melody-aware, positionally variable convolutional module based on the FastSpeech2 architecture, combining melody tempo shifting and phoneme recombination data augmentation strategies to enhance the naturalness of the synthesis. Bai Peng et al.'s paper, "Improving Chinese Pop Song and Hokkien Gezi Opera Singing VoiceSynthesis by Enhancing Local..." published in *Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing* (2023), presents "Improving Chinese Pop Song and Hokkien Gezi Opera Singing VoiceSynthesis by Enhancing Local..." The paper "Modeling" ("Improving the Synthesis of Chinese Popular Songs and Hokkien Opera Vocals through Enhanced Local Modeling") addresses the local inconsistency problem of the Transformer model by proposing a local attention mechanism for neighboring phonemes and a phoneme-level adaptive weighted loss function, achieving improvements in Chinese popular songs and Hokkien opera tasks. The paper "FT-GAN: Fine-Grained Tune Modeling for Chinese Opera Synthesis" by Zheng Meizhen et al., published in Proc. AAAI Conf. Artif. Intell. (Vol. 38, No. 17, March 2024), constructs the Gokkien Opera Phonetic Alignment Dataset (GOAT) and proposes the FT-GAN acoustic model. Through fine-grained tune modeling and a dedicated pitch encoder, it improves the quality and generalization ability of opera vocal synthesis. However, the aforementioned progress still faces a fundamental bottleneck: the recording and detailed annotation of high-quality a cappella opera audio (including phonemes, pitch, duration, roles, rhythmic patterns, etc.) are time-consuming and labor-intensive, resulting in a severe shortage of publicly available training data.

[0004] The most representative work to date comes from "Creating an Acapella Singing Audio Dataset for Automatic Jingju Singing Evaluation Research" by Gong Rong et al., published in the 4th International Digital Libraries for Musicology workshop (DLfM 2017). They constructed an acapella Jingju dataset containing 120 arias and 1265 melodic phrases for automatic Jingju singing evaluation research, and labeled key metadata such as role type, vocal style (e.g., Xipi, Erhuang), and rhythmic pattern, providing valuable resources for tasks such as pitch modeling and pronunciation similarity calculation. However, even so, the data scale is still far from sufficient to support a high-expressive, multi-style opera singing synthesis system.

[0005] The vast amount of online audio and video resources of traditional Chinese opera with accompaniment can alleviate the data scarcity problem by extracting clean vocals using Source Separation (SS) technology. Source separation aims to reconstruct individual sound sources from mixed audio. While existing mainstream models (such as Spleeter, Demucs, and Mel-RoFormer) are advanced in general tasks, they are all designed for Western popular music. Datasets (such as MUSDB18) primarily feature accompaniment with Western instruments like guitar, drums, and bass. Traditional Chinese opera uses folk instruments such as the jinghu, yueqin, and gongs and drums, whose spectral characteristics and time-frequency structures differ significantly from Western instruments. This results in poor separation performance of existing models on opera audio, especially in vocal extraction.

[0006] Chinese patent CN118737184A discloses a method and system for extracting opera vocals based on accompaniment separation and vocal separation. It uses a conditional variational autoencoder (CVAE) to allow the model to learn the distribution differences between "opera vocals" and "accompaniment instruments" (especially easily confused instruments such as the erhu) in the latent space. However, it can only separate single instruments and cannot be applied to real opera performance scenarios with multiple instruments, which has certain limitations.

[0007] Speech Activity Detection (VAD) is a binary classification task used to determine whether a human voice (vocal segment) exists in an audio signal. Its core function is to distinguish between "human voice" and "non-human voice" (such as silence or background noise) in speech processing, thereby improving system robustness and avoiding invalid computation for silent segments. It is mainly used in speech recognition and speech translation tasks. However, in broader contexts such as songs and opera performances, there are also segments containing only musical instruments without human voices. For models, segments containing both musical instruments and human voices are more difficult to separate than segments containing only musical instruments. Currently, no research has applied VAD technology to sound source separation tasks to optimize the model's attention allocation to human voice regions.

[0008] In summary, existing technologies suffer from two main problems: first, the current MSS model lacks training data containing traditional Chinese musical instruments, resulting in insufficient modeling of opera accompaniment and poor separation performance; second, the model overemphasizes areas without effective vocals, reducing the purity of vocal separation. Therefore, there is an urgent need for a sound source separation method that adapts to the characteristics of opera and incorporates temporal information guidance to improve the quality of opera vocal separation. Summary of the Invention

[0009] The purpose of this invention is to address the problems of poor adaptability and unsatisfactory vocal separation effect of existing music source separation technologies in opera audio. It provides an opera audio source separation method based on domain adaptation and temporal information guidance, which combines domain adaptation at the data level with loss weighting guided by temporal VAD information, so as to significantly improve the separation quality and model generalization ability of opera vocals.

[0010] To achieve the above-mentioned objectives, the present invention provides the following technical solution:

[0011] A method for separating opera sound sources based on domain adaptation and temporal information guidance includes the following steps:

[0012] Step 1: Construct a domain-adaptive training dataset that incorporates traditional Chinese musical instruments: Obtain audio clips of commonly used pure musical instruments from public datasets, segment them, and then superimpose them with random gain onto the mixed audio of a standard pop music source separation dataset to generate new mixed audio, while retaining the original vocal tracks as target labels.

[0013] Step 2: Generate temporal weight sequence: The new mixed audio is processed using a pre-trained speech activity detection model to output a temporal confidence sequence; for each continuous segment containing human voice, a Gaussian decay weight is applied to the center of its temporal confidence sequence and normalized to obtain a temporal weight sequence.

[0014] Step 3: Construct and train a sound source separation model with VAD weighted loss: The frequency domain music sound source separation (MSS) model is used as the backbone, the input is the mixed audio generated in step 1, and the output is the target human voice estimate; the loss function is weighted by the aforementioned temporal weight sequence, the VAD information weighted loss function is defined, and the model is trained with this loss function as the optimization target;

[0015] Step 4: Voice Separation and Testing: Apply the trained audio source separation model to real opera audio to perform voice separation, and conduct index testing on the test set to evaluate the separation quality.

[0016] In step 1, the specific steps for constructing the domain-adaptive training dataset that integrates traditional Chinese musical instruments can be as follows:

[0017] 1.1 Audio clips of commonly used pure musical instruments in traditional Chinese opera were obtained from public datasets. The pure musical instruments include erhu, pipa, sanxian, dizi, suona, zhuiqin, zhongruan, liuqin, guzheng, yangqin, and sheng.

[0018] 1.2 Use audio processing tools (such as ffmpeg) to segment the pure instrumental audio into non-overlapping segments of 10-20 seconds;

[0019] 1.3 Randomly select mixed audio from the standard pop music source separation dataset MUSDB18;

[0020] 1.4 Use audio processing tools (such as ffmpeg or Python audio processing libraries) to superimpose the instrument clips from step 1.2 onto the mixed audio of MUSDB18 with a random gain of -6dB to +3dB to generate a new mixed audio;

[0021] 1.5 Ensure that the instrument fragments inserted into the training set, validation set, and test set do not overlap to avoid data leakage;

[0022] 1.6 The vocal tracks in the original MUSDB18 are retained as target labels to form a domain-adapted dataset, denoted as MUSDB18_mix.

[0023] In step 2, the specific steps for training the Voice Activity Detection (VAD) model and generating the time-domain confidence sequence can be as follows:

[0024] 2.1 A pre-trained VAD model is used to process each mixed audio generated in step 1.4;

[0025] 2.2 Input the audio into the VAD model, detect segments in the audio where human voice activity exists, and output the confidence value of each segment where speech activity exists, forming a time-domain confidence sequence v(t)∈[0,1] of the same length as the audio;

[0026] 2.3 For each continuous segment containing human voice, apply a Gaussian decay weight with its temporal confidence sequence center as the Gaussian kernel center:

[0027]

[0028] in, For time-domain confidence sequences, At the current time point, Let σ be the center time of the segment, σ be a hyperparameter controlling the decay range, and exp be an exponential function. This is the final confidence value after Gaussian attenuation weighting; preferably, the hyperparameter σ is 1 / 6 of the segment length.

[0029] 2.4 The final confidence value after Gaussian attenuation weighting Normalized to the [0,1] interval, it serves as the temporal weight sequence for subsequent loss weighting.

[0030] In step 3, the specific steps for constructing and training the MSS model with VAD weighted loss can be as follows:

[0031] 3.1 The Frequency Domain Sound Source Separation (MSS) model adopts the Mel-Band RoFormer model as its basic architecture, which includes a frequency band segmentation module, a rotation position embedding Transformer module, and a mask generation module. The frequency band segmentation module is used to divide the input spectrum into multiple sub-frequency bands; the rotation position embedding Transformer module is used to perform hierarchical attention modeling on the time and sub-frequency band axes; and the mask generation module is used to output a complex proportional mask to separate the target human voice.

[0032] 3.2 Input the mixed audio generated in step 1.4, and output the target human voice estimate;

[0033] 3.3 Define the VAD information-weighted loss function:

[0034]

[0035] in, Output human voice for the model. For real human voices As weight, For real human voice audio in the corresponding window value The spectrum below, The human voice audio output by the model is within the corresponding window value. The spectrum below, For weight The weight matrix is ​​extended to the corresponding spectrum size; S is the total number of windows. The range of values ​​for is ; Represents norm operations;

[0036] 3.4 Model Training: Using the MUSDB18_mix dataset constructed in step 1.6 as the training data, and the VAD information weighted loss function defined in step 3.3 as the optimization objective, the MSS model is trained using the Adam or AdamW optimizer until the separation metric (such as SDR) on the validation set converges.

[0037] Furthermore, in step 3.1, the execution flow of embedding the rotation position into the Transformer module includes:

[0038] The input tensor is processed along the time axis using a Transformer.

[0039] Apply the Transformer to the output tensor along the sub-band axis;

[0040] In each Transformer layer, the query and key are positionally encoded using rotational position encoding.

[0041] In step 4, the specific steps for performing metric testing on the test set for the vocal separation applied to real opera audio can be as follows:

[0042] 4.1 Convert the real opera audio and video with accompaniment into a mono audio file with a sampling rate of 44.1kHz, input it into the MSS model with VAD weighted loss trained in step 3, and output the separated opera vocal audio.

[0043] 4.2 Construct a test set GeZiXi-mix containing mixed audio and corresponding manually annotated clean human voice audio. Input the mixed audio of the test set into the model and calculate the separation indexes Signal Distortion Ratio (SDR) and Signal Interference Ratio (SIR).

[0044] Compared with the prior art, the outstanding advantages and technical effects of the present invention are as follows:

[0045] 1. To address the problem of insufficient modeling and poor separation performance of traditional Chinese opera accompaniment in existing MSS models due to the lack of relevant training data containing traditional Chinese musical instruments, this invention eliminates the need for re-annotating opera data. Instead, it constructs a domain-adaptive dataset (MUSDB18_mix) by injecting audio of traditional Chinese musical instruments into a general dataset (MUSDB18), thereby reducing data acquisition costs and rapidly improving the model's adaptability to opera accompaniment. This approach bridges the domain differences between Western instruments and traditional Chinese opera instruments at the data level, enhancing the model's ability to model opera accompaniment.

[0046] 2. To address the issue of the model focusing on areas without effective human voices, thus reducing the purity of human voice separation, this invention utilizes the VAD model to generate a time-domain confidence sequence. A weighted loss function guides the model to focus on active human voice areas, and Gaussian attenuation processing is combined to mitigate the impact of boundary misjudgment by the VAD model, reducing the model's ineffective attention to silent segments.

[0047] 3. This invention innovatively applies VAD technology to the MSS task, uses VAD confidence to weight the loss function, introduces Gaussian decay to alleviate the impact of boundary misjudgment, enables the model to focus on the active area of ​​human voice, guides the model's attention allocation, and optimizes the human voice region separation effect.

[0048] 4. Experiments show that on the MUSDB18 standard test set, the SDR of our method (Mel-RoFormer-VAD) reaches 11.56dB, which is better than mainstream models such as HDemucs (8.04dB) and BSRNN (10.01dB). On the constructed opera test set GeZiXi-mix, the SDR of our method reaches 10.81dB, which is more than 4.4dB higher than the baseline Mel-RoFormer (6.32dB), and the SIR reaches 28.77dB, which is significantly better than existing methods. Attached Figure Description

[0049] Figure 1 This is a diagram of the overall architecture of the model of this invention.

[0050] Figure 2 A schematic diagram of embedding a Transformer module for rotational positioning.

[0051] Figure 3 A schematic diagram of embedding a Transformer module for a single rotational position. Detailed Implementation

[0052] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0053] The specific implementation methods of this invention are as follows:

[0054] Step 1: Construct a domain-adaptive training dataset that incorporates traditional Chinese musical instruments;

[0055] 1.1 From Xia Gong et al.'s article published in "Proceedings of the 4 thDownload the ChMusic dataset, a public dataset from the International Conference on Big Data Technologies (ICBDT'21, pp. 184-189), featuring audio clips of commonly used instrumental instruments in traditional Chinese opera. The instruments include erhu, pipa, sanxian, dizi, suona, zhuiqin, zhongruan, liuqin, guzheng, yangqin, and sheng. See Table 1 for audio duration details.

[0056] Table 1

[0057]

[0058] Each music segment is saved as a .wav file. The file names follow the format "abwav", where 'a' represents the instrument number (ranging from 1 to 11) and 'b' represents the music segment number corresponding to that instrument (ranging from 1 to 5). These audio files are recorded in stereo, with a sampling rate of 44100Hz and a duration between 25s and 280s.

[0059] 1.2 Use ffmpeg to segment the pure instrumental audio into non-overlapping segments of 10-20 seconds. The specific segmentation process is as follows:

[0060] 1.2.1 Setting up the path and environment;

[0061] 1.2.2 Specify the list of audio files to be processed;

[0062] 1.2.3 Process each audio file;

[0063] 1.2.3.1 Use ffprobe (part of the FFmpeg tool) to read the total duration (in seconds) of the audio file.

[0064] 1.2.3.2 Calculate the number of segments, with the target segment duration preset to L. segment (e.g. L) segment (∈[10s, 20s]). Based on the total audio duration L total and segment duration L segment Calculate the number of segments N segment =floor(L total / L segment The longer the total duration of the instrument, the more segments N should be divided. segment The larger the value. If the total duration cannot be increased by L... segment If divisible by L, then the last segment is the remainder (which may be less than L). segment(seconds). For example, as shown in Table 1, the total duration of the erhu is 9 minutes and 31 seconds. If the duration of each erhu segment is set to 20 seconds, then 29 erhu segments can be divided into them. The total duration of the sanxian is only 4 minutes and 13 seconds. If the duration of each sanxian segment is set to 10 seconds, then 26 sanxian segments can be divided into them.

[0065] 1.2.3.3 Audio is segmented segment by segment using ffmpeg's -ss (start time) and -to (end time) parameters for precise segmentation. Stream copy (-c copy) mode is employed: the audio is not re-encoded, but the original data is directly copied, resulting in fast segmentation and no loss of audio quality. The file is saved as abpart_c.wav, where a and b correspond to the naming convention for audio files in the ChMusic dataset mentioned above, and c indicates that this segment is the c-th segment of abwav.

[0066] 1.3 Randomly select mixed audio (mixture.wav) from the standard pop music source separation dataset; In this embodiment, the MUSDB18 dataset is used, which contains 150 mixed audio segments, each segment is 3 to 6 minutes long, and the sampling rate is 44.1 kHz. 120 segments are randomly selected from the MUSDB18 dataset as training set source data, 15 segments as validation set source data, and 15 segments as test set source data.

[0067] 1.4 Use Python audio processing libraries such as soundfile and pydub to overlay the instrument fragments from step 1.2 onto the mixed audio of the MUSDB18 dataset to generate new mixed audio. The specific method is as follows;

[0068] Load all instrument fragments: Read all the small WAV files from step 1.2 and save their audio data, sample rate, and filename.

[0069] Balanced distribution of instrument fragments: Count the total number of songs in MUSDB18 (train + test + valid), and calculate the number of times each erhu fragment should be used to ensure balanced usage (e.g., assuming 100 songs and 7 erhu fragments → some fragments used 14 times, some fragments used 15 times). Generate a shuffled assignment list (instrument_assignment) to ensure that instrument fragments are randomly and uniformly distributed in the dataset. Process each song individually: Select an MUSDB18HQ music file, choose an instrument fragment from the assignment list, adjust the sampling rate of the instrument fragment audio to match the current song (using pydub for resampling), and then superimpose the instrument fragment audio at 70% volume into mixture.wav and other.wav. Finally, in most digital audio processing systems (including Python libraries such as soundfile, librosa, and pydub), floating-point format (float32 or float64) audio signals are typically normalized to the range [-1.0, +1.0]: -1.0 represents the maximum negative amplitude (trough), and +1.0 represents the maximum positive amplitude (peak). Values ​​outside this range cannot be correctly represented during playback or saving. Therefore, the mixed audio is clipped from -1.0 to 1.0 to prevent audio overflow distortion.

[0070] 1.5 Ensure that the instrumental clips inserted into the training set, validation set, and test set do not overlap. Divide the instrumental audio clips according to the ratio of the training set, validation set, and test set. 80% of the instrumental audio clips will be mixed into the training set, 10% of the instrumental audio clips will be mixed into the test set, and the other 10% will be mixed into the validation set to avoid data leakage.

[0071] 1.6 The original vocal tracks in the original MUSDB18 are retained as target labels and paired with the new mixed audio to form a new domain-adapted dataset, denoted as MUSDB18_mix.

[0072] Step 2: Train the Voice Activity Detection (VAD) model and generate a temporal confidence sequence;

[0073] 2.1 A pre-trained VAD model (such as a model based on WavLM or ECAPA-TDNN) is used to process each mixed audio generated in step 1.4; the VAD model is configured to use a sliding window with a frame length of 200ms and a step size of 100ms, and output a frame-level speech activity probability sequence with a time resolution of 100ms.

[0074] 2.2 Input the new mixed audio from the MUSDB18_mix training set into the VAD model to detect audio segments with human voice activity, and output the confidence value of speech activity for each segment, forming a time-domain confidence sequence of the same length as the audio. ,in, t represents the current time point.

[0075] 2.3 For each continuous segment containing human voice, apply a Gaussian decay weight with its temporal confidence sequence center as the Gaussian kernel center:

[0076]

[0077] in, Here is the time-domain confidence sequence, where t is the current time point. Let σ be the center time of the segment, σ be a hyperparameter controlling the decay range (preferred value is 1 / 6 of the segment length), and exp be an exponential function. This is the final confidence value after Gaussian weighting.

[0078] 2.4 The final confidence value after Gaussian attenuation weighting The weights are normalized to the [0,1] interval using Min-Max, and this sequence is used as the temporal weights for subsequent loss weighting. Min-Max normalization is defined as follows: .

[0079] in, It is a value in a set of data. It is the minimum value in the entire set of data. It is the maximum value in the entire set of data. It is the normalized value, ranging from [0,1].

[0080] Step 3: Construct and train the MSS model with VAD-weighted loss;

[0081] 3.1 The frequency domain MSS backbone model (Mel-Band RoFormer) is selected as the basic architecture;

[0082] 3.2 The input is the mixed audio generated in step 1.4, and the output is the target human voice estimate;

[0083] 3.3 Define the VAD information-weighted loss function:

[0084]

[0085] Where ŷ represents the model's output human voice, and y represents the actual human voice. This represents the spectrum of a real human voice audio at the corresponding window value S. The spectrum of the human voice audio output by the model at the corresponding window value S. For weight Extend to a weight matrix of the corresponding spectrum size.

[0086] 3.4 Model Training: Using the MUSDB18_mix dataset constructed in step 1.6 as the training data, and the VAD information-weighted loss function defined in step 3.3 as the optimization objective, the MSS model was trained using the Adam optimizer until the separation metric (such as SDR) on the validation set converged. The training parameters were as follows: frequency domain dimension of 1024, number of time frames of 256, frame shift of short-time Fourier transform of 441, window size of short-time Fourier transform of 2048, number of channels of 2, sampling rate of 44100, hidden layer dimension of the model of 384, number of Transformer modules with rotational position embedding of 6, number of sub-bands of 60, batch size of 4, learning rate of 0.00001, maximum number of training epochs of 500, number of training steps per epoch of 1000, and training device of 4 Nvidia L40 graphics cards.

[0087] 3.5 Model Explanation:

[0088] like Figure 1 As shown, let This represents the audio waveform of the input mixed audio, where, and These represent the number of channels and the number of audio samples, respectively. Representing the real number domain; the model's frequency domain feature learning module uses the complex spectrum as input and performs a short-time Fourier transform to... Convert to time-frequency representation ,in, and These correspond to the number of frames and the number of frequency points, respectively. Indicates the number of vocal tracts; Representing the complex field; the short-time Fourier transform formula is defined as:

[0089]

[0090] in, t represents the current time point of the waveform audio, and x(t) represents the waveform audio value at time t. It is the frame shift size. It refers to the number of frequency points. It is a window function. It is the complex spectrum In the Frame, frequency The value that can be taken on. It is the imaginary unit; It is the base of the natural logarithm; It is the kernel function of the Fourier transform, used to extract specific frequencies from a signal.

[0091] Segmenting the frequency band based on Mel frequency can help the model refine the learned representations across different frequency bands, thereby enhancing its robustness to cross-band ambiguity. The method used here involves dividing the complex spectrum... Divided along the frequency axis into The system comprises three unequal-length, non-overlapping sub-bands, each for which an independent multilayer perceptron (MLP) is applied. The output of each sub-band is denoted as... ,in, Indicates the first Number of frequency points in each sub-band Indicates the number of vocal tracts. Indicates the time frame number; all sub-bands Constitutes a complete complex spectrum And satisfy .

[0092] Each multilayer perceptron (MLP) contains a root mean square normalization (RMSNorm) layer followed by a linear layer. The RMSNorm layer regularizes the sum of inputs to neurons in a layer based on the root mean square. RMSNorm is an efficient alternative to layer normalization. The nth linear layer contains a layer with shape... A learnable matrix and a shape of The learnable bias, in which Indicates the number of vocal tracts. Indicates the first Number of frequency points in each sub-band This represents the number of features (i.e., latent dimension) output by the linear layer. The transform output of the nth sub-band is denoted as... Its shape is .in, Represents the number of frames. Represents potential dimensions. (This refers to all...) Stacking along the sub-band axis yields a shape of Stacked tensors , as the input to the rotation position embedded Transformer module (RoPE Transformer), where This represents the total number of sub-bands.

[0093] like Figure 2 As shown, the hierarchical processing flow of the rotation position embedding Transformer module is as follows: The input is a three-dimensional tensor with dimensions of... (Latent Dimensions) (Total number of sub-bands) (Frame count); First, Transformer processing is performed along the 3rd dimension (time frame) to model the temporal features within each sub-band; after processing, the tensor dimension maintains the latent dimension. Total number of sub-bands Frame rate Without changing the previous step, a Transformer process is then performed along the second dimension (sub-band) to globally model the spectral features across sub-bands. Finally, the processed three-dimensional tensor is output. Through hierarchical and interleaved Transformer operations, the model's ability to capture complex relationships in the time and frequency domains is enhanced.

[0094] Assuming the rotation position is embedded in the Transformer module, there are a total of The block, the first The first sub-band The output of each block is denoted as ,in, The rotation position indicates the layer number of the Transformer module embedded in the module, where n represents the output of the nth sub-band, n=1,...,N. Represents the real number field. Represents the number of frames. Representing the potential dimension, N is the total number of sub-bands; band segmentation output. This serves as the input to the first Transformer block. Unlike traditional Transformer encoders that apply self-attention mechanisms only to one-dimensional sequences (such as time), the Transformer structure of this invention is hierarchical: in each Transformer block, staggered Transformer layers are applied sequentially to the time frame and the frequency axis. The former, called the time-domain Transformer, is used to model the (local) timing within a sub-band; the latter, called the subband-transformer, is used to process cross-band (global) spectral sequences to ensure information exchange between bands. The processing flow of the Transformer block is shown in Algorithm 1.

[0095] Algorithm 1 is an explanation of the execution logic of the Transformer module embedded in the rotation position.

[0096]

[0097] For time-domain Transformers, all mini-batches (batch size is...) The main layers and sub-bands are stacked; for the sub-band Transformer, all batches and frames are stacked. All Transformer layers use the same architecture.

[0098] Figure 3 A Transformer layer is described, which contains an attention module (containing rotational position encoding) and a feedforward module. In the attention module, the input signal is first processed by root mean square normalization; then by the weight matrix... , and Generate queries respectively ( ),key( ),value( ),in and The process requires rotational position embedding; it then passes through an attention layer + random deactivation, a fully connected layer + random deactivation, and finally is added to the root mean square normalized input signal via a residual connection (⊕) to complete the attention module's processing. In the feedforward module, the output of the attention module is received, first normalized by the root mean square; then it enters a fully connected layer + GELU activation function + random deactivation unit; then passes through another attention layer + random deactivation; finally, it outputs through a residual connection (⊕) to complete the entire Transformer module's computation.

[0099] Specifically, the input is rearranged according to the processing requirements of the time axis or sub-band axis (see Algorithm 1), and then RMSNorm is applied to it. Next, query, key, and value projection layers are used to generate the corresponding... , and : , , .in, For the first The input tensor of the layer, query ,key Sum All shapes are (When processing the timeline) or (When dealing with sub-band axes), the learnable weight matrix , and All shapes are .in, Represents the number of frames. Representing potential dimensions, This represents the number of sub-bands. Subsequently... It is divided into multiple attention heads.

[0100] For position encoding, Rotary Position Embedding (RoPE) is used, and its application is as follows: , Rot(·) is the RoPE encoder shared throughout the entire Transformer block. The time-Transformer and subband-Transformer each have their own independent RoPE encoders, which are determined by the query ( ) and key ( For each position embedded in its corresponding sequence, apply a rotation matrix.

[0101] Using the processed query, key, and value, apply attention operations as follows:

[0102]

[0103] in, Z represents the number of attention heads, and Z is a parameter related to the feature dimension. For queries after applying RoPE; For the key after applying Rotational Position Embedding (RoPE); Value; The query is multiplied by the transpose of the key; Scaling factor For normalization function, The attention matrix is ​​a randomized deactivation function. The computational complexity of the attention matrix is ​​the largest component. To accelerate computation, a fast attention technique is used. A fully connected layer with randomized dropout is used after the attention module. Residual connections are applied between the input and output of the attention module.

[0104] The feedforward module consists of a root mean square normalized (RMSNorm) layer, a fully connected layer with Gaussian error linear unit (GeLU) activation function, and a dropout layer. A further fully connected layer with dropout is then applied. Similarly, a residual connection is applied between the input and output of the feedforward module.

[0105] The output of the RoPE Transformer module, which embeds the rotation position, is denoted as... ,in, Indicates the number of layers in the RoPETransformer. It indicates that it is the first Output of each sub-band N represents the total number of sub-bands. Similar to the frequency band segmentation module, for each sub-band... N independent multilayer perceptron (MLP) layers are applied. Each MLP layer consists of a root mean square normalization (RMSNorm) layer, a fully connected layer followed by a Tanh activation function, and a fully connected layer followed by a gated linear unit (GLU). The nth MLP layer outputs a subband mask. Its shape is ,in It is the number of vocal tracts. It's the frame rate. It is the first The number of frequency points in each sub-band. The output sub-band mask of the nth MLP layer. It includes both real and imaginary part masks. All outputs are concatenated along the frequency axis to obtain the Complex Ideal Ratio Mask (cIRM). .in, Represents the field of complex numbers. It is the number of vocal tracts. It's the frame rate. It refers to the number of frequency points.

[0106] Separated complex spectrum By using complex scaling masks With the complex spectrum of the input Multiplying them together gives: ;in, The spectrum of the separated complex numbers. Represents the complex field, where C is the number of channels, T is the number of frames, and F is the number of frequency points; X is the Complex Ideal Ratio Mask (cIRM); X is the complex spectrum; This indicates an element-wise multiplication operation.

[0107] Finally, a short-time inverse Fourier transform (iSTFT) is performed on the complex spectrum to obtain the model-predicted separated human voice audio. The short-time inverse Fourier transform is defined as follows:

[0108]

[0109] in, For the reconstructed waveform audio at time points The sampled value at that location. To predict the complex spectrum The values ​​at time frame m and frequency k. is a complex exponential term, representing the phase of a sine wave with frequency k at time point t. This represents the number of frequency points. For window functions At the point of time The value at that location, This is the frame shift size.

[0110] In audio source separation tasks, SDR (Signal-to-Noise Ratio) and SIR (Signal-to-Interference Ratio) are core metrics for evaluating separation quality. SDR measures the ratio of the target signal to all distortion components (including interference, noise, and artifacts), reflecting the overall reconstruction quality; SIR measures only the ratio of the target signal to other interference sources (i.e., other speakers or background sound sources), reflecting the model's separation capability.

[0111] The formulas for calculating the signal-to-noise ratio (SDR) and interference ratio (SIR) are as follows:

[0112]

[0113]

[0114] In the formula, s represents the clean target source signal (ground truth); ŝ represents the separation signal estimated by the model; This represents the portion of the estimated signal that is aligned with the target (obtained through projection or optimal filtering). This represents the portion of the estimated signal that originates from other interference sources; ||·||² represents the L² norm (i.e., signal energy). This represents a logarithmic operation with base 10.

[0115] The following verifies the effectiveness of the VAD information guidance method on the MUSDB18 dataset:

[0116] Experimental results on the MUSDB18 standard test set demonstrate that the proposed Mel-RoFormer-VAD method exhibits significant performance advantages in speech separation tasks. Table 2 shows the validation results of the effectiveness of VAD information guidance on the MUSDB18 dataset.

[0117] Table 2

[0118]

[0119] As shown in Table 2, the method of this invention achieves an SDR score of 11.56 dB, which is superior to existing mainstream models, including HDemucs (8.04 dB), BSRNN (10.01 dB), BS-RoFormer (10.78 dB), and the basic version of Mel-RoFormer (11.21 dB). This improvement verifies the effectiveness of combining the VAD (Voice Activity Detection) mechanism with Mel spectral features, which not only enhances the model's ability to model the time-frequency structure of speech but also improves its robustness to non-speech interference, thereby achieving higher-quality speech separation in complex mixed-audio scenarios.

[0120] The following verifies the effectiveness of the VAD information guidance method and the domain adaptation method on the GeZiXi-mix opera test set:

[0121] Existing mainstream audio source separation models are mostly designed for Western pop music, and their performance is generally low in traditional Chinese opera scenarios. This experiment selects representative models from this domain and general domains. Experimental results on the GeZiXi-mix traditional Chinese opera test set (containing mixed audio of various traditional Chinese instruments) constructed in this invention show that the Mel-RoFormer-VAD method proposed in this invention exhibits strong generalization and adaptability. The effectiveness index verification results of the domain adaptation method on the GeZiXi-mix traditional Chinese opera test set are shown in Table 3.

[0122] Table 3

[0123]

[0124] As shown in Table 3, the basic Mel-RoFormer model outperforms HDemucs, BSRNN, and BS-RoFormer on this dataset (SDR: 6.32dB, SIR: 16.54dB). After fine-tuning on the mixed instrument dataset, the model performance significantly improves to SDR 10.58dB and SIR 28.65dB, demonstrating the effectiveness of domain-adaptive training for non-Western music scenes. Further introduction of the VAD mechanism further enhances the performance of the Mel-RoFormer-VAD model, ultimately achieving SDR 10.81dB and SIR 28.77dB, the highest among all compared methods. This result fully validates that the VAD mechanism effectively optimizes attention allocation by guiding the model to focus on the active vocal regions, thereby extracting purer vocals even with complex accompaniments. Fine-tuning for the target domain data combined with VAD-guided temporal modeling is a key path to improving the separation quality of complex ethnic musical instruments. The Mel-RoFormer-VAD (fine-tuned with a mixed dataset) model of this invention has the highest performance level among all comparison methods.

[0125] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.

Claims

1. A method for opera sound source separation based on domain adaptation and time domain information guidance, characterized in that The method comprises the following steps: Step 1: constructing a field adaptation training data set of fusing Chinese traditional musical instruments: obtaining audio clips of commonly used pure musical instruments of Chinese traditional operas from a public data set, cutting the clips, superimposing random gain to mixed audio of a standard popular music source separation data set to generate new mixed audio, and retaining the original vocal track as a target label; Step 2: generating a time domain weight sequence: processing the new mixed audio by using a pre-trained voice activity detection model to output a time domain confidence sequence; for each continuous clip containing a voice, a Gaussian attenuation weight is applied to the center of the time domain confidence sequence of the clip, and normalization processing is performed to obtain a time domain weight sequence; The calculation formula of the Gaussian attenuation weight is: wherein, is a time-domain confidence sequence, is a waveform audio current time point, is a segment center time, σ is a hyperparameter that controls the decay range, and exp is an exponential function, is the final confidence value after Gaussian decay weight processing. The value of the hyperparameter σ controlling the attenuation range is 1 / 6 of the length of the clip; Step 3: constructing and training a source separation model with a VAD weighted loss: taking a frequency domain music source separation model as the main part, the input is the mixed audio generated in step 1, and the output is the target voice estimation; the loss function is weighted by the aforementioned time domain weight sequence, a VAD information weighted loss function is defined, and the model is trained with the loss function as the optimization target; The function of the VAD information weighted loss is: in, Output human voice for the model. As weight, For real human voice audio in the corresponding window value The spectrum below, The human voice audio output by the model is within the corresponding window value. The spectrum below, For weight The weight matrix is ​​extended to the corresponding spectrum size; S is the total number of windows. The range of values ​​is ; Represents norm operations; Step 4: voice separation and testing: applying the trained source separation model to real opera audio for voice separation, and testing indicators on a test set to evaluate the separation quality.

2. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 1, the commonly used pure musical instruments of Chinese traditional operas include at least one of an erhu, a pipa, a sanxian, a dizi, a suona, a qin, a zhongruan, a liuqin, a guzheng, a yangqin, and a sheng; the cut clips are non-overlapping clips with a length of 10-20s; and the random gain range is -6 dB to +3 dB.

3. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 3, the frequency domain music source separation model adopts a Mel-Band RoFormer model, which includes a frequency band segmentation module, a rotary position embedding Transformer module, and a mask generation module; the frequency band segmentation module is used to divide the input frequency spectrum into multiple sub-bands; the rotary position embedding Transformer module is used to model hierarchical attention on the time and sub-band axes; The mask generation module is used to output a complex ratio mask to separate the target voice.

4. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 3, characterized in that The execution process of the rotary position embedding Transformer module includes: Applying a Transformer to the input tensor along the time axis; Applying a Transformer to the output tensor along the sub-band axis; In each Transformer layer, the query and key are position encoded using rotary position encoding.

5. The opera source separation method based on domain adaptation and time-domain information guidance according to claim 1, characterized in that In step 4, the indicators for evaluating the separation quality include a signal distortion ratio and a signal interference ratio.

Images