Music source separation method and system based on hybrid expert self-attention network

Abstract

The invention discloses a music source separation method and system based on a hybrid expert self-attention network, and belongs to the technical field of audio signal processing and deep learning. The method comprises the following steps: receiving a time domain mixed audio signal, and obtaining a complex frequency spectrum through short-time Fourier transform; the method comprises the following steps: estimating a complex ideal proportion mask through an improved separator network MoEFormer, respectively modeling a time domain dependency relationship and a frequency domain dependency relationship in the separator network by adopting an axial attention mechanism, and replacing a feedforward network in a standard Transformer with a hybrid expert layer; the hybrid expert layer comprises an expert network specially designed for drum sound, Bass and human voice characteristics, and a self-adaptive weight distribution mechanism based on gating routing; and finally, reconstructing the separated time domain signal through inverse short-time Fourier transform. Through expert specialization and feature adaptive fusion, the problem of multi-sound-source feature confusion is effectively solved, and low calculation complexity is kept while the separation precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of audio signal processing and deep learning, and particularly relates to a music source separation method and system based on a hybrid expert self-attention network. BACKGROUND

[0002] Music source separation (MSS) aims to extract independent sound source components from mixed audio signals, such as separating a song into independent vocals, drum sets, bass, and other instrumental parts. This technology has wide application value in music production, intelligent mixing, lyrics recognition and alignment, karaoke generation, music copyright detection and content analysis, etc. However, due to the characteristics of music signals, such as multiple sound sources highly overlapping in time-frequency domain, complex instrument types and playing techniques, and large dynamic range, it has been a long-term challenge in the field of audio processing to achieve high-precision sound source separation.

[0003] Early music source separation methods were mostly inspired by speech separation techniques, including methods based on computational auditory scene analysis (CASA), separation algorithms based on sparse representation, non-negative matrix factorization (NMF) and its variants, and robust principal component analysis (RPCA). These methods usually explicitly model the structure of sound sources or noise, and although they have certain interpretability, they are difficult to effectively capture the complex nonlinear structure and highly overlapping sound source components in music, and there is a clear bottleneck in separation performance.

[0004] In recent years, deep neural networks (DNNs) have become the mainstream technology path for music source separation due to their strong nonlinear fitting and hierarchical feature learning capabilities. According to the different preferences of network structure for feature representation, existing DNN-based MSS methods can be roughly divided into three categories: one is represented by convolutional neural networks (CNNs), which focus on extracting local patterns and short-time structural features in the time-frequency domain, and have good representation ability for transient notes and local harmonic components; two is represented by recurrent neural networks (RNNs / LSTMs / GRUs), which focus on modeling time series dependencies and can capture dynamic evolution and rhythm continuity in music; three is based on self-attention mechanisms (such as Transformers), which are more suitable for describing global features such as homophonic progressions and long-range rhythm structures by establishing global context dependencies.

[0005] Although the above three types of methods have their own advantages, a single type of network structure often has difficulty in comprehensively considering the multi-scale feature expression of local details, time series dynamics and global structure. Therefore, attempts have been made in the prior art to mix CNN, RNN and Transformer to improve the separation ability of the model in complex music scenes through structure combination. However, this type of method still has a common limitation: all sound sources share the same set of feature extraction and representation rules, such as the same set of convolution kernels that need to process the fine harmonics of the human voice and the impact transient of the drum, which easily causes feature confusion and representation conflict, thereby limiting the further improvement of separation accuracy. SUMMARY

[0006] To overcome the above-mentioned defects, the present application introduces a Mixture-of-Experts (MoE) mechanism and proposes a new music source separation network architecture named MoEFormer. The core of the architecture is to regard the separation of different sound sources as different specialized "tasks" and design expert modules dedicated to specific sound sources such as drums, vocals and bass, so that each expert can focus on the deep feature representation and reconstruction of its target sound source. At the same time, through the designed gating routing network, the output results of the most relevant experts are dynamically activated and fused according to the input audio content, so as to realize feature adaptive allocation and fusion in the multi-sound source separation task, significantly improving the model expression ability and separation accuracy.

[0007] The present application provides a music source separation system and method based on a Mixture-of-Experts self-attention network. The system is improved and innovated on the basis of the existing BS-RoFormer system architecture, adopts an encoder-separator framework as a whole, and integrates a short-time Fourier transform (STFT) as a domain transformation module. When the system is working, it first receives a time-domain mixed audio signal, which is transformed into a complex spectrum by STFT. Then, the separator network estimates a complex ideal ratio mask (cIRM). Finally, the separated time-domain signal is reconstructed by inverse STFT (ISTFT) transformation.

[0008] The music source separation method based on the Mixture-of-Experts self-attention network provided by the present application comprises the following steps:

[0009] Step S1: receiving a time-domain mixed audio signal and performing a short-time Fourier transform on the time-domain mixed audio signal to obtain a corresponding complex spectrum;

[0010] Step S2: inputting the complex spectrum into an improved separator network, the separator network being a Transformer network based on a Mixture-of-Experts mechanism, for estimating a complex ideal ratio mask corresponding to each sound source in the complex spectrum;

[0011] Step S3: operating the estimated complex ideal ratio mask with the complex spectrum to obtain an estimated spectrum of each sound source;

[0012] Step S4: performing inverse short-time Fourier transform on the estimated spectrum of each sound source to reconstruct a separated time-domain signal;

[0013] The improved separator network models the time-domain and frequency-domain dependency relationship through an axial attention mechanism, and replaces the feedforward network in the standard Transformer layer with a hybrid expert layer; the hybrid expert layer includes a plurality of expert networks specially designed for different sound source characteristics, and an adaptive weight distribution mechanism based on a gate routing, used for dynamically fusing the outputs of the plurality of expert networks.

[0014] The core of the application is to optimize the separator network in the BS-RoFormer system and transform it into a MoEFormer network. Specifically, the standard Transformer encoder layer is replaced by a dynamic gate hybrid expert (MoE) layer, while the original residual connection and layer normalization structure are retained to ensure the stability of the training. The network continues the axial attention mechanism, and processes the time-domain and frequency-domain features through two independent Transformer modules, thereby effectively capturing the long-term time dependency relationship and spectral harmonic structure in the audio signal.

[0015] The application is a substantial improvement based on the BS-RoFormer system. By introducing the hybrid expert mechanism, the model can be specialized to process the unique acoustic characteristics of different sound sources (such as vocals, drums, bass, etc.) in music, significantly improving the separation accuracy and generalization ability in complex music scenes.

[0016] The application uses the following mathematical symbols to describe the system. Assume that the mixed music signal to be separated is , wherein is the number of channels, is the length. The mixed signal contains source signals :

[0017]

[0018] In the application, =4, and the sound source signals are respectively: vocals, drums, bass and other instruments. The application uses a network framework with STFT domain transformation, , , are respectively , and the estimated values Representation in STFT transform domain. The network first estimates the complex ideal ratio mask , assuming the separation network is , then we have

[0019] (2)

[0020] Accordingly, we estimate :

[0021] (3)

[0022] Then the estimated value of the music signal is recovered by the inverse STFT transform .

[0023] (4)

[0024] where is the inverse STFT transform.

[0025] The MoE layer is composed of shared feature extraction, gating network and weighted summation of three expert networks.

[0026] As an embodiment of the present application, the hybrid expert layer comprises:

[0027] a shared feature extraction module for normalizing and nonlinearly transforming the output of the axial attention mechanism to generate shared features;

[0028] a gating network for generating weights corresponding to each expert network according to the shared features;

[0029] a plurality of expert networks, each of which is designed for the acoustic characteristics of one or a type of sound source;

[0030] The final output of the hybrid expert layer is the weighted sum of the outputs of each expert network, and the weights are determined by the gating network.

[0031] As an embodiment of the present application, the hybrid expert layer comprises:

[0032] Shared feature extraction: the shared feature extraction layer normalizes the attention output through RMS, linear projection and GELU activation to finally generate shared features:

[0033] (5)

[0034] where is the projection matrix, denotes RMS normalization, and is a learnable parameter.

[0035] Gating network: shared features are compressed into global features by sequence mean pooling, and then 3 expert weights are predicted by two layers of MLP (with GELU activation) and normalized by Softmax:

[0036] (6)

[0037] where , and are learnable parameters.

[0038] Weighted sum: the model dynamically fuses expert outputs by the weights generated by the gating network.

[0039] (7)

[0040] The expert networks are specially designed for different physical and perceptual characteristics of sound sources.

[0041] As an embodiment of the present application, the plurality of expert networks at least includes:

[0042] Drum sound expert network, adopting one-dimensional convolution and grouped convolution structure, is used to capture the short-time pulse and transient characteristics of drum sound;

[0043] Bass expert network, adopting multi-scale one-dimensional convolution, grouped convolution and gated linear unit structure, is used to capture the low-frequency harmonic structure and long-time energy distribution of bass;

[0044] Vocal expert network, adopting bidirectional long short-term memory network structure, is used to model the continuous pitch variation and complex formant structure of vocal.

[0045] As an embodiment of the present application, the three expert networks include:

[0046] 1) Drum sound expert network: as Figure 3 For the short-time pulse and high-frequency energy burst characteristics of drum sound, one-dimensional convolution structure is used to capture the time-domain transient mode, and grouped convolution is used to process different frequency bands in parallel to optimize the detection of vertical energy burst in the spectrum.

[0047] 2) Bass expert network: as Figure 4 For the low-frequency harmonic structure and long-time energy distribution of bass, multi-scale one-dimensional convolution is used to capture complete harmonic periods and fine local features, and grouped convolution is introduced to separate the fundamental frequency and overtone processing paths, and GLU gating mechanism is used after linear transformation to strengthen the selective response to harmonic characteristics.

[0048] 3) Vocal expert network: as Figure 5For the continuous pitch variation and complex formant structure of human voice, a bidirectional long short-term memory network (BiLSTM) is used for time series modeling to track the melody contour variation, and a time series priority processing mechanism and feature dimension transposition are used to strengthen the modeling capability of long-time pitch continuity.

[0049] As a preferred embodiment of the present application, the drum expert network uses one-dimensional convolution with a convolution kernel size of 5, and uses grouped convolution with a group number of 4 to process different frequency bands in parallel.

[0050] As a preferred embodiment of the present application, the bass expert network uses multi-scale one-dimensional convolution with a convolution kernel size of 9 and 3 respectively, and uses grouped convolution with a group number of 2 to separate audio, and uses a gated linear unit mechanism after linear transformation.

[0051] As a preferred embodiment of the present application, the human voice expert network uses a bidirectional long short-term memory network, and uses a time series priority processing mechanism and feature dimension transposition to strengthen the modeling capability of long-time pitch continuity.

[0052] As an embodiment of the present application, when training the separator network, the standard deviation of the expert weight distribution is added as a regularization term in the loss function to prevent expert degradation.

[0053] The present application also provides a music source separation system for implementing the music source separation method based on the mixed expert self-attention network, comprising:

[0054] A preprocessing module configured to receive a time-domain mixed audio signal and perform a short-time Fourier transform;

[0055] The improved separator network of claim 1 is configured to estimate a complex ideal ratio mask from a complex spectrum;

[0056] A post-processing module configured to convert the estimated spectrum into a separated time-domain signal.

[0057] The present application also provides a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the music source separation method based on the mixed expert self-attention network.

[0058] The present application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the music source separation method based on the mixed expert self-attention network when executing the computer program.

[0059] The MoEFormer architecture realizes the specialized extraction and adaptive fusion of different sound source characteristics, effectively solves the multi-sound source feature confusion problem, improves the separation precision while maintaining a low computational complexity. Experiments on the MUSDB18-HQ dataset show that the separation precision can be greatly improved under the condition of limited computing power (training for 60 rounds), the signal distortion ratio (SDR) index is about one-third higher than the current mainstream method (SOTA) on average, effectively verifying the advancement and practicality of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0060] is the overall system flowchart of the present application.

[0061] MoE replaces the FFN of the Transformer (left), and the mixed expert network module (right).

[0062] is the drum expert network layer.

[0063] is the bass expert network layer.

[0064] is the human voice expert network layer. DETAILED DESCRIPTION

[0065] The music source separation method described in the present application can be implemented by the following preferred embodiments. Those skilled in the art should understand that the described embodiments are only used to illustrate the present application, and should not be regarded as limiting the scope of the present application.

[0066] 1) Model training and optimization

[0067] To implement the present application, the MoEFormer model needs to be trained. To prevent expert degradation and ensure that all expert modules are effectively utilized, in the loss function design, in addition to the original loss terms of BS-RoFormer, the standard deviation of the expert weight distribution is added as a regularization term, and the weight coefficient λ of this term is set to 0.01. The specific hyperparameter configuration of the model is as follows: the dimension (Dim) of the Transformer is set to 256, and the number of encoder layers (depth) is 4 layers. Each layer uses a multi-head self-attention mechanism, the attention head dimension is 64, and the attention Dropout rate and the feedforward network Dropout rate are both set to 0.3. The model outputs the separation results of 4 audio tracks of human voice, drums, bass and other instruments. To optimize the memory usage during training, the gradient checkpointing technique is enabled in the model implementation.

[0068] The AdamW optimizer is used for training, and the initial learning rate is set to 2×10⁻ 4, the weight decay coefficient is 0.01. The learning rate scheduling adopts a step decay strategy, that is, the learning rate is decayed to 90% of the original value every 10 epochs. The batch size is set to 8, and a double-GPU data parallel strategy is used to accelerate training. The entire training process uses the mixed precision (AMP) technique to further reduce memory usage and improve training speed.

[0069] 2) Experimental environment and settings

[0070] The training and evaluation experiments were conducted on a server equipped with two NVIDIA V100 (32G memory) GPUs. The deep learning framework used PyTorch. To speed up the data loading process, 8 worker threads were used for training data loading, 4 worker threads were used for validation data loading, and memory pinning was enabled to optimize the data transfer efficiency from the host to the GPU. During model training, a checkpoint was saved every epoch, containing model parameters, optimizer state, and current training configuration information to support resuming training from the breakpoint in case of unexpected interruption.

[0071] 3) Performance evaluation

[0072] The model performance uses the signal distortion ratio (SDR) as the core evaluation indicator. The SDR values of vocals, bass, drums, and others are calculated respectively, and their average value is taken as the evaluation of the overall separation performance. To improve the accuracy of the evaluation, a silence detection mechanism was introduced before the calculation to filter out the silent segments in the source signal and the estimated signal, avoiding interference with the SDR calculation. The formula for calculating SDR is as follows:

[0073] (8)

[0074] where is the source signal, is the estimated value. The higher the SDR score, the better the separation quality.

[0075] The experimental conditions of the present application are as follows:

[0076] 1) Dataset;

[0077] MUSDB18-HQ was used as the dataset, which contains 100 and 50 songs for training and validation, respectively. All audio files in the dataset are saved in uncompressed WAV format, ensuring high fidelity of audio quality. Each audio contains four tracks: vocals, drums, bass, and others, and the sampling rate is 44.1 kHz.

[0078] 2) BS-RoFormer algorithm for comparison;

[0079] To objectively evaluate the performance of the proposed model (MoEFormer) of the application, the optimal model BS-RoFormer in SDX23 (music separation challenge) is selected as the baseline model for comparison. Since the original BS-RoFormer is a large-scale Transformer model (which requires the use of 16 V100 GPUs to train for about a week), in order to conduct a fair comparison under limited experimental conditions (2 V100 GPUs are used in this experiment), the depth of the Transformer is uniformly set to 4 layers, and the dimension (dim) is uniformly set to 256, and all comparison experiments are conducted in the same hardware and software environment.

[0080] 3) Experimental results;

[0081] The performance comparison results of the model (MoEFormer) of the application and the baseline model (BS-RoFormer) under different training rounds are shown in Table 1.

[0082] Table 1: Model SDR comparison

[0083]

[0084] As can be seen from Table 1, the MoEFormer model proposed in the application is consistently significantly better than the baseline model in all training rounds and all audio track separation tasks. Specifically, after 60 rounds of training, the average SDR of the model of the application reaches 6.00 dB, compared with 4.00 dB of the baseline model, the performance improvement is 50% (i.e. absolute improvement of 2.00 dB, relative improvement of one-third), which fully proves the effectiveness of introducing the mixed expert mechanism.

[0085] In addition, the experimental results also reveal the following important phenomena:

[0086] Performance gain persistence: As the training rounds increase, the performance advantage of MoEFormer compared to the baseline model continues to expand (from 0.81 dB difference at 10 rounds to 2.00 dB at 60 rounds), indicating that the mixed expert structure used in the application has excellent continuous learning ability and can continuously improve performance over a longer training period.

[0087] Superiority of convergence speed: The performance of the model of the application (average SDR 3.33 dB) has exceeded the effect of the baseline model trained for 30 rounds (average SDR 3.27 dB) when trained for only 10 rounds, indicating that MoEFormer has a faster convergence speed and can effectively save training time and computing cost.

[0088] Enhanced complex sound source separation capability: It is particularly noteworthy that the most significant performance improvement (from 2.41 dB to 5.26 dB, an increase of about 118%) of the model of the application is achieved on the most challenging complex sound effect separation task of Others (other accompaniment), which indicates that the strategy of the application for designing expert networks customized for different sound source characteristics greatly enhances the separation capability of complex and heterogeneous sound sources.

[0089] In summary, through a series of comparative experiments, it is proved that the music source separation method based on the mixed expert mechanism provided by the application is significantly better than the existing advanced technology in terms of separation accuracy, convergence speed and complex sound source processing capability, etc. in multiple dimensions, and exhibits outstanding technical advantages and wide application potential.

Claims

1. A method for music source separation based on a hybrid expert self-attention network, characterized in that, The method comprises the following steps: Step S1: receiving a time-domain mixed audio signal, and performing a short-time Fourier transform on the time-domain mixed audio signal to obtain a corresponding complex spectrum; Step S2: inputting the complex spectrum into an improved separator network, the separator network being a Transformer network based on a mixed expert mechanism, for estimating a complex ideal ratio mask corresponding to each sound source in the complex spectrum; Step S3: performing an operation on the estimated complex ideal ratio mask and the complex spectrum to obtain an estimated spectrum of each sound source; Step S4: performing an inverse short-time Fourier transform on the estimated spectrum of each sound source to reconstruct a separated time-domain signal; The improved separator network models time-domain and frequency-domain dependency relationships through an axial attention mechanism, and replaces a feedforward network in a standard Transformer layer with a mixed expert layer; the mixed expert layer comprises a plurality of expert networks specially designed for different sound source characteristics, and an adaptive weight distribution mechanism based on a gated routing mechanism, for dynamically fusing outputs of the plurality of expert networks.

2. The music source separation method based on the hybrid expert self-attention network according to claim 1, characterized in that, The mixed expert layer comprises: a shared feature extraction module configured to normalize and nonlinearly transform an output of the axial attention mechanism to generate shared features; a gating network configured to generate weights corresponding to each expert network according to the shared features; a plurality of expert networks, each expert network being designed for acoustic characteristics of one or a type of sound source; a final output of the mixed expert layer being a weighted sum of outputs of the expert networks, the weights being determined by the gating network.

3. The music source separation method based on the hybrid expert self-attention network according to claim 2, characterized in that, The plurality of expert networks at least comprises: a drum sound expert network adopting a one-dimensional convolution and a grouped convolution structure, for capturing short-time pulse and transient characteristics of drum sounds; a bass sound expert network adopting a multi-scale one-dimensional convolution, a grouped convolution, and a gated linear unit structure, for capturing low-frequency harmonic structure and long-time energy distribution of bass sounds; a vocal sound expert network adopting a bidirectional long short-term memory network structure, for modeling continuous pitch variation and complex formant structure of vocal sounds.

4. The music source separation method based on the hybrid expert self-attention network according to claim 3, characterized in that, The drum sound expert network uses a one-dimensional convolution with a convolution kernel size of 5, and parallel processes different frequency bands using a grouped convolution with a grouping number of 4.

5. The music source separation method based on the hybrid expert self-attention network according to claim 3, characterized in that, The bass sound expert network uses a multi-scale one-dimensional convolution with convolution kernel sizes of 9 and 3, and separates audio using a grouped convolution with a grouping number of 2, and uses a gated linear unit mechanism after linear transformation.

6. The music source separation method based on the hybrid expert self-attention network according to claim 3, characterized in that, The vocal sound expert network enhances modeling capability for long-time pitch continuity through a time priority processing mechanism and feature dimension transposition.

7. The music source separation method based on the hybrid expert self-attention network according to claim 1, characterized in that, When training the separator network, a standard deviation of expert weight distribution is added as a regularization term in a loss function to prevent expert degradation.

8. A music source separation system for implementing the music source separation method based on the hybrid expert self-attention network according to any one of claims 1-7, characterized in that, The method comprises: a preprocessing module configured to receive a time-domain mixed audio signal and perform a short-time Fourier transform; the improved separator network of claim 1, configured to estimate a complex ideal ratio mask from a complex spectrum; a post-processing module configured to convert an estimated spectrum into a separated time-domain signal. 9.A non-transitory computer readable storage medium storing a computer program, wherein the computer program comprises instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 8. The computer program, when executed by a processor, implements the method of any one of claims 1-7.

10. An electronic device, comprising: A computer program product comprising a storage medium to store the program code of a computer program, the program code dynamicall y executable by a processor to cause the processor to carry out the method according to any one of claims 1 to 7. A computer program for performing the method according to any one of claims 1 to 7 when the computer program is executed by a processor. A computer program for performing the method according to any one of claims 1 to 7 when the computer program is executed by a processor. A computer program for performing the method according to any one of claims 1 to

Images