Target sound source extraction method based on cross attention-state space model

Abstract

The invention discloses a target sound source extraction method based on a cross attention-state space model, and the method comprises the steps: 1) constructing and training a Wave-CA-Mama neural network which comprises a one-dimensional convolution layer, a plurality of dilated convolution layers, a CA-Mama layer, a one-dimensional deconvolution layer, and a plurality of full-connection embedding layers; 2) inputting the multi-sound-source mixed signal into a Wave-CA-Mama neural network, and performing feature extraction on the multi-sound-source mixed signal through a one-dimensional convolutional layer and each dilated convolutional layer in sequence to obtain features of the mixed signal; 3) inputting the category of a target sound source into the full-connection embedding layer to obtain category embedding of the target sound source; 4) embedding and inputting the features of the mixed signal and the category into the CA-Mama layer for feature fusion; and 5) inputting the fused features into the one-dimensional deconvolution layer to obtain a target sound source signal in the multi-sound-source mixed signal.

Description

Technical Field

[0001] The present invention belongs to the fields of machine learning and sound source separation, and particularly relates to a method for extracting a target sound source based on a cross-attention-state space model. Background Art

[0002] The Transformer model is widely used in feature fusion in the task of target sound source extraction. Its typical model, such as Waveformer, uses a Transformer decoder with cross-attention to extract the target sound using the class label as a clue. However, the computational cost and memory requirements of the Transformer are usually high, mainly because the memory requirements and computational complexity of the attention mechanism in the Transformer grow quadratically with the sequence length. In recent years, the state space model has received increasing attention. Especially the latest selective state space model, Mamba, can be trained in parallel and can perform efficient inference at linear time complexity like an RNN during inference. Mamba has achieved performance comparable to that of the Transformer in the fields of natural language, image, and audio signal processing with fewer parameters and computational loads. However, due to the internal structure limitations of Mamba, it cannot capture the dependencies between different sequences like the cross-attention mechanism, so it cannot extract the target signal from the mixed signal based on the clue of the target signal, which limits its application in the target sound extraction task.

[0003] Due to the above reasons or the deficiencies of existing methods, there is a need for a Mamba model that can accept two different signals as inputs, extract the relevant information between different signals, and apply it to the target sound source extraction model. Summary of the Invention

[0004] The technical problem to be solved by the present invention is how to enable Mamba to have the ability to process different sequences simultaneously and apply it to the target sound source extraction model. The purpose of the present invention is to provide a method for extracting a target sound source based on a cross-attention-state space model. Based on the implicit attention mechanism in Mamba, the present invention calculates the dependency between a given clue and the audio mixed signal, and proposes a cross-attention-state space model (CA-Mamba) that can process the correlation between different sequences, and applies it to the target sound source extraction model Waveformer, and proposes a target sound source extraction network Wave-CA-Mamba based on CA-Mamba to achieve more efficient target sound source extraction.

[0005] The technical solution adopted by the present invention is as follows:

[0006] A method for extracting a target sound source based on a cross-attention-state space model, the steps of which include:

[0007] 1) Input the multi-source mixed signal into a one-dimensional convolutional layer and a dilated convolutional layer to obtain the features of the mixed signal;

[0008] 2) Input the clues (such as category) of the target sound source into a fully connected embedding layer to obtain the clue features (such as category embedding) of the target sound source;

[0009] 3) Input the features of the mixed signal and the category embedding into the CA-Mamba layer for feature fusion;

[0010] 4) Input the fused features into a one-dimensional transposed convolutional layer to obtain the target sound source signal in the multi-source mixed signal.

[0011] Furthermore, the Wave-CA-Mamba neural network includes one one-dimensional convolutional layer, ten dilated convolutional layers, one CA-Mamba layer, one one-dimensional transposed convolutional layer, and two fully connected embedding layers. The fully connected embedding layer includes one linear layer and one layer normalization (LayerNorm) layer.

[0012] After the multi-source mixed signal passes through the one-dimensional convolutional layer, it will be output to the first dilated convolutional layer, and the first dilated convolutional layer will output to the second dilated convolutional layer, and so on, to obtain the features of the mixed signal.

[0013] After the category features of the target sound source are converted into one-hot vectors, they will be input to the first fully connected embedding layer. Among them, the output after passing through the linear layer will pass through the layer normalization layer and the ReLU activation function, and then output to the second fully connected embedding layer. The second fully connected embedding layer undergoes the same operation, and the output is the category embedding of the target sound source.

[0014] Multiply the category embedding of the target sound source by the features of the mixed signal to obtain the time-varying category embedding, use it as the query, use the features of the mixed signal as the key and value, and input them into the CA-Mamba layer. The CA-Mamba layer outputs to the one-dimensional transposed convolutional layer.

[0015] Furthermore, the CA-Mamba layer includes: a query fully connected layer, a key-value fully connected layer, a Mamba layer, and an output fully connected layer.

[0016] The query input of the CA-Mamba layer passes through the query fully connected layer to obtain the query features, and the key-value input passes through the key-value fully connected layer to obtain the key-value features. Input the query features, as well as the key and value features, into the corresponding positions of the Mamba layer. That is, the C parameter in Mamba is obtained from the query features, and other parameters and the original input of Mamba are the key-value features, to obtain the fused features, and then output to the output fully connected layer to map the feature dimension back to the dimension before inputting to CA-Mamba.

[0017] Furthermore, the Wave-CA-Mamba neural network is trained using the Adam optimizer. The dataset used for training is the FSD Kaggle 2018 dataset, and the loss function used is the negative of 0.9 times the scale-invariant signal-to-noise ratio (SI-SNR) and 0.1 times the signal-to-noise ratio (SNR), which is the same as the original Waveformer.

[0018] A server, comprising a memory and a processor, wherein the memory stores a computer program configured to be executed by the processor, and the computer program includes instructions for performing the steps in the above method.

[0019] A computer-readable storage medium, on which a computer program is stored, wherein the computer program, when executed by a processor, implements the steps of the above method.

[0020] The beneficial effects of the present invention are:

[0021] 1. Expand the application scenarios of Mamba, enabling CA-Mamba to process two different signals like a cross-attention network.

[0022] 2. By applying CA-Mamba to the Waveformer network, Wave-CA-Mamba has fewer network parameters and computational loads, and maintains or improves the sound source separation performance compared to the original Waveformer. Description of the Drawings

[0023] Figure 1 It is a typical framework diagram for target sound source extraction.

[0024] Figure 2 It is a structural diagram of the target sound source extraction network Wave-CA-Mamba based on CA-Mamba. Detailed Embodiments

[0025] The following introduces a method for extracting target sound sources based on a cross-attention state space model provided by the present invention in combination with the drawings and formulas:

[0026] The structure of a classic target sound source extraction is as Figure 1As shown in the figure, the network takes the multi-source mixed signal and the clues of the target sound source (the category, fundamental frequency, video, etc. of the target sound source) as inputs, encodes them into feature representations, realizes feature fusion in the separation layer, extracts the features of the target sound source based on the correlation between the clue features of the target sound source and the features of the mixed signal, and then obtains the signal of the target sound source through the decoder. Using the cross-attention module of Transformer, with the clue features as the query input and the features of the mixed signal as the key and value inputs, is an effective way of feature fusion.

[0027] Mamba is a state space model whose performance is not weaker than that of Transformer, and the number of parameters and computational complexity are much smaller than those of Transformer. Its calculation formula in matrix form is:

[0028] y = αx, (I)

[0029]

[0030]

[0031] Among them, S C (·), S Δ (·), S B (·) are all linear layers, and A is the parameter of the model. By defining:

[0032] Q i = S C (x i ) = f m，q (x i ), (4)

[0033]

[0034]

[0035] V i = x i , (7)

[0036] Then the calculation of Mamba can be written as:

[0037]

[0038] This is similar to the calculation of the attention mechanism, the difference being the addition of H i,jThe item controls the importance of the most recent (i-j) elements. In this way, the input that is to be used as the query item in the cross-attention mechanism is brought into Equation (4), and the items that serve as keys and values are brought into Equations (5), (6), and (7). That is, CA-Mamba similar to the cross-attention mechanism is implemented through Mamba. Thus, the process of the target sound extraction network Wave-CA-Mamba implemented based on Waveformer using CA-Mamba is as follows:

[0039] Step 1: Input the mixed signal of multiple sound sources collected by the microphone into a one-dimensional convolutional layer to obtain the feature X1 of the mixed signal.

[0040] Step 2: Input X1 into a dilated convolutional layer to extract local temporal information and obtain the feature X2 of the mixed signal.

[0041] Step 3: Convert the category information of the target sound source into a one-hot vector and input it into a fully connected embedding layer to obtain the category feature E of the target sound source cls , and then multiply it by X2 to obtain the time-varying category feature E c .

[0042] Step 4: Input E c as the query and X2 as the key and value into CA-Mamba. Specifically, after passing E c through the query fully connected layer, it is brought into x in (4) i , and after passing X2 through the key-value fully connected layer, it is brought into x in Equations (5), (6), and (7) i to achieve feature fusion. Then, the output passes through the output fully connected layer to extract the feature X of the target sound source trg .

[0043] Step 5: Input X trg into a one-dimensional transposed convolutional layer to obtain the time-domain signal of the target sound source.

[0044] Method evaluation experiment

[0045] The evaluation of the method of the present invention is carried out through simulation experiments. The dataset used in the experiment is the FSD18K dataset, which contains 41 different categories of sounds. The experiment is based on the mirror model and uses the gpuRIR library to simulate the transfer function. The size range of the simulated room is 3×3×3m 3 ~10×8×6m 3 , the reverberation time range is 0.2s~l.0s, and the room size and reverberation time are random. The installation position of the microphone array is within 10% of the center of the room, and the Eigenmike microphone array configuration is adopted. The multi-channel microphone signal is then converted into a FOA (First-Order Ambisonics) signal. The number of sound sources set in the experiment is 2, and the number of target sound sources is 1.

[0046] To better fit the real - world situation, the sound source we consider is a moving sound source. During the generation of the sound - source trajectory, we randomly select two points in the room, which are used as the starting point p0=(x0, y0, z0) and the ending point p N =(x N , y N , z N ) of the sound - source trajectory respectively. The line segment connecting these two points forms the basic motion trajectory of the sound source. Subsequently, sinusoidal curves with random amplitudes A=(A x , A y , A z ) and frequencies ω=(ω x , ω y , ω z ) are added along each coordinate axis on this line segment, thus constructing the final trajectory of the sound source. In addition, we ensure that the number of oscillation periods does not exceed 2. The position of the sound source at the time - sampling point n can be calculated as follows:

[0047]

[0048] The evaluation and comparison methods are the original Transformer - based Waveformer network and Wave - CA - Mamba (the network proposed in this invention). The network parameter settings of Wave - CA - Mamba are the same as those of Waveformer. The d_model and d_state parameters of Mamba are set to 32, the d_conv parameter is set to 4, and the expand parameter is set to 2. Table 1 shows the SI - SNR values of the two methods. It can be seen that Wave - CA - Mamba achieves a higher SI - SNR, which proves that the feature - fusion method of this invention effectively extracts the part related to the given - class clues in the mixed - audio features and realizes better performance in target - sound - source extraction.

[0049] Table 1: SI - SNR values of target - sound - source extraction for different methods

[0050]

[0051] Table 2 shows the number of model parameters and the computational complexity of the two methods. It can be seen that compared with Waveformer, the model size of Wave - CA - Mamba is reduced by 20%. In addition, in terms of computational load, the number of MACs is reduced by 45%. This reduction is attributed to the fact that the calculation of CA - Mamba has a linear inference complexity, which is much lower than the quadratic complexity of the attention mechanism.

[0052] Table 2: Number of model parameters and computational complexity for different methods

[0053]

[0054] In summary, the Wave-CA-Mamba based on CA-Mamba of the present invention can achieve separation performance comparable to or even better than that of the Transformer-based model with lower resource consumption. By utilizing the attention mechanism inside Mamba, the present invention expands it into CA-Mamba that supports the calculation of cross-attention. The conditional sound source extraction network based on CA-Mamba achieves higher separation performance with fewer model parameters and lower computational load.

[0055] Although specific embodiments and drawings of the present invention are disclosed for illustrative purposes, which are intended to help understand the content of the present invention and implement it accordingly, those skilled in the art can understand that various substitutions, changes, and modifications are possible without departing from the spirit and scope of the present invention and the appended claims. Therefore, the present invention should not be limited to the content disclosed in the preferred embodiments and drawings.

Claims

1. A method for extracting a target sound source based on a cross-attention-state space model, the steps of which include: 1) Construct and train a Wave-CA-Mamba neural network; The Wave-CA-Mamba neural network includes a one-dimensional convolutional layer, multiple dilated convolutional layers, a CA-Mamba layer, a one-dimensional transposed convolutional layer, and several fully connected embedding layers; 2) Input a multi-source mixed signal into the Wave-CA-Mamba neural network, and the multi-source mixed signal sequentially performs feature extraction through the one-dimensional convolutional layer and each dilated convolutional layer to obtain the features of the mixed signal; 3) Input the category of the target sound source into the fully connected embedding layer to obtain the category embedding of the target sound source; 4) Input the features of the mixed signal and the category embedding into the CA-Mamba layer for feature fusion; 5) Input the fused features into the one-dimensional transposed convolutional layer to obtain the target sound source signal in the multi-source mixed signal.

2. The method according to claim 1, characterized in that Use the Adam optimizer to train the Wave-CA-Mamba neural network. The dataset used for training is the FSD Kaggle 2018 dataset, and the loss function used is the same as that of the training sound source extraction model Waveformer.

3. The method according to claim 1, wherein The Wave-CA-Mamba neural network includes a one-dimensional convolutional layer, ten dilated convolutional layers, a CA-Mamba layer, a one-dimensional transposed convolutional layer, and two fully connected embedding layers; Each of the fully connected embedding layers includes a linear layer and a layer normalization layer.

4. The method according to claim 1 or 2 or 3, characterized in that, The CA-Mamba layer includes a query fully connected layer, a key-value fully connected layer, a Mamba layer, and an output fully connected layer; multiply the category embedding of the target sound source by the features of the mixed signal to obtain a time-varying category embedding as the query, and use the features of the mixed signal as the key and value; input the query into the query fully connected layer to obtain query features, and input the key and value into the key-value fully connected layer to obtain key-value features; input the query features and the key-value features into the Mamba layer for fusion, obtain the fused features and input them into the output fully connected layer, and map the feature dimension back to the dimension before inputting into the CA-Mamba.

5. A server, characterized in that, It includes a memory and a processor. The memory stores a computer program, and the computer program is configured to be executed by the processor. The computer program includes instructions for executing any one of the methods according to claims 1 to 4.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements any one of the methods according to claims 1 to 4.

Images