A noise-containing speech separation method based on a selective state space model

Abstract

The application discloses a noisy speech separation method based on a selective state space model and belongs to the technical field of speech separation. The method introduces a Mamba with a selection mechanism into a time-domain speech separation framework to design a U-shaped selective speech separation module (UMamba), so as to improve the distinguishing ability of the model to noise and speech, reduce the misallocation of signals and noise in the noisy speech separation task, and reduce the parameter quantity of the model. The method firstly extracts speech features by using one-dimensional convolution, then separates speech by using the designed U-shaped selective speech separation module (UMamba), further enhances the speech features after separation by using a feature refiner (FeaturePcs), finally restores the speech signal by using a one-dimensional deconvolution layer, and realizes end-to-end speech separation. Finally, the performance of the proposed method is evaluated. The application keeps a very small model parameter quantity but can achieve excellent performance, and is a very friendly method for users with limited resources.

Description

Technical Field

[0001] This invention belongs to the field of speech separation technology, specifically relating to a noisy speech separation method based on a selective state-space model. Background Technology

[0002] Speech separation, aiming to isolate target speech from mixed signals, is a crucial front-end task in speech recognition applications. In recent years, significant progress has been made in speech separation, driven by robust techniques such as sparse component analysis, Bayesian inference methods, and deep learning-based approaches. Deep learning has become the dominant paradigm in speech separation, leveraging the power of neural networks to extract relevant features from audio signals. Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), and their variants, such as Long Short-Term Memory (LSTM) networks, have been widely applied to speech separation tasks and have achieved excellent separation performance.

[0003] Deep learning-based speech separation methods commonly employ time-domain and frequency-domain modeling, which utilize different representation methods and processing strategies when handling speech signals. Frequency-domain speech separation methods use spectral information to separate the source signal based on frequency distribution differences, but the separation and reconstruction of phase information remains a challenge. Time-domain speech separation models, due to their simplicity and intuitiveness, and their effectiveness in processing dynamic signals, have gained widespread application.

[0004] Temporal speech separation models process the temporal information of speech signals directly through the speech waveform on the time axis, typically analyzing the short-term features of the signal. In recent years, numerous studies have been conducted on end-to-end temporal audio processing methods, achieving excellent performance. Conv-TasNet, proposed by the Facebook AI Research team, is a fully convolutional neural network model that achieves efficient end-to-end speech separation by directly processing speech signals in the time domain. DPRNN (Dual-Path RNN), proposed by the Merl research team, significantly improves speech separation performance by performing recurrent neural network processing at both the frame and segment levels. Sepformer, proposed by the Meta AI team, combines Transformer and convolutional neural networks to directly process speech signals in the time domain, achieving efficient end-to-end speech separation. Conv-TasNet, DPRNN, and Sepformer are currently the mainstream models for temporal speech separation, and many other models have evolved from them. These deep learning-based speech separation models have achieved significant performance on clean datasets. However, accurately separating target speech under noisy conditions, i.e., noisy speech separation, remains a very challenging task.

[0005] In noisy speech separation, traditional mask-based speech separation frameworks often encounter difficulties because they tend to incorrectly assign background noise to the target speaker. A popular approach is to use a speech enhancement module as a preprocessing step before separation. While this method is effective, it can introduce a new problem: oversuppression. This is because the speech enhancement module inevitably removes some information useful for separation, leading to suboptimal separation performance. Summary of the Invention

[0006] To reduce noise interference in speech separation, this invention treats background noise as an independent output, predicting it along with the target speaker's speech. Furthermore, to mitigate the risk of oversuppression, this invention employs a codec speech separation framework based on the Selective State Space Model (Mamba). While Transformer-based models currently dominate the field of speech separation, their large number of network parameters leads to significant computational overhead during training and inference. Inspired by recent breakthroughs of the Selective State Space Model (Mamba) in computer vision, satellite remote sensing, and bioengineering, this invention leverages Mamba's linear-time processing capabilities to significantly reduce the model's parameter count. Moreover, this invention utilizes its selectivity mechanism to effectively capture the dynamic features of speech, enhancing the model's ability to distinguish between noisy and clean speech, and improving the model's ability to separate noisy speech.

[0007] The technical solution adopted by this invention to solve the above-mentioned technical problems is as follows: a noisy speech separation method based on a selective state-space model, which includes the following steps:

[0008] Step 1: Mixing speech signals The voice signal of the target speaker 1 The voice signal of target speaker 2 The signal consists of a noise signal n, which is input into an encoder based on a selective state space-based noisy speech separation model (SsMamba) for processing to obtain the encoder's output features. ;

[0009] Step 2: Output characteristics of the encoder The input is a U-shaped selective state-space speech separation module (UMamba). The UMamba module incorporates the U-shaped structure of the Mamba module into a traditional encoding / decoding temporal speech separation framework. It utilizes Mamba's selection mechanism to improve the model's ability to distinguish between noise and speech, thereby reducing the misclassification of signal and noise components in noisy speech separation tasks and improving speech separation performance. In UMamba, the encoding end obtains multi-scale features at different resolutions by stacking three dilated convolutions, while the separation end uses Mamba blocks to capture global and local features. The decoding end fuses local and global features by stacking three dual-branch one-dimensional convolutional fusion modules. The resulting speech features are the output of UMamba. ;

[0010] Step 3: After the initial separation, the separated speech features are... The input is fed into the Feature Refiner (FeaturePcs), which further optimizes the separation results, making the separated speech features clearer, thus obtaining the refined speech features. ;

[0011] Step 4: Extract the refined speech features The input is processed by the decoder of the noisy speech separation model (SsMamba) to recover the speech signal of the target speaker 1. The voice signal of target speaker 2 and noise signals ;

[0012] Step 5: Calculate the estimated difference between the speech signal and the target speech signal based on the loss function of noisy speech separation, so that the separated speech signal is close to the target speech signal.

[0013] Step 6: Evaluate the performance of the proposed noisy speech separation method based on a selective state-space model.

[0014] Further, step 1 includes the following steps:

[0015] Step 1-1: Input a time-domain speech signal with a feature dimension of 89600.

[0016] Steps 1-2: Construct an encoder for a noisy speech separation model based on selective state space. The encoder consists of one-dimensional convolutions. Specifically, the one-dimensional convolution has 1 input channel, 33 output channels, a kernel size of 64, a kernel stride of 16, and kernel padding of 32.

[0017] Steps 1-3: Initialize the parameters of the one-dimensional convolutional layer.

[0018] Further, step 2 includes the following steps:

[0019] Step 2-1: Construct a U-shaped selective state space speech separation module, consisting of an encoder, a separator, and a decoder;

[0020] Step 2-2: The encoding end consists of three dilated convolutions. Each dilated convolution has 512 input channels, 128 output channels, a kernel size of 5, a stride of 1, and a dilation factor of 2. The decoupling end mainly consists of a selective state-space model module (Mamba block). The Mamba block comprises a layer normalization, a Mamba layer, a Gelu activation function, and a linear layer. To prevent gradient explosion and vanishing gradients, the input and output are directly connected using residuals. The decoding end consists of three bi-branch one-dimensional convolutional feature fusion modules. Each one-dimensional convolution has 128 input channels, 512 output channels, a kernel size of 1, and a stride of 1. The first dilated convolution in the encoding end has an input speech feature size of 5610x512 and an output speech feature size of 2805x512. The second dilated convolution has an input speech feature size of 2805x512 and an output speech feature size of 1402x512. The third dilated convolution has an input speech feature size of 1402x512 and an output speech feature size of 701x512. The Mamba block has an input speech feature size of 701x512 and an output speech feature size of 701x512. At the decoder end, the first two-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 701x512 and an output speech feature size of 1402x512. The second two-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 1402x512 and an output speech feature size of 2805x512. The third two-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 2805x512 and an output speech feature size of 5610x512.

[0021] Steps 2-3: Multiply the estimated speaker mask, the noise mask, and the noisy speech features to obtain the separated speech feature size of 5610x99.

[0022] Furthermore, step 3 includes the following steps:

[0023] Step 3-1: Pad the input feature length and then perform a Short-Time Fourier Transform (STFT) on the signal. The STFT window uses 400 points (n_fft), the interval between adjacent STFT windows (hop_length) is 100, and the actual length of each window (win_length) is 400. After obtaining the phase and amplitude spectrum information of the signal, appropriate weights are assigned according to the importance of the spectral components to achieve further refinement.

[0024] Step 3-2: Convert the frequency domain signal to a time domain signal using inverse short-time Fourier transform (iSTFT). Perform inverse padding on the time domain signal to obtain a refined time-domain speech feature size of 5610x99.

[0025] Furthermore, step 4 includes the following steps:

[0026] Step 4-1: Input speech features with a size of 99x5610.

[0027] Step 4-2: Construct a decoder for a noisy speech separation model based on selective state space. The decoder consists of a one-dimensional transposed convolution. Specifically, the one-dimensional transposed convolution has 99 input channels, 3 output channels, a kernel size of 64, a kernel stride of 16, and kernel padding of 32.

[0028] Step 4-3: Obtain the estimated speech signal of target speaker 1. The estimated speech signal of the target speaker 2 and estimated noise signal .

[0029] Furthermore, step 5 includes the following steps:

[0030] Step 5-1: Construct the loss function for noisy speech separation

[0031] (1)

[0032] in It is the signal from the nth speaker. The corresponding signal is estimated by SsMamba. for The transpose of . for The L2 norm of the signal. N is the total number of output signals, set to 3. This method treats noise and the target speaker signal equally and estimates the noise independently.

[0033] Step 5-2: Scale-invariant source-to-noise ratio improvement (SI-SNRi) and signal-to-distortion ratio improvement (SDRi) are used to evaluate the speech intelligibility after separation. Para. is used to measure model size, and multiply-accumulate (MAC) is used to evaluate model efficiency. For SI-SNRi and SDRi, higher scores indicate better model separation performance. For Para. and MAC, lower values ​​indicate more efficient models.

[0034] Furthermore, step 6 includes the following steps:

[0035] Step 6-1: Compare the noisy speech separation method based on the selective state space model with mainstream noisy speech separation methods to verify the performance and efficiency of the present invention.

[0036] Step 6-2: Compare and analyze the role of each module in the noisy speech separation method based on the selective state space model.

[0037] Beneficial effects:

[0038] 1. This invention proposes a temporal speech separation framework based on Mamba encoding and decoding. This framework enables resource-constrained users to achieve high-performance noisy speech separation using a lightweight model. The model proposed in this invention has significantly fewer parameters than large models.

[0039] 2. This invention utilizes Mamba's selection mechanism to dynamically filter information from the input model, improving the model's ability to distinguish between noise and speech, thereby reducing misassignment problems in noisy speech separation tasks.

[0040] 3. This invention further refines the separated speech features, thereby improving the clarity of the recovered speech signal and enabling the time-domain speech separation model to take into account the frequency domain features of the speech signal, thus further improving the performance of the model. Attached Figure Description

[0041] Figure 1 This is a schematic diagram of the operation process of the present invention;

[0042] Figure 2 This is a schematic diagram of the decoder structure in UMamba in this invention;

[0043] Figure 3 This is a comparison of the spectrograms of the target speaker signal and the noise signal obtained by SsMamba and Sepformer respectively in an embodiment of the present invention. Detailed Implementation

[0044] The invention will now be described in further detail with reference to the accompanying drawings.

[0045] like Figure 1 As shown, this invention provides a noisy speech separation method based on a selective state-space model. In practical applications, acoustic features of the speech signal are generally extracted, the acoustic features are input into the separation model for training, and then the trained separation model is used for speech separation.

[0046] In noisy speech separation tasks, the most common problem is oversuppression caused by the method of enhancing before separating. Furthermore, current high-performing models in noisy speech separation tasks are mostly based on Transformer models with large parameter counts, which is unfriendly to users with limited resources. To address these issues, this invention proposes an encoding / decoding speech separation framework based on a selective state-space model. This framework mainly consists of one-dimensional convolution, UMamba, and one-dimensional transposed convolution.

[0047] One-dimensional convolution first transforms the input speech signal into a two-dimensional tensor. This two-dimensional tensor is then input into UMamba for speech separation. Finally, a one-dimensional transposed convolution restores the speech features to the original speech signal, achieving end-to-end speech separation. Specifically, UMamba contains three encoders, two Mamba blocks, and three decoders. The three encoders employ dilated convolutions with the same hyperparameter settings. By superimposing the three encoders, feature representations of different dimensions are obtained. The Mamba block is used to capture global and local features. The Mamba block consists of a layer normalization, a Mamba layer, a Gelu activation function, and a linear layer. All three decoders are constructed using a two-branch one-dimensional convolutional feature fusion module. This module allows the model to better fuse global and local information, contributing to better separation of mixed speech. The following is a detailed description of the embodiments of this invention:

[0048] Step 1: Input mixed signal Transform into two-dimensional features through one-dimensional convolution. , where N and L represent the number of channels and the length of the feature sequence, respectively.

[0049] Step 2: Previous research has shown that using multi-scale feature representations with different resolutions is more advantageous for speech separation tasks. Therefore, in UMamba, a one-dimensional dilated convolution is used as the encoder, and three encoders are stacked to obtain feature representations of different dimensions. Setting the dilation factor of the extended convolution to 2 allows the model to increase the receptive field while reducing the feature dimensionality. This compresses the dimensionality of the speech features layer by layer, from N×L to N×L / 8. Finally, the output features of all encoders are fed into the Mamba block. The above process can be represented by equations (2), (3), (4), and (5):

[0050] (2)

[0051] (3)

[0052] (4)

[0053] (5)

[0054] ( This represents the aggregated speech features input to the Mamba block.

[0055] To obtain global dependencies, a residual Mamba block is proposed. Specifically, the Mamba block consists of a layer normalization, a Mamba layer, a Gelu activation function, and finally a linear layer. For the input speech features... First, normalized speech features are obtained through layer normalization. Then, Mamba selectively processes the speech features, removing invalid information and retaining important information for a longer period. The output of the Mamba block can be represented as:

[0056] (6)

[0057] Then, the speech features output by the Mamba block are processed using the Gelu activation function and a linear layer. These features are combined with the input of the Mamba block via residual connections. The final output of the Mamba block is:

[0058] (7)

[0059] It is worth noting that the Mamba block has the property of preserving dimensionality, so it does not change the size of the input. Therefore, the dimension of the speech features output after processing by the Mamba block is still N×L / 8.

[0060] Mamba uses a hidden middle layer A one-dimensional function or sequence Mapped to The above process can be represented by formulas (8) and (9):

[0061] (8)

[0062] (9)

[0063] in , and These represent the input continuous time series, the output continuous time series, and the latent spatial series, respectively. , , These represent the state transition matrix, input projection matrix, and output projection matrix, respectively. Mamba is defined by four parameters: ,in This represents the time step. First, the continuous parameters need to be... Convert to discretization parameters Through a fixed formula and Implementation, including pairwise rules ( , The discretization rule is called the zero-order preservation rule, and the most commonly used discretization rule is zero-order preservation (ZOH).

[0064] The above process can be defined by formulas (10) and (11):

[0065] (10)

[0066] (11)

[0067] Where I is the identity matrix. After obtaining the discretization parameters, the discrete-time signal can be calculated using formulas (12) and (13):

[0068] (12)

[0069] (13)

[0070] To achieve parallel computing and improve computational efficiency, the above calculation process can be represented by formula (14):

[0071] (14)

[0072] In the above formula, the kernel of the state-space model is... With input sequence Convolution is performed to obtain the output sequence The calculation of the state-space model kernel can be expressed by formula (15):

[0073] (15)

[0074] When discrete parameters are predetermined, the kernel of the SSM can be pre-calculated, thereby improving computational efficiency. However, this fixed parameter configuration may cause the model to become overly reliant on predefined settings, thus reducing its adaptability to changes in incoming data.

[0075] To compensate for this inflexibility, Mamba integrates a selection mechanism that enables the model to be content-aware. This is achieved by adjusting the input based on the time interval. Difference dynamic update parameters This is achieved by (t represents different time steps), and the parameters are updated accordingly. , and This allows the system to respond more flexibly to changes in input information.

[0076] Following the Mamba block, the speech features are upsampled. This is done through nearest neighbor interpolation. After upsampling, these features are then compared with... , and Element-wise multiplication is performed. The results of these operations are then input to three encoders for processing. This process can be described by formulas (16), (17), and (18):

[0077] (16)

[0078] (17)

[0079] (18)

[0080] In the above formula, This represents nearest neighbor interpolation. This represents the element-wise product operation. (j = 1,2,3) represents local features. These local and global features from Mamba are sent to the decoder in the final stage.

[0081] The decoding end of the UMamba framework consists of three decoders stacked sequentially. For example... Figure 2 As shown, each decoder receives two types of input: local feature representations. (j = 1,2,3) and global feature representation The decoder processes these features as follows: Upsampling is performed using nearest neighbor interpolation. After upsampling, each branch passes through a one-dimensional convolutional layer, followed by a normalization layer. The left branch is then processed through a sigmoid activation function. The right branch directly yields The above process can be represented by formulas (19) and (20):

[0082] (19)

[0083] (20)

[0084] In the above formula, and These represent the functions of the one-dimensional convolutional layer and the normalized layer in the two branches, respectively. This represents the sigmoid activation function. The global feature input for the first decoder is:

[0085] (twenty one)

[0086] The decoder output can be represented as:

[0087] (twenty two)

[0088] To reconstruct abstract speech features and extract the most important components from the decoder's output, an additional Mamba block is used at the decoder end.

[0089] The final output of UMamba can be represented as:

[0090] (twenty three)

[0091] in This is the output of the last decoder in UMamba.

[0092] Step 3: After the initial separation, the separated speech features are... The input is fed into the Feature Refiner (FeaturePcs), which further optimizes the separation results, making the separated speech features clearer, and outputs the refined speech features. ;

[0093] Step 3-1: Pad the input feature length and then perform a Short-Time Fourier Transform (STFT) on the signal. The STFT window uses 400 points (n_fft), the interval between adjacent STFT windows (hop_length) is 100, and the actual length of each window (win_length) is 400. After obtaining the phase and amplitude spectrum information of the signal, appropriate weights are assigned according to the importance of the spectral components to achieve further refinement.

[0094] Step 3-2: Convert the frequency domain signal to a time domain signal using inverse short-time Fourier transform (iSTFT). Perform inverse padding on the time domain signal to obtain a refined time-domain speech feature size of 5610x99.

[0095] Step 4: Extract the refined speech features The input is processed by the decoder of the noisy speech separation model (SsMamba) to recover the speech signal of the target speaker 1. The voice signal of target speaker 2 and noise signals ;

[0096] Step 4-1: Input speech features with a size of 99x5610.

[0097] Step 4-2: Construct a decoder for a noisy speech separation model based on selective state space. The decoder consists of a one-dimensional transposed convolution. Specifically, the one-dimensional transposed convolution has 99 input channels, 3 output channels, a kernel size of 64, a kernel stride of 16, and kernel padding of 32.

[0098] Step 4-3: Obtain the estimated speech signal of target speaker 1. The estimated speech signal of the target speaker 2 and estimated noise signal .

[0099] Step 5: Calculate the estimated difference between the speech signal and the target speech signal based on the loss function of noisy speech separation, so that the separated speech signal is close to the target speech signal.

[0100] Constructing a loss function for noisy speech separation

[0101] (1)

[0102] in It is the signal from the nth speaker. The corresponding signal is estimated by SsMamba. This method treats noise and the target speaker signal equally and estimates the noise independently. In the experiment, N=3 was set, indicating that the mixed signal comes from two speaker voices and one noise source.

[0103] Step 6: Evaluate the performance of the proposed noisy speech separation method based on a selective state-space model.

[0104] First, a comparative experiment will be conducted between the noisy speech separation method based on the selective state space model and the mainstream noisy speech separation method to verify the performance and efficiency of the present invention. Then, an ablation experiment will be carried out on the noisy speech separation method based on the selective state space model to analyze the contribution rate of each module in the model to the model performance.

[0105] Example 1:

[0106] The simulation experiments and data include the following:

[0107] This invention was conducted on two noisy datasets: Libri2Mix and WHAM!. Libri2Mix is ​​a dataset for multi-speaker separation tasks. This experiment aimed to separate noisy speech from two speakers, so Libri2Mix was chosen. In the selected dataset, clean mixed speech was obtained from the LibriSpeech train-100 dataset, with loudness ranging from -25dB to -33dB. Noise samples were obtained from WHAM! noise and added to the mixed speech at loudness levels ranging from -38dB to -30dB, producing noisy mixed speech. Libri2Mix contains a total of 331 speakers and 62 hours of speech. WHAM! is a noisy version of WSJ0-2Mix. Clean mixed speech was obtained from the WSJ0-2Mix dataset, with loudness levels ranging from -25dB to -33dB. Noise samples were obtained from noise samples recorded in coffee shops, restaurants, and bars and added to the mixed speech at loudness levels ranging from -38dB to -30dB, producing noisy mixed speech. The size of the one-dimensional convolutional kernel in the encoder was set to 4ms, and the stride was set to 1ms. SsMamba and UMamba networks were built using the PyTorch deep learning framework, and GPU acceleration was used for network model training. In UMamba, the number of channels in the dilated convolutional layers and the number of hidden units in all linear layers of the encoder were both set to 512. The batch size was set to 1, and the number of worker threads was set to 6. For each experiment, training was performed for 500 epochs on an NVIDIA GeForce RTX 3090 using the adam optimizer with a learning rate of 0.001. Early stopping was used; the experiment was automatically stopped if the validation loss did not decrease within 10 epochs. The evaluation metrics used in this invention include scale-invariant signal-to-noise ratio improvement (SI-SNRi), signal-to-distortion ratio improvement (SDRi), model parameter count (Para.), and multiply-accumulate operation (MAC). For SI-SNRi and SDRi, higher scores indicate better model separation. For Para. and MAC, lower values ​​indicate more efficient models.

[0108] Experimental performance evaluation

[0109] First, the effectiveness of the state-space model-based noisy speech separation method is verified by comparing the proposed model with mainstream noisy speech models, including: Conv-tasnet, DPRNN, Sepformer, TDANet, SPP+IEDS, MSGT-TasNet (Dens), and HTCNAF. Conv-tasnet is a pioneering end-to-end speech separation network that uses a fully convolutional encoder-decoder architecture to perform separation directly in the temporal domain. DPRNN is an efficient neural network architecture for long-range sequence modeling, which divides the input sequence into smaller overlapping blocks and processes them using intra-block and inter-block recurrent neural networks. SepFormer is a speech separation model that uses a dual-channel Transformer architecture to effectively separate multiple speech signals directly in the temporal domain. TDANet is a lightweight temporal audio separation network employing an encoder-decoder structure. In SPP+IEDS, the authors propose a Separation Priority Pipeline (SPP) for single-channel speech separation in noisy environments, which prioritizes speech separation over enhancement and introduces Independent Encoder and Decoder (IEDS) to mitigate gradient conflicts. MSGT-TasNet (Dens) improves performance and reduces computational complexity by incorporating multi-scale group Transformers into Conv-TasNet. HTCNAF is a temporal target speaker extraction system that leverages robust speaker tracking through a hierarchical temporal convolutional network with attention fusion and hierarchical Res2block, achieving good performance on WSJ0-2mix and WHAM.

[0110] Table I first compares the speech separation performance of Conv-TasNet, DPRNN, Sepformer, TDANet, SPP+IEDS, and SsMamba models on the Libri2Mix dataset, and finds that the SsMamba model performs better in terms of the number of parameters and computational cost. Compared to Conv-TasNet, SI-SNRi improves by 4.5 dB, SDRi by 4.7 dB, reduces the number of parameters by 2.9M, and reduces MACs by 5.0 G / s; compared to DPRNN, SI-SNRi improves by 0.4 dB, SDRi by 0.4 dB, reduces the number of parameters by 12.6M, and reduces MACs by 79.1 G / s; compared to Sepformer, SI-SNRi improves by 0.3 dB, SDRi by 0.3 dB, reduces the number of parameters by 23.6M, and reduces MACs by 80.7 G / s; compared to TDANet, SI-SNRi improves by 0.9 dB, SDRi by 1.1 dB, reduces the number of parameters by 0.2M, and reduces MACs by 2.9 G / s; compared to SPP+IEDS, SI-SNRi improves by 1.2 dB, and SDRi improves by 1.2 dB. It can be seen that the SsMamba model achieves better speech separation performance than other models while maintaining fewer parameters and less computation.

[0111] Table I: Comparison of results for SI-SNRI, SDRI, PARA, and MACS on LIBRI2MIX.

[0112]

[0113] Secondly, Table II presents a performance comparison of the SsMamba model with Conv-TasNet, MSGT-TasNet, HTCNAF, DPRNN, and TDANet on the WHAM! dataset. Compared to Conv-TasNet, SI-SNRi improves by 3.9 dB and SDRi by 3.8 dB; compared to MSGT-TasNet, SI-SNRi improves by 1.6 dB and SDRi by 1.5 dB; compared to HTCNAF, SI-SNRi improves by 0.7 dB and SDRi by 0.7 dB; compared to DPRNN, SI-SNRi improves by 0.4 dB and SDRi by 0.5 dB; and compared to TDANet, SI-SNRi improves by 0.3 dB and SDRi by 0.3 dB. It can be seen that the SsMamba model exhibits excellent speech separation performance on various benchmark datasets.

[0114] Table II: Comparison of results for SI-SNRI, SDRI, PARA, and MACS on WHAM!

[0115]

[0116] Furthermore, Table III presents the ablation experiment results, specifically analyzing the contributions of each component in the SsMamba model. The changes in model performance were evaluated by removing different components (such as the cross-channel attention module and the dynamic convolution module). Compared to the complete SsMamba model, removing the cross-channel attention module resulted in a 1.2 dB decrease in SI-SNRi and a 1.3 dB decrease in SDRi; removing the dynamic convolution module resulted in a 0.9 dB decrease in SI-SNRi and a 1.0 dB decrease in SDRi; removing these two modules resulted in decreases of 2.1 dB and 2.3 dB in SI-SNRi and SDRi, respectively. This indicates that these modules play a crucial role in improving the speech separation performance of the SsMamba model. In addition, Table III shows that the SI-SNRi of the model without ANO is 3% lower than that of SsMamba, and the SI-SNRi of the model without FeaturePcs is 2% lower than that of SsMamba. This indicates that ANO has a more significant effect on improving SsMamba performance than FeaturePcs.

[0117] Table III: Ablation experiments of SsMamba on Libri2Mix. ANO represents adding noise output, which means that the proposed SsMamba treats background noise as an independent output and predicts it equally with other speakers' speech. w / o indicates removing a module.

[0118]

[0119] The above results demonstrate that this invention proposes a state-space model-based method for noisy speech separation, addressing the challenges of speech separation under noisy conditions. Unlike traditional methods, the proposed method treats background noise as an independent output, thereby improving separation accuracy. By abandoning SE preprocessing and utilizing Mamba's selective state-space concept, an efficient framework is developed while maintaining competitive performance. Experimental results on Libri2Mix and WHAM! prove the effectiveness of the proposed method, achieving performance comparable to existing methods, but with a significant reduction in parameters and computational cost.

[0120] The foregoing has shown and described the basic principles, main features, and advantages of this embodiment. Those skilled in the art should understand that this embodiment is not limited to the specific embodiments described above. The specific embodiments and descriptions in the specification are merely for further illustrating the principles of this embodiment. Various changes and modifications can be made to this embodiment without departing from the spirit and scope of this embodiment, and all such changes and modifications fall within the scope of this embodiment as claimed. The scope of protection of this embodiment is defined by the claims and their equivalents.

Claims

1. A noisy speech separation method based on a selective state-space model, characterized in that, Includes the following steps: Step 1: Mixing speech signals The voice signal of the target speaker 1 The voice signal of target speaker 2 The mixed speech signal, consisting of a noise signal n, is input into an encoder of the SsMamba noisy speech separation model based on selective state space. The encoder is composed of one-dimensional convolutions, where the input channel is 1, the output channel is 33, the kernel size is 64, the kernel stride is 16, and the kernel padding is 32, thus obtaining the output features of the encoder. ; Step 2: Encoder Output Characteristics The input is fed into the U-shaped selective state-space speech separation module UMamba. The UMamba module incorporates the U-shaped structure of the Mamba module. In UMamba, the encoding end obtains multi-scale features at different resolutions by stacking three dilated convolutions. The separation end uses Mamba blocks to capture global and local features. The decoding end fuses local and global features by stacking three dual-branch one-dimensional convolutional fusion modules to obtain the speech features output by UMamba. ; Step 2 includes the following steps: Step 2-1: Construct a U-shaped selective state space speech separation module, consisting of an encoder, a separator, and a decoder; Step 2-2: The encoding end consists of three dilated convolutions, each with 512 input channels, 128 output channels, a kernel size of 5, a stride of 1, and a dilation factor of 2. The separating end consists of a selective state-space model module, Mambablock, where Mamba... The block consists of a LayerNorm normalization layer, a Mamba layer, a Gelu activation function, and a linear layer, with residual connections between the input and output. The decoding end comprises three bi-branch one-dimensional convolutional feature fusion modules, where the one-dimensional convolution has 128 input channels, 512 output channels, a kernel size of 1, and a stride of 1. In the encoding end, the first dilated convolution has an input speech feature size of 5610x512 and an output speech feature size of 2805x512; the second dilated convolution has an input speech feature size of 2805x512 and an output speech feature size of 1402x512; and the third dilated convolution has an input speech feature size of 1402x512 and an output speech feature size of 701x512. Mamba... The input speech feature size of the block is 701x512, and the output speech feature size is 701x512. At the decoding end, the first dual-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 701x512 and an output speech feature size of 1402x512; the second dual-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 1402x512 and an output speech feature size of 2805x512; and the third dual-branch one-dimensional convolutional feature fusion module has two input speech feature sizes of 2805x512 and an output speech feature size of 5610x512. Steps 2-3: Multiply the estimated speaker mask and noise mask with the noisy speech features to obtain the separated speech feature size of 5610x99; Step 3: After the initial separation, the separated speech features are... The input is fed into the FeaturePcs feature refiner, which further optimizes the separation results, making the separated speech features clearer, and outputs the refined speech features. ; Step 3 includes the following steps: Step 3-1: Fill the input feature length, then perform a Short Time Fourier Transform (STFT) on the signal. The STFT window uses 400 points, the interval between adjacent STFT windows is 100, and the actual length of each window is 400. After obtaining the phase information and amplitude spectrum information of the signal, assign corresponding weights according to the importance of the spectral components to achieve further refinement. Step 3-2: Then use the inverse short-time Fourier transform (iSTFT) to convert the frequency domain signal into a time domain signal, and perform inverse filling on the time domain signal to obtain a refined time domain speech feature size of 5610x99. Step 4: Extract the refined speech features The input is processed by the decoder of the noisy speech separation model SsMamba. The decoder consists of a one-dimensional transposed convolution, with 99 input channels, 3 output channels, a kernel size of 64, a kernel stride of 16, and kernel padding of 32, to recover the speech signal of the target speaker 1. The voice signal of target speaker 2 and noise signals ; Step 5: Calculate the estimated difference between the speech signal and the target speech signal based on the loss function for noisy speech separation, so that the separated speech signal is close to the target speech signal. The loss function is... in It is the signal from the nth speaker. The corresponding signal is estimated by SsMamba. for transpose, for The L2 norm of the signal is set to 3, where N is the total number of output signals. This treats noise and the target speaker signal equally and estimates the noise independently. Step 6: Evaluate the performance of the proposed noisy speech separation method based on a selective state-space model.

2. The noisy speech separation method based on a selective state-space model according to claim 1, characterized in that, Step 1 includes the following steps: Step 1-1: Input a time-domain speech signal with a feature dimension of 89600; Step 1-2: Construct an encoder for a noisy speech separation model based on selective state space; Steps 1-3: Initialize the parameters of the one-dimensional convolutional layer.

3. The noisy speech separation method based on a selective state-space model according to claim 1, characterized in that, Step 4 includes the following steps: Step 4-1: Input speech features with a size of 99x5610; Step 4-2: Construct a decoder for a noisy speech separation model based on selective state space; Step 4-3: Obtain the estimated speech signal of the target speaker 1. The estimated speech signal of the target speaker 2 and estimated noise signal .

Images