Speech enhancement method and system based on selective state-space model mamba

Abstract

The application discloses a speech enhancement method and system based on a selective state space model Mamba, performs short-time Fourier transform on distorted speech to obtain a time-frequency domain signal, splices the amplitude, real part and imaginary part of the time-frequency domain signal into a signal of three channels as an input signal, expands the input signal from three channels to multiple channels through a first convolution block of a generator network encoder, extracts different resolution features and increases a receptive field through a dilated dense convolution layer of the encoder, reduces the frequency dimension of the signal to 1 / 2 through the last convolution block of the encoder to reduce the calculation complexity, simulates the forward and backward dependence of the speech signal under different resolutions through N TF-mamba blocks to capture the local and global features of the long sequence speech signal, realizes feature enhancement, and then obtains enhanced speech after denoising and dereverberation by predicting the enhanced signal amplitude and phase through a complex decoder and a mask decoder respectively.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of speech signal processing, and is mainly related to a speech enhancement method and system based on a selective state space model Mamba in the direction of speech de-noise and de-reverberation enhancement. BACKGROUND

[0002] In real-life speech applications, speech quality and intelligibility depend on the performance of underlying speech enhancement (SE) systems, such as speech de-noise, de-reverberation, and acoustic echo cancellation. Therefore, the SE framework is an indispensable component in modern automatic speech recognition, telecommunication systems, and hearing aid devices. More and more research is constantly trying to break through the performance limit of the current SE system, and these methods mostly utilize the latest progress of deep learning technology and more and more available speech datasets.

[0003] Currently, many SE models utilize transformers and conformers based on self-attention mechanisms to capture long-term dependencies in waveforms or spectrograms. In speech tasks, training usually includes the entire speech signal, which produces a long context, and the quadratic complexity of the self-attention mechanism with respect to the sequence length poses a huge challenge for deploying these models in limited computing resources. Therefore, there is an urgent need for a more efficient and accurate method to improve the efficiency of SE tasks. SUMMARY

[0004] The present application is exactly aimed at the problem of high computational complexity of the self-attention mechanism in the prior art, and proposes a speech enhancement method and system based on a selective state space model Mamba. First, the distorted speech is subjected to short-time Fourier transform to obtain a time-frequency domain signal, the amplitude, real part, and imaginary part of the time-frequency domain signal are spliced into a 3-channel signal, and then the spliced signal is expanded from 3 channels to multiple channels through the first convolutional block of the encoder of the generator network, different resolution features are extracted and the receptive field is increased through the dilated dense convolutional layer of the encoder, the frequency dimension of the signal is reduced to 1 / 2 through the last convolutional block of the encoder to reduce the computational complexity, the expanded signal is subjected to feature enhancement through N TF-mamba blocks, the forward and backward dependencies of the speech signal at different resolutions are simulated, the local and global features of the long sequence speech are captured, and the enhanced speech after de-noise and de-reverberation is obtained by predicting the enhanced signal amplitude and phase through the complex decoder and the mask decoder, respectively. A discriminator based on a speech quality evaluation index is also used to discriminate between the enhanced speech and the clean speech, and the gradients of the generator and the discriminator network are updated, and multiple rounds of network model training are repeated until the model converges, so that the enhanced speech is close to the clean speech in both subjective auditory and objective evaluation indicators. The present application improves the network training efficiency while ensuring the indicators of the enhanced speech signal.

[0005] In order to achieve the above object, the technical scheme adopted by the present application is: a speech enhancement method based on a selective state space model Mamba, at least comprising the following steps:

[0006] S1, data preprocessing: pre-processing the input distorted speech signal , obtaining a time-frequency domain signal through short-time Fourier transform, L representing the length of the speech time domain; the amplitude, real part and imaginary part of the time-frequency domain signal are spliced into an input signal , wherein B represents the batch size, T is the number of frames, F is the frequency point, and 3 is the number of channels;

[0007] S2, signal expansion: the time-frequency domain signal obtained in step S1 is dimensionally expanded by an encoder from 3 channels to C channels; the encoder comprises two convolution blocks and a dilated dense convolution layer, wherein the first convolution block is used to expand the channel dimension of the input signal from 3 to C, forming a signal The expanded signal is extracted through the dilated dense convolution layer to obtain different resolution features and increase the receptive field to obtain The second convolution block is used to downsample the frequency F of the signal Y" stft after the dilated dense convolution layer to obtain The expanded signal is obtained as

[0008] S3, feature enhancement: through N TF-mamba blocks, the signal expanded in step S2 is subjected to feature enhancement to simulate the forward and backward dependence relationship of the speech signal under different resolutions, and a signal subjected to feature enhancement is obtained The TF-mamba block is composed of a Time Mamba block and a Frequency Mamba block, and each Mamba block adopts a bidirectional SSM mode, and the input signal is processed in parallel through a forward Mamba and a backward Mamba;

[0009] S4, speech signal enhancement: the signal of step S3 is subjected to amplitude and phase processing through a mask decoder and a complex decoder respectively, to obtain a predicted amplitude and a predicted phase, and a predicted spectrum signal is obtained after the predicted amplitude and the predicted phase are combined; after inverse power compression and inverse short-time Fourier transform ISTFT are performed on the predicted spectrum signal, an enhanced speech signal is obtained;

[0010] S5, enhanced speech discrimination: the enhanced speech signal obtained in step S4 and the clean speech signal are discriminated through a measurement discriminator, and the gradient of the generator and the discriminator is updated after the discriminator is trained; the input signal Y inThe generator updated by the input update gradient is repeated steps S2-S4 to obtain new enhanced speech, the enhanced speech is discriminated by the discriminator updated by the gradient, and the update of the generator network and the discriminator is repeated to realize speech enhancement.

[0011] As an improvement of the application, the specific method for obtaining the video domain signal in step S1 is: through STFT operation, the waveform of the input distorted speech signal is converted into a complex spectrum diagram Y Wherein T and F represent the number of frames and frequency points respectively, and the compressed spectrum diagram Y is obtained by power compression:

[0012]

[0013] Wherein c represents the power compression index, Y m , Y p , Y r and Y i are the amplitude, phase, real part and imaginary part of the spectrum diagram Y; Y r , Y i and Y m are connected as the input signal B represents the batch size, T represents the number of frames, and F represents the frequency point.

[0014] As an improvement of the application, the encoder of step S2 is composed of two convolutional blocks and a dilated DenseNet layer, each convolutional block includes a two-dimensional convolutional layer, an instance normalization layer and a PReLU activation function layer. The first convolutional block expands the input Y stft into C channels, the dilated DenseNet layer contains four convolutional blocks with dense residual connection, and the expansion factors of the four convolutional blocks are {1, 2, 4, 8} to effectively increase the receptive field; the second convolutional block is responsible for reducing the frequency dimension to to reduce the complexity, and the expanded signal Y

[0015] As another improvement of the application, in the encoder of step S2, the convolution kernel size of the first convolutional block is (1, 1), the step size is (1, 1); the convolution kernel size of the second convolutional block is (1, 3), the step size is (1, 2), and the padding is (0, 1).

[0016] As another improvement of the application, in step S3, N TF-Mamba blocks are used to obtain the global and local information of the speech signal, each TF-Mamba is composed of a Time Mamba block and a Frequency Mamba block, and the Time Mamba block and the Frequency Mamba block have the same structure, and the specific steps are:​

[0017] S31: The input signal obtained in step S2 Reshape it into The local and global information is processed along the time domain dimension T by the Time Mamba block and then combined with Y tm Obtained through skip connections

[0018] Y t =TimeMamba(Y tm )+Y tm

[0019] S32: The signal obtained from S31 Reshape it into The local and global information is processed along the frequency domain dimension F by the Frequency Mamba block and then combined with Y fm Obtained through skip connections

[0020] Y f =FrequencyMamba(Y fm )+Y fm

[0021] S33: Reshape into Y mamba Enter the next TF-Mamba block, and a total of N TF-Mamba blocks need to be entered, that is, steps S31 and S32 need to be repeated N times, and finally the feature-enhanced signal is obtained.

[0022] As another improvement of the present invention, step S4 specifically includes the following steps:

[0023] S41: In the mask decoder, a convolution block is used to The number of channels is compressed, and then the final amplitude is predicted through another convolutional layer with PReLU activation

[0024] S42: In the complex decoder, the real and imaginary parts of the final enhanced signal are predicted by the convolution block

[0025] S43: The final amplitude predicted in step S41 Sum signal phase Y p The enhanced frequency domain signal is obtained by combining;

[0026] S44: The enhanced frequency domain signal obtained in step S43 is compared with the complex decoder prediction output in step S42. Add them sequentially to get the final enhanced frequency domain signal:

[0027]

[0028] S45: In the complex spectrum Perform inverse power compression and inverse short-time Fourier transform ISTFT to obtain the time domain signal

[0029] As another improvement of the present invention, the metric discriminator of step S5 is composed of four convolution blocks, each convolution block is composed of a two-dimensional convolution layer, an instance normalization and a PReLU activation layer, and the convolution block is followed by a global average pooling layer, two feedforward layers and a sigmoid activation layer.

[0030] As a further improvement of the present invention, the loss function of the generator in step S4 is is the amplitude loss function in the time-frequency domain and complex loss function

[0031]

[0032]

[0033] Among them, α represents the weight, X m represents the amplitude of clean speech, X r and X i Represents the real and imaginary parts of clean speech in the time-frequency domain;

[0034] The generator loss is the association generator loss and time domain loss Combination of:

[0035]

[0036] in, represents the enhanced speech signal, X represents the clean speech signal, and ||·||1 represents norm, ||·||2 represents Norm, γ1, γ2, γ3 represent the corresponding loss weights.

[0037] In order to achieve the above object, the present invention also adopts a technical solution: a speech enhancement system based on the selective state space model Mamba, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0038] Compared with the prior art, the present application has the beneficial effects: the present application provides a speech enhancement method and system based on selective state space model Mamba, which combines the amplitude and phase information of the time-frequency domain speech signal as input data for processing, estimates the amplitude and phase of the enhanced signal through the generator model, effectively improves the efficiency of the generator model in processing complex long sequence data in combination with the TF-Mamba module, enhances the multi-scale information capturing capability, and uses the speech perceptual quality evaluation score PESQ as the discriminator index, optimizes the performance of the speech enhancement model, and makes the enhanced speech more close to the clean speech in perception. BRIEF DESCRIPTION OF DRAWINGS

[0039] Figure 1 is a step flow chart of the method of the present application;

[0040] Figure 2 is a data preprocessing flowchart in the method of the present application;

[0041] Figure 3 is a structure diagram of the speech enhancement network generator in the method of the present application;

[0042] Figure 4 is a TF-Mamba block structure diagram in the method of the present application;

[0043] Figure 5 is a structure diagram of the speech enhancement network discriminator in the method of the present application;

[0044] Figure 6 is a clean speech spectrogram in the test set in the test example of the present application;

[0045] Figure 7 is a noisy reverberation speech spectrogram in the test example of the present application;

[0046] Figure 8 is an enhanced speech spectrogram in the test example of the present application. DETAILED DESCRIPTION

[0047] The present application will be further illustrated below in conjunction with the drawings and specific embodiments, and it should be understood that the following specific embodiments are only used to illustrate the present application and not to limit the scope of the present application.

[0048] Example 1

[0049] As shown in Figure 1 , the present application provides a speech enhancement network method based on selective state space model Mamba, comprising the following steps:

[0050] Step S1: pre-process the input distorted speech signal Y, obtain the frequency spectrum signal through short-time Fourier transform STFT, perform power compression on the signal, and stack the amplitude and phase information as the network input signal

[0051] The distorted speech signal is considered as the speech signal superimposed by the direct signal and reverberation signal and background noise. The background noise is irrelevant to the direct signal, so the denoising task is considered as the speech separation task; the reverberation signal is the attenuated and delayed signal of the direct signal reflected by the walls, ceiling, etc. in the closed acoustic environment, which can be obtained by convolving the direct signal and the room impulse response, and the reverberation signal is relevant to the direct signal. The distorted speech signal can be expressed as:

[0052] y(t) = x(t) * h(t) + n(t)

[0053] where y(t) is the distorted speech, x(t) is the desired clean speech, h(t) is the background noise, and * represents the convolution operation.

[0054] For the distorted speech waveform The STFT operation first converts the waveform into a complex spectrogram where T and F represent the time dimension and the frequency dimension, respectively. Then, by power compression, the importance of smaller sounds is equal to that of larger sounds, which is closer to human perception of sound, and a compressed spectrogram Y is obtained:

[0055]

[0056] c represents the power compression exponent, which ranges from 0 to 1, and c = 0.3 in this embodiment. Y m , Y p , Y r and Y i are the amplitude, phase, real part and imaginary part of the spectrogram Y. Y r and Y i and Y m are connected as input signals B represents the batch size, T represents the number of frames, and F represents the frequency point, and the process is as follows Figure 2 The data preprocessing flowchart is shown in

[0057] Step S2: The signal of step S1 is dimensionally expanded by an encoder, which is composed of one dilated dense convolution layer and two convolution blocks, and the input signal is expanded by the encoder to

[0058] The encoder structure is shown in the encoder part of Figure 3 The encoder is composed of two convolution blocks and one dilated dense convolution layer (Dilated DenseNet). Each convolution block includes a two-dimensional convolution layer, an instance normalization layer and a PReLU activation function layer. The first convolution block expands the input The expansion is C channels, and the dilated dense convolution layer contains four convolution blocks with dense residual connection. The expansion factors of the four convolution blocks are {1, 2, 4, 8} to effectively increase the receptive field. The second convolution block is responsible for reducing the frequency dimension to to reduce complexity.

[0059] In this embodiment, the input Firstly, pass through a two-dimensional convolution block, the input channel number of the convolution block is 3, the output channel number is C, the convolution kernel size is designed as (1, 1), the step is (1, 1), and instance normalization InstanceNorm and PReLU activation layer are adopted to obtain

[0060] Then Y' stft Enter the dilated DenseNet with a depth of 4, that is, composed of four two-dimensional convolution blocks, and the expansion factors of the four convolution blocks are {1, 2, 4, 8}. The dilated convolution is combined with the dense connection to gradually expand the receptive field, and the output is Finally, Y" stft Through the down-sampling convolution layer, the input channel number and the output channel number of the two-dimensional convolution block are both 64, the convolution kernel size is designed as (1, 3), the step is (1, 2), the padding is (0, 1), and instance normalization InstanceNorm and PReLU activation layer are adopted. The function of the second convolution block is to reduce Y" stft frequency F to to reduce complexity, and obtain

[0061] Step S3: pass the signal of step S2 through N TF-mamba blocks to enhance its time domain and frequency domain characteristics.

[0062] The structure of the TF-mamba block is as shown in Figure 4 The TF-mamba block is composed of a Time Mamba block and a Frequency Mamba block, and the Time Mamba block and the Frequency Mamba block have the same structure. The specific steps are as follows:

[0063] S31: reshape the input signal obtained in step S2 to Through the Time Mamba block, local and global information along the time domain dimension T is processed, and then Y tm is obtained through a jump connection

[0064] Y t = TimeMamba(Y tm ) + Y tm

[0065] S32: The signal obtained in S31 is reshaped as The local and global information is processed along the frequency domain dimension F by the Frequency Mamba block, and then combined with Y fm

[0066] Y f = FrequencyMamba(Y fm ) + Y fm

[0067] S33: The signal obtained in S32 is reshaped as Y mamba

[0068] The structure of the TF-Mamba block is two mamba blocks, Time Mamba and Frequency Mamba, which perform forward and backward scanning, respectively, to integrate local and global information. The steps for running the signal in the mamba block are as follows:

[0069] a. For the example input data First, forward scanning is performed by the forward mamba block, then the RMSNorm activation function is used, and then combined with Y eg Y f is obtained by jump connection:

[0070] Y f = RMS(FMamba(Y eg )) + Y eg

[0071] b. For the example input data Y flip is obtained by reversing it, then backward scanning is performed by the backward mamba block, then the RMSNorm activation function is used, and then combined with Y flip Y b is obtained by jump connection:

[0072] Y b = RMS(FMamba(Y flip )) + Y flip

[0073] c. After Y f and Y b are spliced along the channel dimension C, transposed convolution is performed to obtain the output Y​​​​out :

[0074] Y out = TransConv(Concat(Y f ,Y b ))

[0075] where RMS, FMamba, BMamba, Flip, Concat represent forward output, backward output, RMS normalization, forward Mamba, backward Mamba, flip and concatenate operation respectively;

[0076] Step S4: the signal of step S3 is input into the decoder to obtain the predicted spectrum signal. The predicted spectrum signal is obtained by combining the predicted amplitude and phase obtained by respectively processing the amplitude and phase through two independent decoders.

[0077] As shown in the decoder part in Figure 3 , the mask decoder and the complex decoder both contain dilated DenseNet and two-dimensional convolution blocks, the dilated DenseNet structure is the same as that in the encoder, After dilated DenseNet, the frequency F' is up-sampled to F using a sub-pixel convolution layer. In the mask decoder, a convolution block is used to compress the channel number to 1, and then another convolution layer with PReLU activation is used to predict the final amplitude In the complex decoder, the real part and the imaginary part of the final enhanced signal are predicted by a convolution block

[0078] First, the predicted enhanced signal amplitude and the signal phase Y p are combined to obtain the enhanced frequency domain signal, and then the complex decoder prediction output is sequentially added to obtain the final enhanced frequency domain signal:

[0079]

[0080] Then, the complex spectrum graph is subjected to inverse power compression, and inverse short-time Fourier transform ISTFT is performed to obtain the time domain signal

[0081] Step S5: the enhanced signal obtained in step S4 is input into the discriminator to estimate the PESQ score of the enhanced signal, and the generator and the discriminator are updated to make the model converge faster.

[0082] As shown inFigure 5 The clean speech signal X and the predicted enhanced signal are input to the metric discriminator, which is composed of four convolutional blocks, each of which is composed of a two-dimensional convolutional layer and an instance normalization and PReLU activation layer, followed by a global average pooling layer and two feedforward layers and a sigmoid activation layer. The two-dimensional convolutional layer has a kernel size of (4, 4), a stride of (2, 2), and a padding of (1, 1). The sigmoid activation function is a nonlinear activation function, and the purpose is to map the discrimination result to between (0, 1).

[0083] As Figure 5 shown, the discriminator is trained using the perceptual evaluation of speech quality score PESQ, using the amplitude of the clean speech as the reference and the degraded input to estimate the maximum normalized PESQ score. The generator and the discriminator are trained separately at each epoch, and the gradient is updated to generate enhanced speech similar to the clean speech.

[0084] The loss function used by the generator is the amplitude loss function in the time-frequency domain and the complex loss function

[0085]

[0086] where α represents the weight, X m represents the amplitude of the clean speech, X r and X i represent the real and imaginary parts of the clean speech in the time-frequency domain.

[0087] The adversarial training loss of the entire network model is the discriminator loss and the associated generator loss

[0088]

[0089] where Q PESQ is the normalized PESQ score.

[0090] The final generator loss is the combination of and the time domain loss

[0091]

[0092] where, X represents the enhanced speech signal, X represents the clean speech signal, ||·||1 represents the L1 norm, and ||·||2 represents ​The norms, γ1, γ2, γ3 represent the corresponding loss weights.

[0093] Test Example

[0094] The simulation environment for the speech enhancement method experiment based on the selective state space model Mamba is: GPU NVIDIA RTX 4070TI SUPER, CPU I5 13600KF, Ubuntu 20.04LTS, CUDA 12.0, Pytorch 2.0.1.

[0095] In this experiment, the public dataset VCTK is selected as the clean speech dataset, which contains 108 hours of 44-hour recordings with various English accents, with a sampling rate of 22.05KHz. First, the dataset speech is downsampled to 16KHz, then convolved with the RIR_NOISES dataset and added with 5-30dB random noise to obtain the noisy reverberation dataset. The sentences in the training set are cut into 2-second segments, and the sentences in the test set are not cut. The STFT step is 400, the Hamming window is 25ms, and the frame shift is 6.25ms, i.e. 75% overlap. The number of TF-Mamba blocks in the generator is set to 4, the batch size B is 2, and the number of channels C is 64. The number of channels in the metric discriminator is set to {16, 32, 64, 128}. The training times epoch is 100, and during the training phase, both the generator and the discriminator are trained using the AdamW optimizer, the generator learning rate is set to 0.0004, the discriminator learning rate is set to 0.001, and the learning rate scheduler has a decay coefficient of 0.5 every 12 epochs. The generator loss weight is set to {γ1=1, γ2=0.01, γ3=1}.

[0096] Figure 6-8 are the clean speech spectrograms, noisy reverberation speech spectrograms and enhanced speech spectrograms in the test set in this test example, respectively. Figure 6 The clean speech spectrogram has clear harmonic structure, obvious formant, and energy concentrated on the main frequency components of the speech, with less background noise and clean non-speech area. While Figure 7 The noisy reverberation speech is obtained by superimposing clean speech, reverberation and background noise, and its harmonic structure is blurred, the formant is not obvious, the energy distribution is dispersed, and the background noise is increased. The non-speech area has energy distribution, and the time tailing phenomenon is obvious, with the speech component trailing on the time axis. Figure 8 The enhanced speech spectrogram can be seen that the time tailing phenomenon is obviously weakened, the background noise is also reduced, and the overall is closer to the clean speech spectrogram.

[0097] The control model of the experiment is MetricGAN+, CMGAN and TSTNN. MetricGAN is a speech enhancement network that connects the metric discriminator with the evaluation index for training, MetricGAN+ is a speech enhancement network that optimizes the sigmoid activation function based on MetricGAN, CMGAN is a speech enhancement network that combines MetricGAN with a conformer structure based on a self-attention mechanism, and TSTNN is a speech enhancement network that enhances speech in the time domain through a transformer structure.

[0098] The experimental evaluation indexes are perceptual evaluation of speech quality PESQ, speech intelligibility evaluation STOI, background noise intrusion CBAK, and speech signal distortion CSIG. PESQ simulates human auditory perception, calculates the difference between enhanced speech and original clean speech, and the index range is [-0.5, 4.5], the higher the value, the better the speech quality; STOI is used to measure the intelligibility of speech, the index range is [0, 1], the higher the value, the better the speech intelligibility; CBAK is mainly used to measure the degree of residual background noise, that is, how much noise interference is in the enhanced speech, the range is [1, 5], the higher the score, the less the background noise, that is, the better the enhancement effect; CSIG evaluates whether the enhanced speech signal retains the original speech characteristics, that is, whether distortion is introduced in the enhancement process, the higher the score, the less the distortion of the enhanced speech, the better the speech quality. Table 1 below is the evaluation index table of speech enhancement in the test example:

[0099] Table 1

[0100] PESQ STOI CBAK CSIG Reverb Speech 1.981 0.836 2.265 3.162 CMGAN 3.237 0.904 2.515 3.504 MetricGAN+ 3.216 0.892 2.477 3.36 TSTNN 3.158 0.887 2.51 3.375 TF-Mamba 3.34 0.925 2.615 3.471

[0101] As can be seen from Table 1, the PSEQ, STOI and CBAK indexes of the speech enhancement network proposed by the method are better than those of the comparison network model, and the CSIG index is better than that of MetricGAN+ and TSTNN, indicating that the speech enhancement effect of the method is better than that of the current mainstream speech enhancement network. Compared with CMGAN, the average training time of the model per round is reduced from 48 minutes to 31 minutes under the same batch size, and the training speed is improved by about 34.8%.

[0102] In summary, the TF-Mamba method proposed in the application can improve the training speed of the model, compared with the transformer and conformer architectures, the training speed has been improved to some extent, and the evaluation indexes are better than those of the speech enhancement network with the transformer architecture, and the method realizes more advanced SE with lower computational complexity.

[0103] It should be noted that the above content only illustrates the technical idea of the present application, and cannot limit the protection scope of the present application. For ordinary skilled in the art, without departing from the principle of the present application, a number of improvements and refinements can be made, which fall within the protection scope of the claims of the present application.

Claims

1. A speech enhancement method based on the selective state space model Mamba, characterized in that: At least the following steps are included: S1. Data preprocessing: distorted speech signal input Perform preprocessing and obtain the time-frequency domain signal through short-time Fourier transform, where L represents the time domain length of the speech; concatenate the amplitude, real part and imaginary part of the time-frequency domain signal into the input signal Where B represents the batch size, T is the number of frames, F is the frequency, and 3 is the number of channels; S2, signal expansion: The time-frequency domain signal obtained by the step S1 is dimensionally expanded by the encoder, from 3 channels to C channels; the encoder includes two convolution blocks and an expansion dense convolution layer, wherein the first convolution block is used to convert the input signal The channel dimension is expanded from 3 to C, forming a signal The expanded signal is obtained by expanding the dense convolution layer to extract features of different resolutions and increase the receptive field The second convolution block is used to convert the signal Y″ after passing through the dilated dense convolution layer stft The frequency F is downsampled to Get the expanded signal S3, feature enhancement: Through N TF-mamba blocks, the feature of the signal expanded in step S2 is enhanced to simulate the forward and backward dependencies of speech signals at different resolutions, and the feature-enhanced signal is obtained. The TF-mamba block consists of a Time Mamba block and a Frequency Mamba block. Each Mamba block adopts a bidirectional SSM mode, and processes the input signal in parallel through the forward Mamba and the backward Mamba. S4, speech signal enhancement: The signal of step S3 is processed by a mask decoder and a complex decoder for amplitude and phase respectively to obtain predicted amplitude and phase, which are combined to obtain a predicted spectrum signal, and the predicted spectrum signal is subjected to inverse power compression and inverse short-time Fourier transform (ISTFT) to obtain an enhanced speech signal; S5, enhanced speech discrimination: The enhanced speech signal obtained in step S4 is discriminated against the clean speech signal by a metric discriminator, and the gradients of the generator and the discriminator are updated after training the discriminator; the input signal Y of step S2 is in Input the generator after updating the gradient, repeat steps S2-S4 to obtain the new enhanced speech, perform enhanced speech discrimination through the discriminator after gradient update, repeat steps S2-S5 to update the generator network and discriminator to achieve speech enhancement.

2. The method for speech enhancement based on the selective state space model Mamba according to claim 1, wherein: The method for obtaining the time-frequency domain signal in step S1 is as follows: by performing STFT operation, the time-domain waveform of the input distorted speech signal is converted into Convert to time-frequency domain signal Where L represents the length of the speech time domain, T and F represent the number of frames and frequency points respectively, and the compressed spectrum Y is obtained by power compression: Where c represents the power compression index, Y m , Y p , Y r and Y i is the amplitude, phase, real component and imaginary component of the spectrum graph Y, ω(τ-t) is the window function, t is the time variable, f is the frequency variable, j is the imaginary unit of the complex number; Y r 、Y i and Y m Connect as input signal B represents the batch size, T represents the number of frames, and F represents the frequency.

3. The method for speech enhancement based on the selective state space model Mamba according to claim 1, wherein: Each convolution block of the encoder in step S2 includes a two-dimensional convolution layer, an instance normalization layer and a PReLU activation function layer; the dilated dense convolution layer contains four convolution blocks with dense residual connections, and the dilation factors of the four convolution blocks are {1, 2, 4, 8}.

4. The method for speech enhancement based on the selective state space model Mamba according to claim 3, wherein: In the encoder of step S2, the convolution kernel size of the first convolution block is (1, 1), and the step size is (1, 1); the convolution kernel size of the second convolution block is (1, 3), the step size is (1, 2), and the padding is (0, 1).

5. The method for speech enhancement based on the selective state space model Mamba according to claim 1, wherein: The specific steps of the signal feature enhancement in step S3 are: S31: The input signal obtained in step S2 Reshape it into The local and global information is processed along the time domain dimension T by the TimeMamba block and then combined with Y tm Obtained through skip connections AND t =TimeMamba(Y tm )+Y tm S32: The signal obtained from S31 Reshape it into The local and global information is processed along the frequency domain dimension F by the FrequencyMamba block and then combined with Y fm Obtained through skip connections AND f =FrequencyMamba(Y fm )+Y fm S33: Reshape into Y mamba Enter the next TF-Mamba block, and a total of N TF-Mamba blocks need to be entered, that is, steps S31 and S32 need to be repeated N times, and finally the feature-enhanced signal is obtained.

6. The method for speech enhancement based on the selective state space model Mamba according to claim 5, characterized in that: The TF-Mamba block structure consists of two mamba blocks: Time Mamba and Frequency Mamba, which perform forward and backward scanning respectively, integrating local and global information. The signal runs in the mamba block as follows: a. For input data First, forward scan is performed through the forward mamba block, and then the RMSNorm activation function is used and then compared with Y eg Get Y through skip connection f : AND f =RMS(FMamba(Y eg ))+Y eg b. For input data Flip it to get Y flip , and then scan backward through the backward mamba block, and then use the RMSNorm activation function and then compare it with Y flip通 Get Y through skip connection b : AND b =RMS(FMamba(Y flip ))+Y flip c. Change Y f and Y b After splicing along the channel dimension C, transposed convolution is performed to obtain the output Y out : AND out =TransConv(Concat(And f ,AND b )) Among them, RMS, FMamba, BMamba, Flip, and Concat represent forward output, backward output, RMS normalization, forward Mamba, backward Mamba, flip, and concatenation operations, respectively.

7. The method for speech enhancement based on the selective state space model Mamba according to claim 1, wherein: The step S4 specifically includes the following steps: S41: In the mask decoder, a convolution block is used to The number of channels is compressed, and then the final amplitude is predicted through another convolutional layer with PReLU activation S42: In the complex decoder, the real and imaginary parts of the final enhanced signal are predicted by the convolution block S43: The final amplitude predicted in step S41 Sum signal phase Y p The enhanced frequency domain signal is obtained by combining; S44: The enhanced frequency domain signal obtained in step S43 is compared with the complex decoder prediction output in step S42. Add them sequentially to get the final enhanced frequency domain signal: S45: In the complex spectrum Perform inverse power compression and inverse short-time Fourier transform (ISTFT) to obtain the time domain signal 8. The method for speech enhancement based on the selective state space model Mamba according to claim 7, characterized in that: The metric discriminator of step S5 is composed of four convolution blocks, each of which is composed of a two-dimensional convolution layer, an instance normalization layer, and a PReLU activation layer. The convolution block is followed by a global average pooling layer, two feedforward layers, and a sigmoid activation layer.

9. The method for speech enhancement based on the selective state space model Mamba according to claim 8, characterized in that: The loss function of the generator in step S5 is the amplitude loss function in the time-frequency domain and complex loss function Among them, α represents the weight, X m represents the amplitude of clean speech, X r and X i Represents the real and imaginary parts of clean speech in the time-frequency domain; Generator loss is the association generator loss and time domain loss Combination of: in, represents the enhanced speech signal, X represents the clean speech signal, and ||·||1 represents l 1 norm, ||·||2 represents l 2 Norm, γ1, γ2, γ3 represent the corresponding loss weights.

10. A speech enhancement system based on the selective state-space model Mamba, comprising a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images