A single-channel speech enhancement method based on multi-attention mechanism

Abstract

The present invention relates to a single-channel speech enhancement method based on a multi-attention mechanism, and belongs to the technical field of audio signal processing. The present invention introduces a complex Conformer into a complex U-Net network to model the correlation between speech amplitude and phase, uses a three-dimensional attention mechanism to construct richer features to enhance the representation ability of the convolutional layer, and fuses speech detail features and deep features through a gated attention mechanism. The method can improve speech quality and intelligibility, and can be used for voice communication in noisy environments, command control, and the pre-processing part of speech-related tasks. Experimental results on public datasets show that the proposed method achieves evaluation results of 3.09, 4.28, 3.47, 3.72, and 95.07 on five objective evaluation indicators: PESQ, CSIG, CBAK, COVL, and STOI, respectively, and can effectively reduce noise and improve speech quality and intelligibility.

Description

Technical Field

[0001] The present invention relates to a single-channel speech enhancement method based on a multi-attention mechanism, and belongs to the technical field of audio signal processing. Background Art

[0002] Speech, the acoustic representation of language, is a crucial medium for human communication and information transmission, offering both convenience and efficiency. Removing various noises from noisy speech in complex acoustic environments, thereby improving speech quality and intelligibility, has attracted widespread attention and has numerous applications in voice communication, hearing aids, voice command control, and speech recognition. In interpersonal communication, poor speech quality can reduce the efficiency of information transmission and even lead to errors. For example, in remote meetings, interference such as noise can cause participants to expend more energy to hear what others are saying, reducing meeting efficiency and, in extreme cases, even compromising the scientific and rational nature of decision-making. In human-computer interaction, poor speech quality can significantly reduce the performance of speech processing systems, ultimately severely impacting device usability. For example, in smart home environments, commands to devices such as televisions, computers, and air conditioners can be affected by noise and other interference, compromising the accuracy of voice control. If users are required to mute the TV before performing an operation, this can significantly hinder the adoption of smart home devices. Therefore, a technology that can reduce the interference of noise on speech information has high research and application value.

[0003] The prototype of speech enhancement technology originated at Bell Labs in the 1960s. After decades of development, various digital signal processing methods, such as spectral subtraction, filtering, and subspace-based methods, have been applied to speech enhancement systems, achieving excellent results in suppressing stationary noise. However, in real-world scenarios, a large amount of non-stationary noise is common, making it difficult for digital signal processing methods to achieve effective noise reduction. In the past decade, deep learning has rapidly developed, and neural networks such as DNNs, RNNs, and GANs have been applied to speech enhancement tasks. These methods effectively suppress not only stationary noise but also non-stationary noise. When processing speech signals in the time-frequency domain, the amplitude spectrum is commonly used as a training target. However, these methods lack the ability to estimate phase information, limiting the performance of the noise reduction network. In recent years, phase spectrum estimation has also gained increasing attention. However, these methods rely solely on real-valued networks to estimate complex information and fail to properly model the correlation information in the speech complex spectrum. Summary of the Invention

[0004] The present invention provides a single-channel speech enhancement method based on a multi-attention mechanism, which is used to improve the problem that the existing complex network cascaded multiple encoders for single-channel speech enhancement leads to insufficient information extraction and severe loss of speech features used for signal reconstruction.

[0005] The technical solution of the present invention is: a single-channel speech enhancement method based on a multi-attention mechanism, the specific steps of the method are as follows:

[0006] Step 1: Perform a short-time Fourier transform on the time-domain speech signal to obtain the speech complex spectrum. The speech complex spectrum features are extracted through a complex encoder and used for intermediate layer modeling and signal reconstruction. The encoder-extracted features are sent to the decoder through a residual connection and fused with the decoded features before decoding.

[0007] Step 2: Introduce a complex Conformer as an intermediate layer in the complex U-Net architecture to model the correlation between speech amplitude and phase from the time and frequency dimensions respectively.

[0008] Step 3: A three-dimensional attention mechanism is proposed to dynamically adjust the high-dimensional speech features extracted by the encoder and the reconstructed speech features output by the decoder, thereby enhancing the representation capability of the complex convolutional codec for speech features.

[0009] Step 4: Use the attention gating mechanism to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder.

[0010] As a further solution of the present invention, in Step 1, the process of obtaining the complex spectrum of speech is as follows: clean speech from the VoiceBank dataset is selected for noise addition, and the noise data comes from the DEMAND dataset, which contains noise in different scenarios such as roads, offices, kitchens, and forests. According to four different signal-to-noise ratios such as 0dB, 5dB, 15dB, and 20dB, different noises are randomly sampled and added to the clean audio as the noisy speech data in the training set. The construction process of the test data is similar, but the clean speech does not appear in the training set and has a different signal-to-noise ratio;

[0011] Spectral analysis of speech signals is a key method in speech recognition and audio signal processing. Speech signals are generally non-stationary processes. Therefore, noisy speech is first sampled and quantized to discretize the continuous speech signal. Then, a short-time Fourier transform (SFT) is used to perform time-frequency decomposition. Based on the short-time stationary nature of speech, the speech is divided into 30ms frames, each approximately stationary, ensuring the relative stability of the basic characteristics of the speech within the frame. The speech frames are then windowed using a Hanning window, ensuring that the time-domain signal better meets periodicity requirements and reduces spectral leakage. Finally, a short-time Fourier transform is used to obtain the speech's time-frequency features, namely the speech complex spectrum.

[0012] Assume that the input noisy speech sequence can be expressed as: x(t) = s(t) + n(t). Here, x(t), s(t), and n(t) represent the noisy speech, clean speech, and noise signal, respectively. The goal of the speech enhancement task is to restore clean speech from the noisy speech. Discretize y(t) using the short-time Fourier transform:

[0013]

[0014] In formula (1), w(t) is the window function, the center of the window function is t0, STFT Z (t,f) is a two-dimensional function of time t and frequency f. t is the time frame length, and f is the number of frequency points. The input feature of the encoder can be expressed as: X = STFT Z (x(t)).

[0015] As a further solution of the present invention, in Step 1, a complex encoder is constructed to extract the complex spectral features of speech, a complex filter is constructed by two convolution kernels to simulate complex operations, a convolution operation is performed with the input complex features, and complex normalization and complex PReLU activation function are used for processing; a complex decoder is constructed to restore the high-dimensional features of speech using complex transposed convolution, and its structure is the same as that of the complex encoder.

[0016] As a further solution of the present invention, the complex spectrum of speech contains the amplitude and phase information of speech, so two two-dimensional convolution filters are constructed to simulate complex operations to fully extract the speech features to obtain better representation, which is then processed by complex normalization and complex PReLU function activation. In Step 1, in the complex encoder, the input features of the encoder are represented as: Constructing a 2D convolution filter The operation of the complex convolution layer in the encoder is shown in formula (2), and the complex normalization and complex PReLU function activation are shown in (3):

[0017]

[0018]

[0019] Where: and Represent the real and imaginary parts of the complex spectrum input to the complex convolution layer, respectively. and W i n Represent the real and imaginary parts of the complex convolution kernel, n∈{1,2,3,4,5} represents the n-th convolution block in the codec, j represents the imaginary part identifier, * represents the convolution operation, S n Represents the output of the convolutional coding layer, PReLU represents the activation function, and NB represents the normalization process.

[0020] As a further solution of the present invention, in Step 1, the complex decoder is the inverse process of the complex encoder, and the output features of the encoder and decoder are fused through the residual attention gating mechanism as the input features of the next layer decoder. In the decoder, the output of the decoder is obtained by complex transposed convolution layer, normalization operation and PReLU function activation. The output of the last layer encoder is subjected to inverse short-time Fourier transform to obtain enhanced speech;

[0021]

[0022] In formula (4), and Represent the real and imaginary parts of the complex spectrum input to the decoder, W r n and W i n denote the real and imaginary parts of the complex convolution kernel, respectively, n∈{1,2,3,4,5} denotes the n-th convolution block in the decoder, NB denotes normalization, and TConv denotes the transposed convolution operation.

[0023] As a further solution of the present invention, a complex Conformer is used as an intermediate layer under the complex U-Net architecture to model the speech features extracted by the complex encoder from the time dimension and the frequency dimension respectively, and the real and imaginary parts of the complex speech features are processed simultaneously under the complex calculation rules, fully considering the correlation between the amplitude and phase of the speech signal. In addition, in order to fully capture the contextual information of the speech spectrum in the frequency and time dimensions, two-way dilated convolution is used to perform complex convolution operations on the real and imaginary features in the complex spectrum, thereby effectively modeling the local features of the speech. In the Step 2, it is proposed to use a complex Conformer module under the complex U-Net architecture to model the correlation between the high-dimensional features of the speech amplitude and phase extracted by the encoder from the two dimensions of time and frequency, respectively, wherein the complex self-attention is expressed as:

[0024] ComplexAttention(Q,K,V)=

[0025] Attrntion(Q r ,K r ,V r )-Attention(Q r ,K i ,V i )-Attention(Q i ,K r ,V i )-Attention(Q i ,K i ,V r )+

[0026] i(Attention(Q r ,K r ,V r )+Attention(Q r ,K i ,V r )+Attention(Q i ,K r ,V r )-Attention(Q i ,K r ,V i ))

[0027] Among them, Q r , Q i , K r , K i 、V r 、V i Both represent the input speech complex spectrum Z=Z r +jZ i Through the complex linear layer W = W i +W r The resulting complex weight matrix.

[0028] Among them, Q r ,Q i The calculation of K is shown in the following formulas (5) and (6): r ,K i ,V r ,V i The calculation process is similar.

[0029]

[0030] Q i =X r W iQ +X r W i Q (6)

[0031] In formula (5) and (6), X r 、X i Represent the real and imaginary parts of the complex spectrum used to model speech, W r Q 、W i Q Represent the linear weight matrices in the attention mechanism respectively.

[0032] As a further solution of the present invention, in the Step 3, the three-dimensional attention mechanism is used to dynamically adjust the speech feature block from three dimensions, namely, channel, time, and frequency, to increase the degree of attention to the feature area related to the task, and to maximize the amount of information within the limited feature space, thereby fully representing the feature information. Complex channel attention uses the global information of the feature block to determine the importance of different channels, which can help capture more details between different feature maps and make up for the important information lost between each feature. Within the speech feature map, different feature areas contribute differently to the enhancement task. Complex time attention deaggregates the information within the speech feature block in the frequency direction through average pooling and maximum pooling, and performs convolution operations along the time direction to focus on the local features between different speech frames. The function of complex frequency attention is similar to that of complex time attention, except that complex frequency attention deaggregates information along the time direction and performs convolution operations along the frequency dimension to model the correlation between data features of different sub-bands. The process can be expressed as:

[0033]

[0034] In formula (7), r,i represent the real part and imaginary part subscripts respectively, U C ,U F ,U T Represent the output features of the attention mechanism in channel, frequency, and time dimensions respectively.

[0035] As a further solution of the present invention, in Step 4, an attention gating mechanism is used to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder, while further suppressing the incoming noise.

[0036] In a U-Net network, directly using residual connections to feed speech features extracted by the encoder into the decoder inevitably introduces noise. Therefore, an attention gating mechanism is added to the residual connection to suppress noise information irrelevant to the task. At the same time, the speech detail features output by the encoder and the high-dimensional features output by the decoder are fused and fed into the next decoder, enhancing the quality of the reconstructed speech through the detailed information. The calculation process can be expressed as:

[0037]

[0038] In formula (8), and Represent the output features of the L-th layer encoder and decoder respectively, ReLU and Sigmoid represent ReLU and Sigmoid activation functions respectively, and Conv represents the convolution operation.

[0039] Finally, the speech complex spectrum Y output by the last layer of decoding r,i Perform Fourier transform and then overlap and add the frame signals to obtain the complete predicted speech signal y(t). The calculation process can be expressed as:

[0040]

[0041]

[0042] In formula (9), Indicates short-time Fourier transform, and formula (10) indicates that each speech frame after short-time Fourier transform is concatenated to restore the final time-domain speech waveform signal.

[0043] The proposed method uses a complex U-Net architecture to perform a Fourier transform on the time-domain speech signal to obtain a complex speech spectrum. Speech feature extraction and recovery are performed through a cascade of multi-layer convolutional encoders and decoders, respectively. Existing deep complex networks for single-channel speech enhancement rely heavily on the representational power of the complex convolutional layers and fail to properly model the correlation information in the complex speech spectrum. Therefore, a complex Conformer module is proposed within the complex U-Net architecture to model the correlation between the real and imaginary parts of the complex speech spectrum in both the time and frequency dimensions. A three-dimensional attention mechanism is incorporated into the encoding and decoding process to dynamically adjust speech features, maximizing the amount of information contained in the speech within a limited feature space. Furthermore, the speech detail features output by the encoder and the high-dimensional features output by the decoder are fused using an attention gating mechanism and then fed into the next decoder layer for decoding. The complex speech spectrum obtained after decoding in the final layer undergoes an inverse short-time Fourier transform to obtain the enhanced speech time-domain waveform. Experimental results demonstrate that the proposed method effectively suppresses noise and improves speech quality and intelligibility.

[0044] The beneficial effects of the present invention are as follows: the present invention can improve speech quality and intelligibility, and can be used for voice communication in noisy environments, password control, and the pre-processing part of voice-related tasks; the experimental results of the present invention on the public dataset Voice Bank-DEMAND show that the proposed method achieves evaluation results of 3.09, 4.28, 3.47, 3.72, and 95.07 on five objective evaluation indicators PESQ, CSIG, CBAK, COVL, and STOI, respectively, and can effectively reduce noise and improve speech quality and intelligibility. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of the present invention. DETAILED DESCRIPTION

[0046] Example 1: Figure 1 As shown in FIG, a single-channel speech enhancement method based on a multi-attention mechanism is described, and the specific steps of the method are as follows:

[0047] Step 1: Perform a short-time Fourier transform on the time-domain speech signal to obtain the speech complex spectrum. The speech complex spectrum features are extracted through a 5-layer complex encoder and used for intermediate layer modeling and signal reconstruction. The features extracted by the encoder are sent to the decoder through a residual connection and fused with the decoded features before decoding. In Step 1, the process of obtaining the speech complex spectrum is as follows: clean speech from the Voice Bank dataset is selected for noise addition. The noise data comes from the DEMAND dataset. The 48KHz multi-channel noise data in the DEMAND noise dataset is downsampled and converted from multi-channel to single-channel. The dataset contains noise from different scenarios such as roads, offices, kitchens, and forests. It is randomly loaded onto the speech in the VoiceBank dataset according to four different signal-to-noise ratios, namely 15dB, 10dB, 5dB, and 0dB, and finally 11,572 sets of audio files of 14 males and 14 females are obtained as training data. The construction process of the test set is the same as that of the training set, but the clean speech does not appear in the training set and has a different signal-to-noise ratio.

[0048] The noisy speech is first sampled and quantized to discretize the continuous speech signal. Then, the short-time Fourier transform is used to perform time-frequency decomposition. Based on the short-time stationary characteristics of speech, the speech is divided into several 30ms frames. Each frame is approximately a stationary signal, ensuring that the basic characteristics of the speech within the frame are relatively stable. The speech frames are windowed using a Hanning window. Finally, the short-time Fourier transform is used to obtain the time-frequency characteristics of the speech to be processed, namely the speech complex spectrum, which is used for neural network training.

[0049] In Step 1, a complex encoder is constructed to extract the complex spectral features of speech, a complex filter is constructed by two convolution kernels to simulate complex operations, a convolution operation is performed with the input complex features, and complex normalization and complex PReLU activation function are performed; a complex decoder is constructed to restore the high-dimensional features of speech using complex transposed convolution, and its structure is the same as that of the complex encoder.

[0050] In the Step 1, in the complex encoder, the input feature of the encoder is expressed as: Constructing a 2D convolution filter The operation of the complex convolution layer in the encoder is shown in formula (1), and the complex normalization and complex PReLU function activation are shown in (2):

[0051]

[0052]

[0053] Where: and Represent the real and imaginary parts of the complex spectrum input to the complex convolution layer, W r n and W i n Represent the real and imaginary parts of the complex convolution kernel, n∈{1,2,3,4,5} represents the n-th convolution block in the codec, j represents the imaginary part identifier, * represents the convolution operation, S n Represents the output of the convolutional coding layer, PReLU represents the activation function, and NB represents the normalization process.

[0054] In Step 1, the complex decoder is the inverse process of the complex encoder. The output features of the encoder and decoder are fused through the residual attention gating mechanism as the input features of the next layer of decoder. In the decoder, the output of the decoder is obtained by complex transposed convolution layer, normalization operation and PReLU function activation. The output of the last layer encoder is subjected to inverse short-time Fourier transform to obtain enhanced speech;

[0055]

[0056] In formula (3), and denote the real and imaginary parts of the complex spectrum input to the decoder, and W i n denote the real and imaginary parts of the complex convolution kernel, respectively, n∈{1,2,3,4,5} denotes the n-th convolution block in the decoder, NB denotes normalization, and TConv denotes the transposed convolution operation.

[0057] Step 2: Introduce a complex Conformer as an intermediate layer in the complex U-Net architecture to model the correlation between speech amplitude and phase in both time and frequency dimensions. Then, use a 5-layer decoder to restore the features to their original size.

[0058] As a further solution of the present invention, in Step 2, it is proposed to use a complex Conformer module under the complex U-Net architecture to model the correlation of the speech amplitude and phase high-dimensional features extracted by the encoder from the two dimensions of time and frequency, respectively. The complex self-attention is expressed as:

[0059] ComplexAttention(Q,K,V)=

[0060] Attrntion(Q r ,K r ,V r )-Attention(Q r ,K i ,V i )-Attention(Q i ,K r ,V i )-Attention(Q i ,K i ,V r )+

[0061] i(Attention(Q r ,K r ,V r )+Attention(Q r ,K i ,V r )+Attention(Q i ,K r ,V r )-Attention(Q i ,K r ,V i ))

[0062] Among them, Q r , Q i , K r , K i 、V r 、V i Both represent the input speech complex spectrum Z=Z r +jZ i Through the complex linear layer W = W i +W r The resulting complex weight matrix.

[0063] Step 3: A three-dimensional attention mechanism is proposed to dynamically adjust the high-dimensional speech features extracted by the encoder and the reconstructed speech features output by the decoder, thereby enhancing the representation capability of the complex convolutional codec for speech features.

[0064] As a further solution of the present invention, in Step 3, a multi-dimensional attention mechanism is designed to dynamically adjust the complex spectrum features of speech from the channel, time, and frequency dimensions, so as to maximize the amount of information in a limited feature space; the operation process is expressed as follows:

[0065]

[0066] Among them, r,i represent the real part and imaginary part subscripts respectively, U C ,U F ,U T Represent the output features of the attention mechanism in channel, frequency, and time dimensions respectively.

[0067] Step 4: Use the attention gating mechanism to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder.

[0068] As a further solution of the present invention, in Step 4, an attention gating mechanism is used to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder, while further suppressing the incoming noise, and then sending them to the next layer of decoder for decoding; the speech features output by the last layer of decoder are subjected to inverse Fourier transform to obtain an enhanced speech waveform.

[0069] In order to illustrate the effect of the present invention, the following experimental verification was conducted. The Adam function was used for optimization and the learning rate was set to 3×10 -4 Power, a total of 200 cycles of learning.

[0070] Table 1 shows the test results after adding the complex Conformer, three-dimensional attention mechanism, and attention gating mechanism in sequence to the network condition with only complex encoding and decoding.

[0071] Table 2 shows the effects of not adding a complex Conformer, adding a complex Transformer, and adding different complex Conformers on the enhancement effect.

[0072] Table 1 Final experimental results of the model after adding different modules in sequence

[0073] model WB-PESQ CSIG CBAK COVL STOI Only the codec infrastructure 2.632 3.54 3.02 3.24 93.56 The result after introducing the complex number Conformer 2.810 4.26 3.28 3.55 94.29 The result after introducing the three-dimensional attention mechanism 2.898 4.28 3.33 3.64 94.37 The result after introducing the attention gating mechanism 2.927 4.28 3.39 3.61 94.32 Final experimental results 3.09 4.38 3.47 3.72 95.07

[0074] Table 2: Results of the impact of different numbers of Conformers on the experiment

[0075]

[0076]

[0077] The evaluation results in Tables 1 and 2 demonstrate that the proposed single-channel speech enhancement method based on a multi-attention mechanism dynamically adjusts speech features through a three-dimensional attention mechanism, enhancing the representational power of the complex convolutional coding layer. The complex Conformer model models the correlation between amplitude and phase in the complex speech feature map. Furthermore, an attention gating mechanism is used to fuse speech features output by the codec, further suppressing speech features used for decoding. Tested on the public Voice Bank-DEMAND dataset, the method achieved scores of 3.09 and 95.07 on the objective evaluation metrics PESQ and STOI, respectively, demonstrating that it effectively suppresses noise and enhances speech quality and intelligibility.

[0078] The specific embodiments of the present invention are described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made within the knowledge of ordinary technicians in this field without departing from the scope of the present invention.

Claims

1. A single-channel speech enhancement method based on a multi-attention mechanism, characterized by: The specific steps of the method are as follows: Step 1: Perform a short-time Fourier transform on the time-domain speech signal to obtain the speech complex spectrum. The speech complex spectrum features are extracted through a complex encoder and used for intermediate layer modeling and signal reconstruction. The encoder-extracted features are sent to the decoder through a residual connection and fused with the decoded features before decoding. Step 2: Introduce a complex Conformer as an intermediate layer in the complex U-Net architecture to model the correlation between speech amplitude and phase from the time and frequency dimensions respectively. Step 3: A three-dimensional attention mechanism is proposed to dynamically adjust the high-dimensional speech features extracted by the encoder and the reconstructed speech features output by the decoder, thereby enhancing the representation capability of the complex convolutional codec for speech features. Step 4: Use the attention gating mechanism to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder.

2. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 1, the process of obtaining the speech complex spectrum is as follows: a data set is selected and noise with different signal-to-noise ratios is randomly extracted and added to the clean audio as the noisy speech data in the training set; the construction process of the test set is the same as that of the training set, except that the clean speech does not appear in the training set and has a different signal-to-noise ratio; The noisy speech is first sampled and quantized to discretize the continuous speech signal. The short-time Fourier transform is then used to perform time-frequency decomposition. Based on the short-time stationary characteristics of speech, the speech is divided into several 30ms frames. Each frame is approximately a stationary signal, ensuring that the basic characteristics of the speech within the frame are relatively stable. The speech frames are windowed using a Hanning window. Finally, the short-time Fourier transform is used to obtain the time-frequency characteristics of the speech to be processed, namely the speech complex spectrum.

3. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 1, a complex encoder is constructed to extract the complex spectral features of speech, a complex filter is constructed by two convolution kernels to simulate complex operations, a convolution operation is performed with the input complex features, and complex normalization and complex PReLU activation function are performed; a complex decoder is constructed to restore the high-dimensional features of speech using complex transposed convolution, and its structure is the same as that of the complex encoder.

4. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In the Step 1, in the complex encoder, the input feature of the encoder is expressed as: Construct a two-dimensional convolution filter W n =W r n +jW i n , then the operation of the complex convolution layer in the encoder is shown in formula (1), and the complex normalization and complex PReLU function activation are shown in (2): Where: and Represent the real and imaginary parts of the complex spectrum input to the complex convolution layer, W r n and W i n Represent the real and imaginary parts of the complex convolution kernel, n∈{1, 2, 3, 4, 5} represents the n-th convolution block in the codec, j represents the imaginary part identifier, * represents the convolution operation, S n Represents the output of the convolutional coding layer, PReLU represents the activation function, and NB represents the normalization process.

5. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 1, the complex decoder is the inverse process of the complex encoder. The output features of the encoder and decoder are fused through the residual attention gating mechanism as the input features of the next layer of decoder. In the decoder, the output of the decoder is obtained by complex transposed convolution layer, normalization operation and PReLU function activation. The output of the last layer encoder is subjected to inverse short-time Fourier transform to obtain enhanced speech; In formula (3), and Represent the real and imaginary parts of the complex spectrum input to the decoder, W r n and W i n denote the real and imaginary parts of the complex convolution kernel, n∈{1, 2, 3, 4, 5} denotes the n-th convolution block in the decoder, NB denotes normalization, and TConv denotes the transposed convolution operation.

6. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 2, it is proposed to use a complex Conformer module under the complex U-Net architecture to model the correlation of the speech amplitude and phase high-dimensional features extracted by the encoder from the two dimensions of time and frequency, respectively. The complex self-attention is expressed as: ComplexAttention(Q,K,V)= Attrntion(Qr,K r ,V r )-Attention(Q r ,K i ,V i )-Attention(Q i ,K r ,V i )-Attention(Q i ,K i ,V r )+i(Attention(Q r ,K r ,V r )+Attention(Q r ,K i ,V r )+Attention(Q i ,K r ,V r )-Attention(Q i ,K r ,V i )) Among them, Q r , Q i , K r , K i 、V r 、V i Both represent the input speech complex spectrum Z=Z r +jZ i Through the complex linear layer W = W i +W r The resulting complex weight matrix.

7. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 3, a multi-dimensional attention mechanism is designed to dynamically adjust the complex spectrum features of speech from the channel, time, and frequency dimensions, maximizing the amount of information in a limited feature space. The operation process is expressed as: Among them, r,i represent the real part and imaginary part subscripts respectively, U C ,U F ,U T Represent the output features of the attention mechanism in channel, frequency, and time dimensions respectively.

8. The single-channel speech enhancement method based on the multi-attention mechanism according to claim 1, characterized in that: In Step 4, an attention gating mechanism is used to fuse the speech detail features output by the encoder and the high-dimensional features output by the decoder, while further suppressing the incoming noise.

Images