A speech enhancement method, device and equipment based on DFSMN model

Abstract

In view of the problems of large parameter quantity and large calculation quantity of existing algorithms, the application discloses a speech enhancement method, device and equipment based on a DFSMN model, which comprises a short-time Fourier transform module, an encoder module, a double-path DFSMN module, a mask estimation module and a short-time inverse Fourier transform module, wherein the short-time Fourier transform module is used for converting noisy speech into a complex spectrum, the encoder module is used for converting the noisy speech complex spectrum into multi-scale coding features, the double-path DFSMN module is used for modeling the features in the time dimension and the frequency dimension, the decoder is used for reducing the dimension of the features, and the mask estimation module is used for estimating an enhanced speech mask. The application has a small parameter quantity and a small calculation quantity, and realizes a good speech enhancement effect, and has strong practicability.

Description

Technical Field

[0001] This invention relates to the field of speech enhancement technology, and in particular to a speech enhancement method based on the DFSMN model. Background Technology

[0002] Voice communication is the most direct, clearest, and most effective way for humans to exchange information. With the advent of the information age and the continuous development of internet and 5G communication technologies, an increasing variety of remote voice communication methods are emerging in people's daily lives and work, such as telephone voice calls, audio and video communication, video conferencing, live streaming, and in-vehicle communication. In noisy environments, machines cannot distinguish the information they want to obtain like the human ear. Therefore, machines have higher requirements for voice quality. Noise interference may cause intelligent devices, such as intelligent voice assistants, to be incorrectly activated at the wrong time or to respond incorrectly to noisy voice commands. To enable machines to respond efficiently and accurately to human voice commands, the voice picked up by the machine needs to be processed in the front end to produce high-quality and highly intelligible speech. Voice enhancement technology plays an important role in voice interaction. It extracts clean and clear target speech from noisy mixed speech, enhances the intelligibility of the target speech, thereby improving the accuracy of machine keyword capture and ultimately enhancing the user's human-computer interaction experience.

[0003] In terms of model parameter quantity and computational cost, existing algorithms typically have a large number of parameters and computational cost, which hinders the practical application of the algorithms; in terms of model design, existing algorithms usually focus on the temporal information between speech signals, thus ignoring the relevant information between frequencies, which limits the effect of speech enhancement. Summary of the Invention

[0004] To address the above problems, this invention provides a speech enhancement method based on the DFSMN model.

[0005] Firstly, a speech enhancement method based on the DFSMN model is provided, comprising the following steps:

[0006] Step 1: Convert the noisy speech into a complex spectral representation;

[0007] Step 2: Encode the complex spectrum of noisy speech using a DFSMN-based encoder module. The encoder module contains several sub-encoder modules. Each sub-encoder module consists of a two-dimensional convolution module, a block normalization layer, a PReLU layer, a first-dimensional transformation operation, a DFSMN layer, and a second-dimensional transformation operation.

[0008] Step 3: Use the dual-path DFSMN module to extract context information from the encoder module output features along the time axis and frequency axis;

[0009] Step 4: Feature decoding is achieved through a DFSMN-based decoder module. The decoder module is a mirror image of the encoder module. The input features of each decoder are the jump connection information between the output of the previous decoder and the corresponding encoder. Feature dimensionality reduction is achieved through the decoder module.

[0010] Step 5: Use the mask estimation module to convert the decoder output features into an estimated complex spectrum of the enhanced speech.

[0011] Step 6: Perform a short-time inverse Fourier transform on the estimated enhanced speech complex spectrum obtained in Step 5 to convert the complex spectrum into a time-domain signal.

[0012] Furthermore, in step 1, the noisy speech is converted into a complex spectrum using a short-time Fourier transform module, as shown in the following formula:

[0013]

[0014] Y = concat(real(Y) c imag(Y) c (2)

[0015] Where Y c ∈C B×T×F Let Y be the complex spectrum of the noisy speech in complex form, where π represents pi, j represents a complex number, t and f represent time and frequency, T represents the number of frames in the complex spectrum, F represents the number of frequency points in the complex spectrum, y(n) represents the nth sampling point of the noisy speech, N is the length of the noisy speech, h(n) represents the Hamming window, B represents the batch size, concat(·,·) represents the feature concatenation operation, real(·) represents the computation of the real part of the complex number, and imag(·) represents the computation of the imaginary part of the complex number. B×2×T×F For the real number representation of the complex spectrum of noisy speech;

[0016] Furthermore, in step 2, the encoder module transforms Y(t,f) into feature representations at different scales. Within the encoder module, 2D convolution achieves feature map downsampling by changing the kernel size. Block normalization is used to normalize the data, preventing gradient explosion or vanishing during network training. The PReLU layer enhances the network's nonlinear mapping representation capability. The first dimension transformation operation converts the feature map dimension. The DFSMN layer models the feature map frequency. The second dimension transformation operation restores the feature dimension, facilitating subsequent processing by the sub-encoder module. The encoding process formula is as follows:

[0017]

[0018] Where E m (·) represents the m-th sub-encoder. This represents the input of the m-th sub-encoder, where M represents the number of sub-encoders in the encoder module. The input of the first sub-encoder module is the complex spectrum Y of the noisy speech, i.e. The input of the m-th sub-encoder module is the output of the (m-1)-th sub-encoder module, that is...

[0019] Furthermore, in step 3, the dual-path DFSMN module consists of two parts: intra-frame modeling and inter-frame modeling. The intra-frame modeling module processes the input features by first transforming their dimensions through a dimensionality transformation operation, then modeling the frequency dimension through a DFSMN layer, followed by normalizing the features through an instance normalization layer, and finally enhancing the feature representation capability through a linear layer. A skip connection operation follows the intra-frame modeling operation to prevent overfitting. The inter-frame modeling module processes the input features by first transforming their dimensions through a dimensionality transformation operation, then modeling the temporal dimension of the features through a DFSMN layer, an instance normalization layer, and a linear layer, and finally connecting the input of the inter-frame modeling module with the output of the linear layer using a skip connection operation to prevent overfitting. The formula for the dual-path DFSMN modeling process is as follows:

[0020]

[0021]

[0022]

[0023]

[0024] in This indicates the output of the encoder module, and trans(·) represents the feature transformation operation. This represents the feature input of the intra-frame modeling module. Indicates intra-frame modeling output. This indicates the output of inter-frame modeling. Indicates the output of the dual-path DFSMN module, M inter (·) indicates intra-frame modeling feature calculation, M cross (·) indicates the calculation of inter-frame modeling features.

[0025] Furthermore, the decoding process of the decoder in step 4 is expressed by the following formula:

[0026]

[0027]

[0028] Where D m (·) represents the m-th sub-decoder. This represents the input of the m-th sub-decoder. This represents the output of the m-th sub-decoder.

[0029] Furthermore, the mask estimation module in step 5 consists of two stages. Stage 1 uses deep filtering to apply a real-valued mask to the amplitude spectrum to estimate the amplitude spectrum of the enhanced speech. Stage 2 applies a complex mask to the amplitude and phase to eliminate residual noise. Stage 1 first reduces the number of feature channels through two-dimensional convolution, then maps the features to the interval [0,1] through the sigmoid function to obtain the time-frequency filter coefficients estimated in Stage 1. Finally, deep filtering is used to convert the time-frequency filter coefficients into the amplitude spectrum of the enhanced speech. This process is expressed by the following formula:

[0030]

[0031] Y mag =||Y(t,f)|| (11)

[0032] Y pha =∠Y(t,f) (12)

[0033]

[0034] in The output of the decoder is represented by Conv(·), which represents a two-dimensional convolution operation, σ(·) represents the sigmoid activation function, ||·|| represents amplitude calculation, ∠ represents phase calculation, t and f represent time and frequency, and Y represents the frequency. mag Y represents the amplitude spectrum of mixed speech. pha Let represent the phase spectrum, 2I+1 and 2J+1 represent the filter window sizes of the depth filter along the time and frequency axes, respectively, and i and j represent the offsets of the depth filter in the time and frequency domains, respectively. This represents the time-frequency filter coefficients estimated in stage 1. This represents the amplitude spectrum estimated in stage 1;

[0035] In stage 2, the decoder output features are first fed into two parallel two-dimensional convolutional modules to estimate the real part mask. and imaginary part mask The amplitude spectrum mask for stage 2 is obtained by performing feature transformation on the estimated complex mask. and phase spectrum mask The formula is expressed as follows:

[0036]

[0037]

[0038]

[0039]

[0040] Then the amplitude spectrum mask from stage 2 is applied to the amplitude spectrum estimated in stage 1. Phase spectral masking is used to compensate for the phase of noisy speech. Finally, the complex spectrum estimated in stage 2 is coupled with the compensated phase spectrum to obtain the complex spectrum of the target speech. The calculation process is as follows:

[0041]

[0042]

[0043] in and These represent the real and imaginary parts of the estimated complex spectrum of the enhanced speech, respectively.

[0044] Furthermore, in step 6, the estimated enhanced speech complex spectrum is subjected to a short-time inverse Fourier transform to recover the time-domain signal, as expressed by the following formula:

[0045]

[0046]

[0047] in This indicates the estimation of the complex spectrum of enhanced speech. The expression represents the estimated time-domain signal of the enhanced speech, where π represents pi, j represents a complex number, t and f represent time and frequency, T represents the number of frames in the complex spectrum, F represents the number of frequency points in the complex spectrum, N is the length of the noisy speech, and h() represents the Hamming window.

[0048] Secondly, the present invention provides a speech enhancement device based on the DFSMN model, comprising the following modules:

[0049] The speech conversion module is used to convert noisy speech into a complex spectrum representation;

[0050] The encoding module is used to encode the complex spectrum of noisy speech using a DFSMN-based encoder module. The encoder module contains several sub-encoder modules, each of which consists of a two-dimensional convolution module, a block normalization layer, a PReLU layer, and a DFSMN layer.

[0051] The information extraction module is used to extract contextual information from the encoder module output features along the time and frequency axes using the dual-path DFSMN module;

[0052] The decoding module is used to implement feature decoding through a DFSMN-based decoder module. The decoder module is a mirror structure of the encoder module. The input features of each decoder are the jump connection information between the output of the previous decoder and the corresponding encoder. Feature dimensionality reduction is achieved through the decoder module.

[0053] The speech enhancement module is used to convert the decoder output features into an estimated enhanced speech complex spectrum using the mask estimation module; the time-domain signal conversion module is used to perform a short-time inverse Fourier transform on the obtained estimated enhanced speech complex spectrum to convert the complex spectrum into a time-domain signal.

[0054] Thirdly, the present invention provides a speech enhancement device based on a DFSMN model, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a speech enhancement method based on a DFSMN model as described above.

[0055] In the above scheme, the DFSMN-based speech enhancement method can reduce the computational load of the model and improve its expressive power according to the encoder module and decoder module. The dual-path DFSMN module can realize time modeling and frequency modeling, improve the speech enhancement effect of the model, and has a smaller number of model parameters. Attached Figure Description

[0056] Figure 1 This is a model structure diagram of an embodiment of the present invention.

[0057] Figure 2 This is a structural diagram of the sub-encoder and sub-decoder modules in a real-time example of the present invention. Figure 2 (a) is a structural diagram of the sub-encoder according to an embodiment of the present invention. Figure 2 (b) is a sub-decoder structure diagram of an embodiment of the present invention.

[0058] Figure 3 This is a structural diagram of the dual-path DFSMN module according to an embodiment of the present invention.

[0059] Figure 4 This is a structural diagram of the mask estimation module in an embodiment of the present invention. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Figure 1 As shown, the present invention includes the following specific steps:

[0061] Step 1: Convert the noisy speech into a complex spectrum Y∈R using the short-time Fourier transform module. B×2×T×F The formula is expressed as follows:

[0062]

[0063] Y = concat(real(Y) c imag(Y) c (2)

[0064] Where Y c ∈C B×T×F Let Y be the complex spectrum of the noisy speech in complex form, where π represents pi, j represents a complex number, t and f represent time and frequency, T represents the number of frames in the complex spectrum, F represents the number of frequency points in the complex spectrum, y(n) represents the nth sampling point of the noisy speech, N is the length of the noisy speech, h(n) represents the Hamming window, B represents the batch size, concat(·,·) represents the feature concatenation operation, real(·) represents the computation of the real part of the complex number, and imag(·) represents the computation of the imaginary part of the complex number. B×2×T×F For the real number representation of the complex spectrum of noisy speech, in this embodiment, the Hamming window length is set to 512 sampling points and the window shift is 128 sampling points. Due to the symmetric nature of the output of the short-time Fourier transform, this method only considers the DC component and the positive frequency part of the spectrum. Therefore, the final frequency feature dimension F = 512 / 2 + 1 = 257.

[0065] Step 2: The noisy speech complex spectrum Y is transformed into high-dimensional feature representations at different scales using an encoder module. The encoder module consists of several sub-encoder modules. In this embodiment, the number of sub-encoder layers is set to 5. The specific parameter settings are shown in Table 1.

[0066] Table 1: Encoder Module Parameter Settings

[0067]

[0068] The overall structure of the encoder is as follows Figure 1 As shown in (a), each sub-encoder module consists of a two-dimensional convolutional module, a block normalization layer, a PReLU layer, and a DFSMN layer, with the specific structure as follows: Figure 2 As shown in (a), the DFSMN layer is a sequence modeling module that offers superior sequence modeling performance compared to BiLSTM, but with lower computational cost and faster speed. In the encoder module, 2D convolution achieves feature map downsampling by changing the kernel size. Block normalization is used to normalize the data, preventing gradient explosion or vanishing during network training. The PReLU layer enhances the network's nonlinear mapping representation capability. The first dimension transformation operation converts the feature map dimension. The DFSMN layer performs frequency modeling on the feature map, and the second dimension transformation operation restores the feature dimension for subsequent processing by the sub-encoder module. The encoding process of the encoder in this embodiment is expressed by the following formula:

[0069]

[0070] Where E m (·) represents the m-th sub-encoder. This represents the input of the m-th sub-encoder. The input of the first sub-encoder module is the noisy speech complex spectrum Y, i.e. The input of the m-th sub-encoder module is the output of the (m-1)-th sub-encoder module, that is... The input and output feature dimensions of each sub-encoder are shown in Table 1;

[0071] Step 3: Perform time and frequency modeling on the encoder output using a dual-path DFSMN module. The structure of the dual-path DFSMN module is as follows: Figure 3 As shown. The dual-path DFSMN module mainly consists of two parts: intra-frame modeling and inter-frame modeling. The intra-frame modeling module is as follows: Figure 3 As shown in (a), the input features are first transformed in terms of feature dimension through a dimension transformation operation, then the frequency dimension is modeled through a DFSMN layer, followed by feature normalization through an instance normalization layer, and finally the feature representation capability is improved through a linear layer. A skip connection operation is performed after the intra-frame modeling operation to avoid model overfitting; the inter-frame modeling module is as follows... Figure 3 As shown in (b), the feature dimensions are first transformed using a dimension transformation operation. Then, the temporal dimension of the features is modeled using a DFSMN layer, an instance normalization layer, and a linear layer. Finally, a skip connection operation is used to connect the input of the inter-frame modeling and the output of the linear layer to avoid model overfitting. The formula for the dual-path DFSMN modeling process in this embodiment is as follows:

[0072]

[0073]

[0074]

[0075]

[0076] in This indicates the output of the fifth sub-encoder module, and trans(·) represents the feature transformation operation. This represents the feature input of the intra-frame modeling module. Indicates intra-frame modeling output. This indicates the output of inter-frame modeling. This indicates the output of the dual-path DFSMN module. In this embodiment, the memory module of the DFSMN layer stores 20 units of past information and does not use future information, resulting in a model with low processing latency and good temporal modeling capabilities. M inter (·) indicates intra-frame modeling feature calculation, M cross (·) indicates the calculation of inter-frame modeling features;

[0077] Step 4: Decode the features using the decoder module. The decoder module is a mirror image of the encoder and consists of 5 sub-decoder modules. The overall structure is as follows: Figure 1 As shown in (b), the specific structure of each sub-decoder is as follows: Figure 2 As shown in (b), the parameter settings are as shown in Table 2:

[0078] Table 2: Decoder Module Parameter Settings

[0079]

[0080] In this embodiment, the decoder decoding process is expressed by the following formula:

[0081]

[0082]

[0083] Where D m (·) represents the m-th sub-decoder. This represents the input of the m-th sub-decoder. This represents the output of the m-th sub-decoder. The input and output feature dimensions of the sub-decoder are shown in Table 2.

[0084] Step 5: The decoder output features and complex spectrum are converted into an estimated complex spectrum of the enhanced speech using the mask estimation module. For example... Figure 4 As shown, the mask estimation module consists of two stages: Stage 1 estimates the amplitude spectrum of the enhanced speech, and Stage 2 estimates the complex spectrum of the enhanced speech. Stage 1 first reduces the number of feature channels through two-dimensional convolution, then maps the features to the interval [0,1] using the Sigmoid function, obtaining the time-frequency filter coefficients estimated in Stage 1. Finally, deep filtering is used to convert the time-frequency filter coefficients into the amplitude spectrum of the enhanced speech. This process is expressed by the following formula:

[0085]

[0086] Y mag =||Y(t,f)|| (11)

[0087] Y pha =∠Y(t,f) (12)

[0088]

[0089] in Y represents the output of the fifth sub-decoder, Conv(·) represents the two-dimensional convolution operation, σ(·) represents the sigmoid activation function, ||·|| represents the magnitude calculation, ∠ represents the phase calculation, and Y... mag ∈R B×T×257 Y represents the amplitude spectrum of mixed speech.pha ∈R B×T×257 Let represent the phase spectrum, 2I+1 and 2J+1 represent the filter window size of the depth filter along the time axis and frequency axis, respectively, and i and j represent the offset of the depth filter in the time domain and frequency domain, respectively. In this embodiment, I is set to 1 and J is set to 3. This represents the time-frequency filter coefficients estimated in stage 1. This represents the amplitude spectrum estimated in stage 1.

[0090] In stage 2, the decoder output features are first fed into two parallel two-dimensional convolutional modules to estimate the real part mask. and imaginary part mask The amplitude spectrum mask for stage 2 is obtained by performing feature transformation on the estimated complex mask. and phase spectrum mask The formula is expressed as follows:

[0091]

[0092]

[0093]

[0094]

[0095] Then the amplitude spectrum mask from stage 2 is applied to the amplitude spectrum estimated in stage 1. Phase spectral masking is used to compensate for the phase of noisy speech. Finally, the complex spectrum estimated in stage 2 is coupled with the compensated phase spectrum to obtain the complex spectrum of the target speech. The calculation process is as follows:

[0096]

[0097]

[0098] in and These represent the real and imaginary parts of the estimated complex spectrum of enhanced speech, respectively.

[0099] Step 6: Perform a short-time inverse Fourier transform on the estimated enhanced speech complex spectrum to recover the time-domain signal, as shown in the following formula:

[0100]

[0101]

[0102] in This indicates the estimation of the complex spectrum of enhanced speech. This indicates the estimation of the enhanced speech time-domain signal.

[0103] Secondly, the present invention provides a speech enhancement device based on the DFSMN model, comprising the following modules:

[0104] The speech conversion module is used to convert noisy speech into a complex spectrum representation;

[0105] The encoding module is used to encode the complex spectrum of noisy speech using a DFSMN-based encoder module. The encoder module contains several sub-encoder modules, each of which consists of a two-dimensional convolution module, a block normalization layer, a PReLU layer, and a DFSMN layer.

[0106] The information extraction module is used to extract contextual information from the encoder module output features along the time and frequency axes using the dual-path DFSMN module;

[0107] The decoding module is used to implement feature decoding through a DFSMN-based decoder module. The decoder module is a mirror structure of the encoder module. The input features of each decoder are the jump connection information between the output of the previous decoder and the corresponding encoder. Feature dimensionality reduction is achieved through the decoder module.

[0108] An enhanced speech module is used to convert the decoder output features into an estimated complex spectrum of enhanced speech using a mask estimation module.

[0109] The time-domain signal conversion module is used to perform a short-time inverse Fourier transform on the obtained estimated enhanced speech complex spectrum, converting the complex spectrum into a time-domain signal.

[0110] The specific implementation methods of each module are the same as those of each step, and will not be described in this invention.

[0111] Thirdly, the present invention provides a speech enhancement device based on a DFSMN model, including a processor and a memory. The memory is used to store program instructions, and the processor is used to call the program instructions in the memory to execute a speech enhancement method based on a DFSMN model as described above.

[0112] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to substitute them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A speech enhancement method based on the DFSMN model, characterized in that, Includes the following steps: Step 1: Convert the noisy speech into a complex spectral representation; Step 2: Encode the complex spectrum of noisy speech using a DFSMN-based encoder module. The encoder module contains several sub-encoder modules. Each sub-encoder module consists of a two-dimensional convolution module, a block normalization layer, a PReLU layer, a first-dimensional transformation operation, a DFSMN layer, and a second-dimensional transformation operation. Step 3: Use the dual-path DFSMN module to extract context information from the encoder module output features along the time axis and frequency axis; Step 3's dual-path DFSMN module consists of two parts: intra-frame modeling and inter-frame modeling. The intra-frame modeling module processes the input features by first transforming their dimensions using a dimensionality transformation operation, then modeling the frequency dimension using a DFSMN layer, followed by normalizing the features using an instance normalization layer, and finally enhancing the feature representation capability using a linear layer. A skip connection operation follows the intra-frame modeling operation to prevent overfitting. The inter-frame modeling module processes the input features by first transforming their dimensions using a dimensionality transformation operation, then modeling the temporal dimension of the features using a DFSMN layer, an instance normalization layer, and a linear layer, and finally using a skip connection operation to connect the input of the inter-frame modeling module and the output of the linear layer to prevent overfitting. The formula for the dual-path DFSMN modeling process is as follows: (4) (5) (6) (7) in This indicates the output of the encoder module, where M represents the number of sub-encoders in the encoder module. Indicates feature transformation operation, This represents the feature input of the intra-frame modeling module. Indicates intra-frame modeling output. This indicates the output of inter-frame modeling. This indicates the output of the dual-path DFSMN module. This indicates the calculation of intra-frame modeling features. This indicates the calculation of inter-frame modeling features; Step 4: Feature decoding is achieved through a DFSMN-based decoder module. The decoder module is a mirror image of the encoder module. The input features of each decoder are the jump connection information between the output of the previous decoder and the corresponding encoder. Feature dimensionality reduction is achieved through the decoder module. Step 5: Use the mask estimation module to convert the decoder output features into an estimated complex spectrum of the enhanced speech; Step 6: Perform a short-time inverse Fourier transform on the estimated enhanced speech complex spectrum obtained in Step 5 to convert the complex spectrum into a time-domain signal.

2. The speech enhancement method based on the DFSMN model as described in claim 1, characterized in that: In step 1, the noisy speech is converted into a complex spectrum using the short-time Fourier transform module, as shown in the following formula: (1) (2) in The complex spectrum of noisy speech in complex form. Represents pi (π). To represent a complex number, and Representing time and frequency, Indicates the frame number of the complex spectrum. Represents the number of frequency points in the complex spectrum. The first noisy speech One sampling point, The length of the noisy speech. Indicates Hanming window, B Indicates the batch size. Represents the feature concatenation operation, real imag represents the calculation of the real part of a complex number. This indicates the calculation of the imaginary part of a complex number. is the real number representation of the complex spectrum of noisy speech.

3. The speech enhancement method based on the DFSMN model as described in claim 2, characterized in that: In step 2, the encoder module will... The feature maps are transformed into feature representations at different scales. In the encoder module, 2D convolution downsamples the feature maps by changing the kernel size. Block normalization is used to normalize the data, avoiding gradient explosion or vanishing during network training. PReLU layers enhance the network's nonlinear mapping representation capability. The first dimension transformation operation transforms the feature map dimension. The DFSMN layer models the feature map frequency. The second dimension transformation operation restores the feature dimension, facilitating subsequent processing by the sub-encoder module. The encoding process is expressed by the following formula: (3) in Indicates the first m Individual encoders, Indicates the first m The input of each sub-encoder, This represents the output of the m-th sub-encoder module. M This indicates the number of sub-encoders in the encoder module. The input to the first sub-encoder module is the complex spectrum of noisy speech. Y ,Right now The next m The input of the sub-encoder module is the first m -1 is the output of the sub-encoder module, i.e. .

4. The speech enhancement method based on the DFSMN model as described in claim 3, characterized in that: The decoding process in step 4 is expressed by the following formula: (8) (9) in Indicates the first m Individual decoder, Indicates the first m The input of each sub-decoder Indicates the first m The output of each sub-decoder This indicates the output of the dual-path DFSMN module. This indicates the output of the first sub-encoder module.

5. A speech enhancement method based on a DFSMN model as described in claim 2, characterized in that: Step 5, the mask estimation module consists of two stages. Stage 1 uses deep filtering to apply a real-valued mask to the amplitude spectrum to estimate the enhanced speech amplitude spectrum. Stage 2 applies a complex mask to the amplitude and phase to eliminate residual noise. Stage 1 first reduces the number of feature channels through two-dimensional convolution, then maps the features to the interval [0,1] using the sigmoid function to obtain the time-frequency filter coefficients estimated in Stage 1. Finally, deep filtering is used to convert the time-frequency filter coefficients into the enhanced speech amplitude spectrum. This process is expressed by the following formula: (10) (11) (12) (13) in Indicates the decoder output. Represents a two-dimensional convolution operation. This represents the sigmoid activation function. Indicates amplitude calculation, Indicates phase calculation, and Representing time and frequency, Represents the amplitude spectrum of mixed speech. Represents the phase spectrum, 2 +1 and 2 +1 represents the filter window size along the time axis and frequency axis, respectively. i and j These represent the offsets of the depth filter in the time and frequency domains, respectively. This represents the time-frequency filter coefficients estimated in stage 1. This represents the amplitude spectrum estimated in stage 1; In stage 2, the decoder output features are first fed into two parallel two-dimensional convolutional modules to estimate the real part mask. and imaginary part mask The estimated complex mask is then subjected to feature transformation to obtain the amplitude spectrum mask for stage 2. and phase spectrum mask The formula is expressed as follows: (14) (15) (16) (17) Then the amplitude spectrum mask from stage 2 is applied to the amplitude spectrum estimated in stage 1. Phase spectrum masking is used to compensate for the phase of noisy speech. Finally, the complex spectrum estimated in stage 2 is coupled with the compensated phase spectrum to obtain the complex spectrum of the target speech. The calculation process is as follows: (18) (19) in and These represent the real and imaginary parts of the estimated complex spectrum of the enhanced speech, respectively.

6. A speech enhancement method based on a DFSMN model as described in claim 5, characterized in that: In step 6, the estimated enhanced speech complex spectrum is further subjected to a short-time inverse Fourier transform to recover the time-domain signal, as expressed by the following formula: (20) (21) in This indicates the estimation of the complex spectrum of enhanced speech. This indicates the estimation of the enhanced speech time-domain signal. Represents pi (π). To represent a complex number, and Representing time and frequency, Indicates the frame number of the complex spectrum. Represents the number of frequency points in the complex spectrum. The length of the noisy speech. It refers to the Hanming window.

7. A speech enhancement device based on the DFSMN model, characterized in that, Includes the following modules: The speech conversion module is used to convert noisy speech into a complex spectrum representation; The encoding module is used to encode the complex spectrum of noisy speech using a DFSMN-based encoder module. The encoder module contains several sub-encoder modules, each of which consists of a two-dimensional convolution module, a block normalization layer, a PReLU layer, and a DFSMN layer. The information extraction module is used to extract contextual information from the encoder module output features along the time and frequency axes using the dual-path DFSMN module; The dual-path DFSMN module consists of two parts: intra-frame modeling and inter-frame modeling. The intra-frame modeling module processes the input features by first transforming their dimensions using a dimensionality transformation operation, then modeling the frequency dimension using a DFSMN layer, followed by feature normalization using an instance normalization layer, and finally enhancing feature representation using a linear layer. A skip connection operation follows the intra-frame modeling operation to prevent overfitting. The inter-frame modeling module processes the input features by first transforming their dimensions using a dimensionality transformation operation, then modeling the temporal dimension of the features using a DFSMN layer, an instance normalization layer, and a linear layer. Finally, a skip connection operation connects the input of the inter-frame modeling module with the output of the linear layer to prevent overfitting. The formula for the dual-path DFSMN modeling process is as follows: (4) (5) (6) (7) in This indicates the output of the encoder module, where M represents the number of sub-encoders in the encoder module. Indicates feature transformation operation, This represents the feature input of the intra-frame modeling module. Indicates intra-frame modeling output. This indicates the output of inter-frame modeling. This indicates the output of the dual-path DFSMN module. This indicates the calculation of intra-frame modeling features. This indicates the calculation of inter-frame modeling features; The decoding module is used to implement feature decoding through a DFSMN-based decoder module. The decoder module is a mirror structure of the encoder module. The input features of each decoder are the jump connection information between the output of the previous decoder and the corresponding encoder. Feature dimensionality reduction is achieved through the decoder module. An enhanced speech module is used to convert the decoder output features into an estimated complex spectrum of enhanced speech using a mask estimation module. The time-domain signal conversion module is used to perform a short-time inverse Fourier transform on the obtained estimated enhanced speech complex spectrum, converting the complex spectrum into a time-domain signal.

8. A speech enhancement device based on the DFSMN model, characterized in that: It includes a processor and a memory, the memory being used to store program instructions, and the processor being used to call the program instructions in the memory to execute a speech enhancement method based on the DFSMN model as described in any one of claims 1-6.

Images