MP-SENet speech enhancement method based on frequency domain channel attention and feature compression

Abstract

The application provides an MP-SENet speech enhancement method based on a frequency domain channel attention and feature compression, and relates to the technical field of speech signal processing. The method first performs spectral decomposition on noisy speech and inputs the noisy speech into an encoder of an MP-SENet model to which FcaNet (a frequency domain channel attention network) and SCConv (a spatial and channel reconstruction convolution) are introduced; the FcaNet realizes adaptive feature compression and restoration through discrete cosine transformation and a frequency domain channel attention mechanism, improves global modeling and feature expression capability for key speech components; furthermore, the SCConv can perform convolution operation in a spatial domain and a channel domain, adaptively compresses redundant spatial and channel features, reduces the parameter quantity and the calculation complexity of the model while retaining discriminative information to the maximum extent; finally, the processed features are input into a decoder for reconstruction, and enhanced speech is output. In this way, the speech enhancement quality in a complex noise environment is improved.

Description

Technical Field

[0001] This invention relates to the field of speech processing technology, and in particular to a speech enhancement method based on MP-SENet (Magnitude-Phase Squeeze-and-Excitation Network) with frequency domain channel attention and feature compression. Background Technology

[0002] Speech enhancement in complex environments is a key technology for improving speech recognition accuracy and communication quality. As a crucial component of speech signal processing, speech enhancement technology aims to recover clear speech from noisy speech signals, thereby improving speech intelligibility and perceived quality.

[0003] Traditional speech enhancement methods generally assume the stationarity of noise. When faced with complex, unknown, or non-stationary noise environments, these methods often struggle to accurately identify and separate noise from clean speech, potentially leading to distorted speech output. In recent years, with the rapid development of deep learning, deep learning-based speech enhancement methods have been extensively researched and explored, demonstrating good performance in various complex scenarios. However, most models based on Convolutional Neural Networks (CNNs) are limited by their local receptive fields, making it difficult to effectively model the global correlation of speech components in the frequency domain. This insufficient global modeling capability in the frequency domain results in inaccurate recovery of key speech components, particularly under low signal-to-noise ratio conditions and in complex noise scenarios, ultimately leading to poor overall speech enhancement performance. Summary of the Invention

[0004] Based on this, the present invention provides an MP-SENet speech enhancement method based on frequency domain channel attention and feature compression to improve the overall performance and computational efficiency of speech enhancement.

[0005] According to one aspect of the present invention, an MP-SENet speech enhancement method based on frequency domain channel attention and feature compression is provided, comprising: acquiring noisy speech; performing a short-time Fourier transform (STFT) on the noisy speech to obtain its amplitude spectrum and phase spectrum; stacking the amplitude spectrum and the phase spectrum to form a two-dimensional time-frequency feature map; inputting the two-dimensional time-frequency feature map as input features into a speech enhancement model; extracting features from the input features by the first convolutional block of the speech enhancement model, and performing joint feature filtering in the spatial and frequency domains using a spatial and channel reconstruction convolution (SCConv) to output a compressed feature map F1; and segmenting the feature map F1 along the channel dimension by the frequency channel attention network (FcaNet) of the speech enhancement model, assigning a two-dimensional (2D) discrete cosine transform to each group. The speech enhancement model uses a DCT (Distributed Transform) frequency component basis to extract multispectral vectors. After learning, channel attention weights are generated, and feature map F1 is recalibrated and enhanced to suppress unimportant channels and enhance important channels, resulting in feature map F2. The Dense Connected Convolutional Network (DenseNet) of the speech enhancement model takes feature map F2 as input and uses four convolutional layers with different dilation sizes to expand the receptive field on the time axis, employing dense connections to avoid the gradient vanishing problem, resulting in feature map F3. The second convolutional block of the speech enhancement model downsamples feature map F3, outputting a low-resolution feature map F4. Finally, the two-stage convolutional enhancement transformer of the speech enhancement model... The Transformer (TS-Conformers) captures the long-range dependencies of the feature map F4 in the time and frequency dimensions in stages, fuses local and global information, and outputs the feature map F5. The amplitude spectrum decoder of the speech enhancement model decodes the feature map F5 to obtain the enhanced amplitude spectrum, and the phase spectrum decoder decodes the feature map F5 to obtain the enhanced phase spectrum. Based on the enhanced amplitude spectrum and the enhanced phase spectrum, the enhanced speech is generated by the Inverse Short-Time Fourier Transform (ISTFT).

[0006] Optionally, the first convolutional block consists of a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function. The first convolutional block extracts features from the input features and uses SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. This includes: the 2D convolutional layer extracting features from the input features to obtain the first convolutional features; SCConv performing joint feature filtering in the spatial and frequency domains on the first convolutional features to output the first filtered features; the normalization layer normalizing the first filtered features to output the first normalized features; and the PReLU activation function activating the first normalized features to output the feature map F1.

[0007] Optionally, SCConv includes a Spatial Reconstruction Unit (SRU) and a Channel Reconstruction Unit (CRU). The SRU is used to separate the input features into informative features and redundant features using a learnable scaling factor, and to suppress spatial redundancy through cross-reconstruction operations, outputting spatially refined features. The CRU is used to apply segmentation, transformation, and fusion strategies to the spatially refined features, and to reduce channel redundancy through a combination of grouped convolution and point convolution as well as feature reuse, outputting channel-refined features.

[0008] Optionally, FcaNet segments the feature map F1 along the channel dimension, assigns a 2D DCT frequency component basis to each group, extracts a multi-spectral vector through DCT transformation, generates channel attention weights after learning, and recalibrates and enhances the feature map F1 to suppress unimportant channels and enhance important channels, outputting the feature map F2. This process includes: FcaNet segments the feature map F1 into n groups along the channel dimension; assigns a 2D DCT frequency component basis to each group based on the key frequency components of the speech signal; transforms the feature map of each group using its assigned 2D DCT frequency component basis to obtain its spectral vector; concatenates the spectral vectors corresponding to the feature maps of the n groups to obtain a multi-spectral vector; inputs the multi-spectral vector into the weight learning module to generate a channel attention weight vector; and multiplies the channel attention weight vector with the feature map F1 channel by channel to output the feature map F2.

[0009] Optionally, the weight learning module consists of a fully connected layer and a Sigmoid activation function.

[0010] According to another aspect of the present invention, an MP-SENet speech enhancement device based on frequency domain channel attention and feature compression is provided, comprising: an acquisition module for acquiring noisy speech; a processing module for performing short-time Fourier transform on the noisy speech to obtain its amplitude spectrum and phase spectrum; stacking the amplitude spectrum and phase spectrum to form a two-dimensional time-frequency feature map; inputting the two-dimensional time-frequency feature map as input features into a speech enhancement model; and a speech enhancement model comprising: a first convolutional block for extracting features from the input features and using SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1; and FcaNet for segmenting the feature map F1 along the channel dimension and assigning a 2D vector to each group. The DCT frequency component basis extracts multi-spectral vectors through DCT transform, generates channel attention weights after learning, and recalibrates and enhances feature map F1 to suppress unimportant channels and enhance important channels, outputting feature map F2. DenseNet is extended, using feature map F2 as input, employing four convolutional layers with different expansion sizes to expand the receptive field along the time axis, and using dense connections to avoid the gradient vanishing problem, outputting feature map F3. A second convolutional block downsamples feature map F3, outputting a low-resolution feature map F4. TS-Conformers are used to capture the long-range dependencies of feature map F4 in the time and frequency dimensions in stages, fusing local and global information, outputting feature map F5. An amplitude spectrum decoder decodes feature map F5 to obtain an enhanced amplitude spectrum. A phase spectrum decoder decodes feature map F5 to obtain an enhanced phase spectrum. The processing module also generates enhanced speech based on the enhanced amplitude and phase spectra through inverse short-time Fourier transform.

[0011] Optionally, the first convolutional block consists of a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function, wherein: the 2D convolutional layer is used to extract features from the input features to obtain the first convolutional features; SCConv is used to perform joint feature filtering in the spatial and frequency domains on the first convolutional features to output the first filtered features; the normalization layer is used to normalize the first filtered features to output the first normalized features; and the PReLU activation function is used to activate the first normalized features to output the feature map F1.

[0012] Optionally, FcaNet is specifically used to: divide the feature map F1 into n groups along the channel dimension; assign a 2D DCT frequency component basis to each group based on the key frequency components of the speech signal; transform the feature map of each group using the assigned 2D DCT frequency component basis to obtain its spectral vector; concatenate the spectral vectors corresponding to the feature maps of the n groups to obtain a multi-spectral vector; input the multi-spectral vector into the weight learning module to generate a channel attention weight vector; and multiply the channel attention weight vector with the feature map F1 channel by channel to output the feature map F2.

[0013] Optionally, the weight learning module consists of a fully connected layer and a Sigmoid activation function.

[0014] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; and a memory storing a program, wherein the program includes instructions that, when executed by the processor, cause the processor to perform any of the above-described MP-SENet speech enhancement methods based on frequency domain channel attention and feature compression.

[0015] According to another aspect of the present invention, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to perform the MP-SENet speech enhancement method based on frequency domain channel attention and feature compression as described above.

[0016] The present invention provides one or more technical solutions. The method first performs a short-time Fourier transform on the noisy speech to obtain its amplitude and phase spectra. The amplitude and phase spectra are then stacked to form a two-dimensional time-frequency feature map, which is input to the encoder of an MP-SENet model incorporating FcaNet and SCConv. FcaNet, through discrete cosine transform and frequency domain channel attention mechanisms, achieves adaptive feature compression and restoration, effectively alleviating the feature loss problem in MP-SENet and retaining more low-frequency and key frequency band information during feature compression, thus improving the global modeling and feature representation capabilities of key speech components. Furthermore, SCConv can perform convolution operations in the spatial and channel domains, adaptively compressing redundant spatial and channel features. While maximizing the retention of discriminative information, it reduces the number of model parameters and computational complexity, improving the model's robustness in handling complex noise. Finally, the processed features are input into the decoder for reconstruction, outputting enhanced speech. This improves the overall performance of speech enhancement in complex noisy environments and enhances the quality of speech enhancement. Attached Figure Description

[0017] Further details, features, and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1This is a schematic diagram of an FcaNet architecture; Figure 2 This is a schematic diagram of an SCConv architecture; Figure 3 A schematic diagram of the structure of the speech enhancement model provided by the present invention; Figure 4 This is a flowchart illustrating an MP-SENet speech enhancement method based on frequency domain channel attention and feature compression provided by the present invention. Figure 5 This is a schematic diagram of a training process for the speech enhancement model in this invention; Figures 6(a) and 6(b) are spectrograms of noisy samples in the VoiceBank+DEMAND dataset used in this invention; Figures 7(a) and 7(b) are spectrograms of clean speech samples from the VoiceBank+DEMAND dataset used in this invention; Figure 8(a) and Figure 8(b) are spectrograms of the enhanced speech obtained using the MP-SENet-based method; Figures 9(a) and 9(b) are spectrograms of the enhanced speech obtained using the speech enhancement method provided by the present invention; Figure 10 PESQ variation with training steps for the proposed method and the MP-SENet-based method in marine environment speech enhancement scenarios; Figure 11(a) shows the spectrogram of noisy speech enhanced using the MP-SENet-based method; Figure 11(b) is a spectrogram of noisy speech enhanced using the speech enhancement method provided by the present invention; Figure 12 This is a schematic diagram of the structure of the MP-SENet speech enhancement device based on frequency domain channel attention and feature compression provided by the present invention. Figure 13 A structural block diagram of an exemplary electronic device that can be used to implement embodiments of the present invention is shown. Detailed Implementation

[0018] Embodiments of the present invention will now be described in more detail with reference to the accompanying drawings. While some embodiments of the invention are shown in the drawings, it should be understood that the invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the invention. It should be understood that the accompanying drawings and embodiments are for illustrative purposes only and are not intended to limit the scope of protection of the invention.

[0019] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0020] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first", "second", etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0021] It should be noted that the terms "a" and "a plurality of" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0022] MP-SENet is a speech enhancement model based on parallel amplitude and phase spectrum denoising. The MP-SENet model has an overall encoder-decoder structure. The encoder encodes the input noisy amplitude and phase spectra into time-frequency domain representations. The parallel amplitude mask decoder and phase decoder decode the clean amplitude and phase spectra from the time-frequency domain representations, respectively. After reconstructing the short-time spectrum, the clean speech waveform is obtained through inverse short-time Fourier transform.

[0023] In some embodiments of this invention, FcaNet is introduced to address the shortcomings of MP-SENet in frequency domain feature extraction. FcaNet converts time-domain features into frequency-domain features through DCT, effectively capturing the periodic changes and spectral features of speech signals, especially performing better in low signal-to-noise ratio scenarios. Compared to MP-SENet's pure convolution extraction method, FcaNet provides an effective representation of global frequency information and local frequency components, enhancing the modeling ability of complex noise frequency features. Furthermore, FcaNet introduces a frequency-domain channel attention mechanism, which performs weighted operations based on the importance of features in different frequency bands, achieving adaptive feature compression and restoration. This dynamic feature weighting strategy not only effectively alleviates the feature loss problem in MP-SENet but also retains more low-frequency and key frequency band information during feature compression. Combining DCT and the attention mechanism, FcaNet improves feature representation capabilities, particularly in low-frequency noise suppression and complex spectrum recovery.

[0024] Figure 1This is a schematic diagram of an FcaNet architecture, such as... Figure 1 As shown, FcaNet first receives the feature tensor. X (Feature map F1) is used as input, and the size of the tensor is... C×H×W The tensor is divided along the channel dimension into n There are several parts, each with a shape of [missing information]. ×H×W ,in, The segmented parts X i （i= 0,1,…, n -1 ） Using 2D DCT (DCT0...DCT) n ), to obtain the corresponding spectrum vector F i ( F 1 …F n ), and concatenate all the spectral vectors into a multispectral vector. F In the context of X i During 2DDCT, the spectral vector for each channel can be determined using three modes: Low Frequency (LF), Two-Step Selection (TS), and Neural Architecture Search (NAS). Fully connected layers are used to analyze the multiple spectral vectors. Freq The process yields a compressed vector, which is then converted into an attention weight vector using the Sigmoid function. ms att The original feature tensor is obtained by using vector pairs. X Perform channel weighting to obtain the final result tensor. .

[0025] In some embodiments of this invention, to address the spatial feature modeling and feature redundancy issues of MP-SENet, SCConv is employed for improvement. SCConv is a flexible convolutional module capable of performing convolution operations in both the spatial and channel domains, enabling joint modeling of spatial and channel features. Simultaneously with feature extraction, spatial and channel attention weights are generated to adaptively highlight information-dense feature channels, enhance information-rich feature channels, and eliminate low-information features, thereby achieving feature compression. This mechanism effectively compensates for MP-SENet's failure to perform spatial feature extraction and poor multi-scale adaptability, improving robustness to noise in complex scenes.

[0026] Figure 2 This is a schematic diagram of an SCConv architecture, such as... Figure 2 As shown, SCConv consists of two units, SRU and CRU, designed to reduce feature redundancy in CNNs by simultaneously addressing spatial and channel redundancy issues. The SRU and CRU units work sequentially to enhance feature representation and reduce computational complexity. SRU addresses spatial redundancy by segmenting information features. Specifically, the input to the SRU is the first convolutional feature output from a 2D convolutional layer. First, the group normalization (GN) layer in the SRU normalizes the first convolutional feature by grouping it, reducing intra-group differences. The SRU can then determine the importance of sub-convolutional features within a channel through the GN layer, i.e., determine the spatial importance of each channel. Subsequently, the first convolutional feature is reconstructed based on this spatial importance to reduce redundancy and enhance spatial information. Specifically, the GN layer contains a trainable scaling factor. γ The scaling factor γ It can measure the spatial variance of each channel; the larger the variance, the richer the spatial information. A scaling factor for this parameter can then be set. γ Perform normalization weight transformation, γ Convert to normalized correlation weights The conversion process can be represented by the following formula (1).

[0027]

[0028] in, W γ This is a normalized weight vector containing multiple normalized weights. w i , γ i For channel i The scaling factor in i， j All are channel indexes. C This represents the number of channels.

[0029] For normalized weights The normalized weights can be mapped to the (0,1) interval using the Sigmoid function, and then binarized according to a preset weight threshold to obtain the information weights. and non-information weights The weight threshold can be set according to the actual application scenario, such as 0.5. For example, the normalized weight value that is higher than the preset weight threshold can be set as the preset information weight value, and the normalized weight that is not higher than the preset weight threshold can be set as the preset non-information weight value. Both the information weight value and the non-information weight value can be set according to the actual application scenario, such as setting the information weight to 1 and the non-information weight to 0.

[0030] By multiplying the first convolutional features element-wise with both informational and non-informational weights, we obtain an information-rich feature vector. And redundant feature vectors with less information This process can be represented by the following formulas (2) and (3).

[0031]

[0032]

[0033] in, W 1 represents the information weight. W 2 represents non-information weights. X The first convolutional feature, This indicates element-wise multiplication.

[0034] The informative feature vectors can then be merged with the redundant feature vectors. The conventional method for feature merging is to directly add the two feature vectors, but this direct addition may result in the loss of some information. Therefore, to avoid information loss due to direct addition, this invention employs a cross-reconstruction operation to... and By swapping and adding the corresponding parts in the original text, we get... and ,Will as well as The features are merged to form a spatially reconstructed feature matrix, i.e., spatially refined features. .

[0035] Specifically, the informative feature vector and the redundant feature vector can be split separately to obtain a first informative feature component, a second informative feature component, a first redundant feature component, and a second redundant feature component. The first informative feature component and the second redundant feature component are concatenated to obtain a first reconstructed vector. The second informative feature component and the first redundant feature component are concatenated to obtain a second reconstructed vector. The splitting of the informative feature vector and the redundant feature vector can be performed at any splitting ratio, as long as the dimensions of the first informative feature component and the first redundant feature component are the same, and the dimensions of the second informative feature component and the second redundant feature component are the same. The above process can be represented by the following formulas (4), (5), and (6).

[0036]

[0037]

[0038]

[0039] in, The first information feature component, This is the second information feature component. This is the first redundant feature component. This is the second redundant feature component. The first reconstructed vector, For the second reconstructed vector, This indicates element-wise addition. This indicates the concatenation of elements.

[0040] The spatially refined features after spatial reconstruction output by the SRU can be input into the CRU. The CRU reduces channel redundancy through channel segmentation, transformation, and fusion strategies, extracts rich representative features using channel attention mechanisms and lightweight convolution operations, and reduces computational costs through a combination of group convolution and point convolution as well as feature reuse methods, thereby improving the computational efficiency and performance of the network.

[0041] Specifically, CRU can be configured according to a preset channel ratio. α The spatially reconstructed feature matrix is ​​segmented along the channel dimension, and the channel ratio is... α This is a parameter between 0 and 1, and its specific value can be set according to the actual application scenario; for example, it can be set to 0.5. Based on the number of channels... C For example, based on channel ratio α After segmenting the spatially reconstructed feature matrix, two parts can be obtained, one of which contains... One channel, the other part contains For ease of description, this invention refers to the two segments as the first channel feature and the second channel feature. Subsequently, channel compression can be applied to the first and second channel features. Specifically, a 1×1 convolution (1×1 Conv) can be used to compress the segmented channels, reducing the number of channels and improving computational efficiency. For ease of description, in this invention, the feature obtained after channel compression of the first channel feature is referred to as the upper part feature. The feature obtained after channel compression of the second channel feature is called the lower part feature. .

[0042] Regarding the above features This can be processed using group-wise convolution (GWC) and 1×1 point-wise convolution (PWC). GWC connects each output channel to a specific group of input channels, reducing parameters and computation. 1×1 point-wise convolution compensates for potential information loss caused by GWC and allows information to flow between channels. The output features of GWC and PWC are then added together to obtain the first merged feature. .

[0043] For the lower part of the features It is possible to generate a feature matrix with detailed information using 1×1 PwC. Y 2, and reuse To obtain more features without increasing additional computational cost, such as through residual connections, X low If passed to subsequent stages, such as... X low and Y 2. Add each element one by one to obtain the second merging feature.

[0044] The first and second merged features can then be merged. As one possible implementation, global average pooling (GAP) can be performed on the first and second merged features respectively to obtain the first channel descriptor of the first merged feature and the second channel descriptor of the second merged feature. Softmax is then used to calculate the first and second channel descriptors to obtain the first and second importance vectors. Based on the first and second importance vectors, the first and second merged features are weighted and fused to obtain the channel refined features.

[0045] In this invention, the output features of the up-and-down transformation stages can be combined using a simplified Selective Kernel Networks (SKNet) method. and Specifically, it first collects data through global average pooling. and From the global spatial information, the channel description is obtained. and The calculation process is shown in formula (7), where H, W This represents the size of the characteristic matrix.

[0046]

[0047] Then use the channel descriptor and Calculate the feature importance vector using SoftMax. and And based on this, and Weighted fusion is performed along the channel dimension to obtain the feature matrix after channel reconstruction. That is, the channel refinement feature, as shown in formula (8).

[0048]

[0049] Since the codec architecture can effectively extract deep features of speech signals and reconstruct high-quality output, exhibiting good robustness and generalization ability in various noise environments, and has been widely applied in the field of speech enhancement, the speech enhancement model used in this invention can be built based on the codec architecture, aiming to achieve efficient and accurate speech enhancement effects. Specifically, the speech enhancement model is mainly optimized based on the MP-SENet architecture. While MP-SENet improves computational efficiency by processing amplitude and phase spectra in parallel, defines multi-level loss functions, and optimizes the model at different levels, resulting in excellent performance on multiple objective metrics, its computational load is large and training is difficult. Therefore, this invention uses FcaNet and SCConv to optimize MP-SENet, performing frequency domain discriminative enhancement and spatial and channel redundancy optimization simultaneously with feature extraction, achieving a synergistic improvement in computational efficiency and enhancement performance.

[0050] Model Implementation Figure 3 A schematic diagram of the structure of the speech enhancement model provided by the present invention is shown below. Figure 3 As shown, the speech enhancement model can include an encoder and a decoder, as well as TS-Conformers connected between the encoder and decoder. When using the speech enhancement model, noisy speech is acquired, and a short-time Fourier transform is performed on the noisy speech to obtain its amplitude spectrum and phase spectrum. The amplitude spectrum and phase spectrum are stacked to form a two-dimensional time-frequency feature map. This two-dimensional time-frequency feature map is used as input features to the speech enhancement model, which outputs enhanced amplitude and phase spectra. Based on the enhanced amplitude and phase spectra, enhanced speech is generated through inverse short-time Fourier transform.

[0051] The encoder consists of convolutional blocks connecting both ends of an FcaNet and an extended DenseNet, used to encode the input two-dimensional time-frequency feature map as a compressed time-frequency domain representation. Specifically, as... Figure 3As shown, the encoder includes a first convolutional block, FcaNet, dilated DenseNet, and a second convolutional block. The first convolutional block extracts features from the input and uses SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. FcaNet segments feature map F1 along the channel dimension, assigns a 2D DCT frequency component basis to each group, extracts multi-spectral vectors through DCT transformation, generates channel attention weights after learning, and recalibrates and enhances feature map F1 to suppress unimportant channels and enhance important channels, outputting feature map F2. Dilated DenseNet uses feature map F2 as input, employs four convolutional layers with different dilation sizes to expand the receptive field along the time axis, and uses dense connections to avoid the gradient vanishing problem, outputting feature map F3. The second convolutional block downsamples feature map F3, outputting a low-resolution feature map F4. TS-Conformers captures the long-range dependencies of feature map F4 in the time and frequency dimensions in stages, fusing local and global information, outputting feature map F5.

[0052] The first convolutional block increases the feature dimension by increasing the number of channels in the convolutional layer. SCConv filters the spatial and frequency domain features in the first convolutional block, retaining key features and deleting invalid features to ensure feature efficiency. Figure 3 As shown, the first convolutional block includes a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function. The 2D convolutional layer extracts features from the input features, yielding the first convolutional features. SCConv performs joint feature filtering in both the spatial and frequency domains on the first convolutional features, outputting the first filtered features. The normalization layer normalizes the first filtered features, outputting the first normalized features. The PReLU activation function activates the first normalized features, outputting the feature map F1. The 2D convolutional layer increases the feature dimension by increasing the number of channels, and then uses this 2D convolutional layer to extract features from the input features, yielding the first convolutional features containing rich information. SCConv performs joint filtering of the spatial and frequency domain features in the first convolutional block, retaining key features and removing invalid features to ensure feature efficiency. Specifically, as shown... Figure 2 As shown, SCConv includes SRU and CRU. SRU uses a learnable scaling factor to separate input features into informative and redundant features, and suppresses spatial redundancy through cross-reconstruction operations, outputting spatially refined features. CRU applies segmentation, transformation, and fusion strategies to the spatially refined features, reducing channel redundancy through a combination of grouped convolutions and point convolutions, as well as feature reuse, outputting channel-refined features, i.e., the first selection features. The PReLU activation function activates the first normalized features, obtaining the feature map F1. The second convolutional block downsamples features by expanding the stride of the convolutional layer.

[0053] FcaNet is responsible for effectively compressing the extended feature dimensions and embedding frequency channel information. Specifically, FcaNet is used to: divide the feature map F1 into n groups along the channel dimension; assign a 2D DCT frequency component basis to each group based on the key frequency components of the speech signal; transform the feature map of each group using its assigned 2D DCT frequency component basis to obtain its spectral vector; concatenate the spectral vectors corresponding to the feature maps of the n groups to obtain a multi-spectral vector; input the multi-spectral vector into the weight learning module to generate channel attention weight vectors; and multiply the channel attention weight vectors with the feature map F1 channel by channel to output the feature map F2. The extended DenseNet uses four convolutional layers with different expansion sizes to expand the receptive field on the time axis and employs dense connections to avoid the gradient vanishing problem.

[0054] The second convolutional block downsamples features by expanding the stride of the convolutional layer. For example... Figure 3 As shown, the second convolutional block in the encoder consists of a 2D convolutional layer, a normalization layer, and a PReLU activation function.

[0055] The features output by the encoder are further processed by N TS-Conformer units. The TS-Conformer combines the advantages of CNN and Transformer, and can capture local details and global sequence information at the same time, further optimizing feature quality and providing better input for the decoder.

[0056] like Figure 3 As shown, the decoder includes an amplitude spectrum decoder and a phase spectrum decoder. The amplitude spectrum decoder is used to decode the feature map F5 to obtain the enhanced amplitude spectrum. In this embodiment, the phase spectrum decoder is used to decode the feature map F5 to obtain the enhanced phase spectrum. The speech enhancement model adopts a dual-decoder architecture, processing amplitude and phase respectively. The input of the amplitude spectrum decoder is the features processed by TS-Conformers, and the output is the enhanced amplitude spectrum, i.e., the clean speech amplitude after denoising. The input of the phase spectrum decoder is also the features processed by the TS-Conformers unit, and the output is the enhanced phase spectrum, i.e., the optimized phase information.

[0057] By combining the enhanced amplitude spectrum and the enhanced phase spectrum, a complex spectrum is reconstructed. An inverse short-time Fourier transform is performed on the complex spectrum to obtain a time-domain waveform. Finally, a denoised, clear, and natural time-domain speech waveform is output, which is the enhanced speech.

[0058] Speech enhancement method based on the speech enhancement model of this invention This invention provides an MP-SENet speech enhancement method based on frequency domain channel attention and feature compression. This method uses... Figure 3 The speech enhancement model shown performs speech enhancement processing.

[0059] Figure 4 This is a flowchart illustrating an MP-SENet speech enhancement method based on frequency domain channel attention and feature compression provided by the present invention. Figure 4 As shown, the following steps may be included: S401, Acquire noisy speech; S402. Perform a short-time Fourier transform on the noisy speech to obtain its amplitude spectrum and phase spectrum. S403. Stack the amplitude spectrum and phase spectrum to form a two-dimensional time-frequency feature map; S404. Input the two-dimensional time-frequency feature map as input feature into the speech enhancement model; S405. The first convolutional block of the speech enhancement model extracts features from the input features and uses SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. S406. The FcaNet speech enhancement model segments the feature map F1 along the channel dimension, assigns a 2D DCT frequency component basis to each group, extracts multi-spectral vectors through DCT transformation, generates channel attention weights after learning, recalibrates and enhances the feature map F1 to suppress unimportant channels and enhance important channels, and outputs the feature map F2. S407. The DenseNet extended by the speech enhancement model takes feature map F2 as input, uses four convolutional layers with different expansion sizes to expand the receptive field on the time axis, and adopts dense connections to avoid the gradient vanishing problem, and outputs feature map F3. S408. The second convolutional block of the speech enhancement model downsamples the feature map F3 and outputs a low-resolution feature map F4. S409. The TS-Conformers, enhanced by two-stage convolution of the speech enhancement model, captures the long-range dependencies of feature map F4 in the time and frequency dimensions in stages, and fuses local and global information to output feature map F5. S410. The amplitude spectrum decoder of the speech enhancement model decodes the feature map F5 to obtain the enhanced amplitude spectrum, and the phase spectrum decoder decodes the feature map F5 to obtain the enhanced phase spectrum. S411. Based on the enhanced amplitude spectrum and enhanced phase spectrum, the enhanced speech is generated by inverse short-time Fourier transform.

[0060] In this embodiment of the invention, the noisy speech is first subjected to a short-time Fourier transform to obtain its amplitude and phase spectra. The amplitude and phase spectra are then stacked to form a two-dimensional time-frequency feature map, which is input to the encoder of the MP-SENet model incorporating FcaNet and SCConv. FcaNet, through discrete cosine transform and frequency domain channel attention mechanisms, achieves adaptive feature compression and restoration, effectively alleviating the feature loss problem in MP-SENet and retaining more low-frequency and key frequency band information during feature compression, thus improving the global modeling and feature representation capabilities of key speech components. Furthermore, SCConv can perform convolution operations in both the spatial and channel domains, adaptively compressing redundant spatial and channel features. This maximizes the retention of discriminative information while reducing the number of model parameters and computational complexity, improving the model's robustness in handling complex noise. Finally, the processed features are input into the decoder for reconstruction, outputting enhanced speech. This improves the overall performance of speech enhancement in complex noisy environments and enhances the quality of speech enhancement.

[0061] Noisy speech typically refers to speech containing noise that is directly collected from a sound scene. In this invention, noisy speech can be collected in any feasible way. For example, multiple microphones can be used to collect multiple speech signals to obtain noisy speech, or a single microphone can be used to collect speech signals as noisy speech. Noisy speech can include amplitude information and phase information of the speech. The amplitude information reflects the strength, volume, and other characteristics of the speech, while the phase information reflects the position and shape of the speech in the time dimension.

[0062] When the clean speech signal and the noisy speech signal are independent of each other, the single-channel speech enhancement problem can be represented by formula (9).

[0063]

[0064] in, t Indicates the index value of the time frame. Indicates a pure speech signal. This indicates a noisy speech signal. It's a noisy speech signal. The purpose of speech enhancement is to improve the performance of noisy speech signals. Remove noise signals And extract the pure speech signal. .

[0065] Before inputting the noisy speech into the speech enhancement model, the noisy speech can be spectrally decomposed to obtain the amplitude spectrum and phase spectrum of the noisy speech; for example, the noisy speech in the time domain can be converted to the frequency domain by using short-time Fourier transform, which can be expressed by formula (10).

[0066]

[0067] in, , and They are , and The spectrum diagram, This represents the frequency index value.

[0068] The amplitude spectrum and phase spectrum can then be stacked to form a two-dimensional time-frequency feature map, which is then used as input features into the speech enhancement model. Alternatively, the two independent two-dimensional matrices of amplitude spectrum and phase spectrum can be concatenated along the channel dimension to form a dual-channel two-dimensional time-frequency feature map, which can then be input into the speech enhancement model of this embodiment.

[0069] In one possible embodiment, such as Figure 3 As shown, the first convolutional block includes: a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function. The first convolutional block extracts features from the input features and performs joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. This includes: a 2D convolutional layer extracting features from the input features to obtain the first convolutional features; SCConv performing joint feature filtering in the spatial and frequency domains on the first convolutional features to output the first filtered features; a normalization layer normalizing the first filtered features to output the first normalized features; and a PReLU activation function activating the first normalized features to output the feature map F1.

[0070] Specifically, a 2D convolutional layer can be a 2D convolutional layer. The number of channels in a 2D convolutional layer can be selected according to the actual application scenario. As one possible implementation, the feature dimension can be increased by increasing the number of channels in the 2D convolutional layer to extract richer feature information. SCConv performs joint feature filtering in the spatial and frequency domains on the first convolutional features extracted by the 2D convolutional layer. This can be done by filtering feature information in each channel and filtering individual channels, achieving feature compression, retaining key features, deleting invalid features, and ensuring feature efficiency.

[0071] SCConv consists of two units: SRU and CRU. The SRU and CRU units operate sequentially to enhance feature representation and reduce computational complexity. The SRU uses a learnable scaling factor to separate input features into informative and redundant features, and suppresses spatial redundancy through cross-reconstruction operations, outputting spatially refined features. The CRU applies segmentation, transformation, and fusion strategies to the spatially refined features, reducing channel redundancy through a combination of grouped convolutions and point convolutions, as well as feature reuse, outputting channel-refined features. The specific computational process of SCConv has been detailed above and will not be repeated here.

[0072] In one possible embodiment, FcaNet segments the feature map F1 along the channel dimension, assigns a 2D DCT frequency component basis to each group, extracts multi-spectral vectors through DCT transform, generates channel attention weights after learning, recalibrates and enhances the feature map F1 to suppress unimportant channels and enhance important channels, and outputs the feature map F2, including: S461 and FcaNet divide the feature map F1 into n groups along the channel dimension; S462. Based on the key frequency components of the speech signal, assign a 2D DCT frequency component basis to each group; S463. For the feature map of each group, transform it using the 2D DCT frequency component basis assigned to it to obtain its spectrum vector. S464. Concatenate the spectral vectors corresponding to the feature maps of n groups to obtain a multi-spectral vector; S465. Input the multi-spectral vector into the weight learning module to generate the channel attention weight vector; optionally, the weight learning module consists of a fully connected layer and a Sigmoid activation function.

[0073] S466. Multiply the channel attention weight vector with the feature map F1 channel by channel to output the feature map F2.

[0074] FcaNet, a frequency domain channel attention network, is a deep learning module that improves the channel attention mechanism from the frequency domain perspective. Its core is to replace the traditional global average pooling with multi-frequency domain components to generate richer channel descriptors, thereby improving the expressive power of channel attention.

[0075] Traditional speech enhancement networks are typically built on CNN models, which consist of input layers, convolutional layers, activation functions, pooling layers, fully connected layers, and output layers. CNNs have achieved significant results in tasks such as image classification and object detection, and have increasingly been applied to speech recognition in recent years. However, CNN models usually require a large number of parameters and computational resources, with many invalid and redundant parameter calculations. With the development of deep neural networks, attention mechanisms, especially channel attention mechanisms, have achieved great success in computer vision, and recent advancements in CNNs have significantly benefited from channel attention mechanisms. Channel attention mechanisms can effectively reduce the number of parameters and computational resources through methods such as feature compression using global pooling, dimensionality reduction during the excitation phase, and sharing global parameter weights for each channel. However, when representing channels with scalar values, traditional methods often face challenges due to significant information loss. To address this issue, this invention introduces FcaNet, which defines channel attention from a frequency analysis perspective.

[0076] FcaNet treats channel representation as a compression process, using Discrete Cosine Transform (DCT) to capture and preserve fundamental frequency components. Key information in speech (such as formants distinguishing different phonemes and the fundamental frequency profile of intonation) exhibits both local concentration and global structure in the frequency domain. Global Average Pooling (GAP) assumes all frequency channels are equally important, which contradicts the physical reality of speech signals. DCT, as an efficient energy compression tool, can construct a filter bank covering different frequency bands using its different frequency bases, thus providing the model with a priori multi-band analysis perspective that conforms to the energy distribution characteristics of speech signals. DCT enables the model to simultaneously establish a global frequency domain understanding of speech signals from macroscopic to microscopic levels with extremely low computational cost, overcoming the limitations of CNNs, such as limited receptive fields and the need for stacking multiple layers to establish long-range dependencies. FcaNet generalizes the traditional GAP, revealing it as a special case of DCT, and extends its capabilities by introducing multi-spectral channel attention. By effectively utilizing multiple frequency components, it achieves better performance without increasing computational cost.

[0077] In S461, the feature map F1 can be segmented along the channel dimension according to any size. For example, in a feature map F1 with a size of... C×H×W In this case, C For the number of channels, H For feature height, W The feature width is obtained by segmenting along the channel dimension. n The dimension of each segmentation and reconstruction feature is ,in, .

[0078] For each set of feature maps, a 2D DCT frequency component basis can be assigned to each set based on the key frequency components of the speech signal. The assigned 2D DCT frequency component basis is then used for transformation to obtain its spectral vector. The 2D DCT frequency component basis typically refers to a set of cosine wave functions on the feature map. Performing 2D DCT on the feature map is essentially fitting the feature map using the 2D DCT frequency component basis. The key frequencies of the speech signal are the frequency range that have the greatest impact on speech intelligibility and naturalness. In this invention, a two-step selection (TS) method can be used to select a preset number of frequency components with high contribution in each channel to obtain the key frequency components of each set of feature maps. The TS mode can evaluate the contribution of each frequency component to the attention channel, thereby selecting the frequency component with the highest contribution in the channel. The number of frequency components is less than the size of the segmented and reconstructed features, i.e., the aforementioned preset number is less than... H×W .

[0079] The spectral vectors corresponding to the feature maps of n groups are concatenated to obtain a multi-spectral vector. FreqThe multi-spectral vector is input into the weight learning module to generate channel attention weight vectors. The weight learning module consists of fully connected layers and a sigmoid activation function, using the multi-spectral vector... Freq The dimension is Taking a preset number as an example, the number of channels can be compressed using a fully connected layer. The fully connected layer can compress the number of channels according to a preset compression ratio. Compression reduces computational cost, allowing the model to learn more core information while preventing overfitting. The Sigmod function maps the compressed vector to the (0,1) range, thus obtaining the attention weight vector. ms att .

[0080] By multiplying the feature map F1 element-wise by the attention weight vector, channel weighting of feature map F1 can be achieved, resulting in feature map F2.

[0081] Dilated DenseNet is a classic deep learning architecture whose core feature is dense connections. This allows each layer to directly receive the outputs of all preceding layers, significantly mitigating the vanishing gradient problem, enhancing feature reuse, and improving model efficiency. In one possible implementation, Dilated DenseNet uses four convolutional layers with different dilation sizes to expand the receptive field along the time axis and employs dense connections to avoid the vanishing gradient problem, resulting in feature map F3.

[0082] The second convolutional block is used to downsample the feature map F3, outputting a low-resolution feature map F4. Specifically, the second convolutional block can contain a 2D convolutional layer, a normalization layer, and a PReLU activation function. The 2D convolutional layer is a 2D convolutional layer that can downsample features by expanding the stride of the convolutional layer. The normalization layer normalizes the second convolutional features to obtain the second normalized features. The PReLU activation function activates the second normalized features to obtain the low-resolution feature map F4.

[0083] like Figure 3 As shown, the speech enhancement model also includes TS-Conformers. Before the decoder performs the decoding operation, the feature map F4 is first split into N split data, where N is a positive integer. These multiple split data are input into the TS-Conformers, which generate N TS-Conformer computation units. Each TS-Conformer computation unit performs TS-Conformer computation on the input split data to obtain the feature map F5. The amplitude spectrum decoder obtains the enhanced amplitude spectrum based on feature map F5, and the phase spectrum decoder obtains the enhanced phase spectrum based on feature map F5.

[0084] Speech data is typically long-term, and the corresponding feature map F4 is also usually long-term. To avoid memory explosion when Transformer-type models process long-term features, the feature map F4 can be segmented. For example, it can be divided into N segments with a uniform segment size along the temporal dimension, resulting in N split data. The TS-Conformers computation module can generate N TS-Conformers computation units for the N split data to utilize the parallel computing capabilities of the GPU for parallel processing of the split data, thereby improving processing speed. The TS-Conformers computation unit adopts a time-frequency two-stage serial structure; each stage includes layer normalization, multi-head self-attention, convolutional enhancement modules, and a feedforward network, and constructs gradient identity paths through residual connections. This unit models long-term temporal dependencies in the time stage and cross-band correlations in the frequency stage, combined with the local feature extraction capabilities of the convolutional enhancement module, to achieve joint modeling of global and local information of time-frequency features.

[0085] The decoder can include an amplitude spectrum decoder and a phase spectrum decoder, where the amplitude spectrum decoder is specifically an amplitude mask decoder. The amplitude mask decoder predicts the amplitude mask from the time-frequency domain representation and obtains a clean, enhanced amplitude spectrum by multiplying the mask by the noise amplitude spectrum. In one possible embodiment, the amplitude decoder first uses an expanded DenseNet to obtain an estimated compressed mask from an upsampling deconvolution block consisting of 2D transposed convolutional layers, normalization layers, and PReLU activation layers, along with the amplitude mask estimation architecture. Specifically, the amplitude decoder first extracts features using an expanded DenseNet, then amplifies the features through an upsampling deconvolution block (containing 2D transposed convolution + normalization + PReLU), and finally outputs a compressed mask. This compressed mask is a smaller, more channel-rich, and more information-condensed mask.

[0086] To predict the amplitude mask more accurately, the clean amplitude spectrum can be calculated based on the compression factor. and noise amplitude spectrum Power-law compression is performed to obtain the predicted target. The compression factor is The specific value can be set according to the actual application scenario. In this invention, it can be set to 0.3 according to the experimental results to narrow the prediction range of the mask and make it easier to predict. Specifically, the prediction target can be calculated by the following formula (11).

[0087]

[0088] To achieve more accurate predictions, a learnable sigmoid (LSigmoid) function can be used to predict the compressed amplitude mask. The calculation formula is shown in formula (12), where... This is a preset parameter, which can be set to 2 based on experimental results. These are trainable parameters that allow the model to adaptively change the shape of the activation function in different frequency bands.

[0089]

[0090] The predicted target is activated by the LSigmoid function to obtain the estimated compression mask. Then, using compressed mask decoding and noise amplitude spectrum... Element-wise multiplication yields the enhanced amplitude spectrum. The specific calculation is shown in formula (13).

[0091]

[0092] The phase decoder directly predicts a clean phase spectrum from the time-frequency domain representation. To overcome the difficulties caused by the unstructured and encapsulated characteristics of the phase, the phase decoder can employ a parallel phase estimation architecture. Specifically, this architecture first uses two parallel 2D convolutional layers to output pseudo-real parts. and pseudo-virtual part Then, the phase spectrum of the clean package is predicted by the two-parameter arctangent function. The specific calculation is shown in formulas (14) and (15).

[0093]

[0094]

[0095] That is, use first arctan The imaginary / real part is used to obtain a basic phase value, which is then corrected using subsequent correction terms. π arrive π Within the standard wrapping range, the phase wrapping problem is solved. Formula (15) is used to determine the sign of the pseudo-real part and the pseudo-imaginary part. It is the core of the correction term in formula (14) and is used to correct the result of the arctangent to the correct phase interval based on the quadrant information.

[0096] The enhanced speech can be obtained by performing an inverse short-time Fourier transform on the enhanced amplitude spectrum and the cleaned phase spectrum.

[0097] Model training In one possible implementation, the speech enhancement model can be trained through the following steps: S501. Obtain the speech enhancement dataset, which includes clean speech and noisy speech under various noise environments. Input noisy speech; S502. Perform a short-time Fourier transform on the noisy speech to obtain the amplitude spectrum and phase spectrum of the noisy speech. S503. Input the amplitude spectrum and phase spectrum into the speech enhancement model. The speech enhancement model uses MP-SENet as the baseline model and includes an encoder and a decoder, as well as TS-Conformers connected between the encoder and the decoder. The encoder includes a first convolutional block, FcaNet, dilated DenseNet and a second convolutional block. The first convolutional block includes SCConv. The decoder includes a phase spectrum decoder and an amplitude spectrum decoder. S504: The first convolutional block extracts features from the input features and uses SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1; FcaNet segments the feature map F1 along the channel dimension, assigning a 2D array to each group. The DCT frequency component basis extracts multi-spectral vectors through DCT transform, which are then learned to generate channel attention weights. These weights are used to recalibrate and enhance feature map F1, suppressing unimportant channels and enhancing important channels, resulting in feature map F2. DenseNet, taking feature map F2 as input, uses four convolutional layers with different dilation sizes to expand the receptive field along the time axis, employing dense connections to avoid the gradient vanishing problem, resulting in feature map F3. A second convolutional block downsamples feature map F3, outputting a low-resolution feature map F4. TS-Conformers capture the long-range dependencies of feature map F4 in the time and frequency dimensions in stages, fusing local and global information to output feature map F5. The amplitude spectrum decoder decodes feature map F5 to obtain the enhanced amplitude spectrum, and the phase spectrum decoder decodes feature map F5 to obtain the enhanced phase spectrum. Based on the enhanced amplitude and phase spectra, enhanced speech is generated through inverse short-time Fourier transform. S505. Calculate the loss function. Compare the generated enhanced speech with the real clean speech and calculate the mixture loss function value between the two to measure the difference between the model's actual output and the real clean speech. The target mixture loss function includes a time loss function, an amplitude loss function, and a complex loss function. S506. Update the node parameters of the speech enhancement model based on the target mixing loss function value until the preset termination condition is reached, and obtain the trained speech enhancement model.

[0098] The processing flow of the speech enhancement model for noisy speech samples can be referred to the description above, and will not be repeated here, only a brief explanation is given. In this invention, the noisy speech samples used in the training, testing, and validation of the speech enhancement model can be obtained from public datasets. For example, the general speech enhancement benchmark dataset VoiceBank+DEMAND can be used for model training and testing. This dataset contains clean speech and noisy speech in various noisy environments. Its speech data comes from the VoiceBanking Corpus, covering real recordings of various pronunciations, speech rates, and accents, ensuring the diversity and representativeness of the speech samples. The noisy speech data comes from the DiverseEnvironments Multi-channel Acoustic Noise Database (DEMAND). This database includes noise recorded in various real environments such as inside cars, homes, and public places, rather than artificially synthesized noise, and does not assume Gaussian / stationarity. The training set and the test set are completely independent in terms of speaker and noise type, ensuring the model's generalization ability in real-world scenarios.

[0099] After obtaining the enhanced speech output by the decoder, the target mixing loss function value can be calculated based on the generated enhanced speech and the real clean speech. The target mixing loss function includes a time loss function, an amplitude loss function, and a complex loss function.

[0100] This invention employs time loss magnitude loss and complex loss The sum of these losses is used as a hybrid loss function for training, and can be calculated using the following formulas (16), (17) and (18).

[0101]

[0102]

[0103]

[0104] in E As expected, x This is the time-domain waveform of pure speech. For the predicted enhanced speech time-domain waveform, x m For the amplitude spectrum of pure speech, X r and The real part of the complex spectrum of the real, clean speech and the enhanced speech are respectively the real part of the STFT; X i and The imaginary part of the complex spectrum of STFT for real, clean speech and enhanced speech, respectively.

[0105] The speech enhancement model is then adjusted based on the target loss function value until a preset termination condition is met, thus obtaining the speech enhancement model. This preset termination condition can be the convergence of the target mixture function value, which can be defined as the PESQ (Perceptual Evaluation of Speech Quality) value decreasing by less than 0.01 for five consecutive times. In this invention, the gradient can be updated using the AdamW (Adam with Weight Decay) optimization method, and then the parameters of the speech enhancement model can be adjusted based on the gradient. The speech enhancement model can include various parameters and hyperparameters, among which the hyperparameter can include the learning rate. In this invention, the learning rate of the speech enhancement model can be set to 0.0005.

[0106] After obtaining the trained speech enhancement model, the model performance can be evaluated using the validation set provided by the speech enhancement benchmark dataset VoiceBank+DEMAND, and the hyperparameters can be adjusted accordingly to optimize the model. Finally, the model is tested on the test set, and objective evaluation metrics are calculated to measure its speech enhancement effect.

[0107] like Figure 5 As shown, Figure 5 This is a schematic diagram of a training process for the speech enhancement model in this invention, which may include the following steps: Input and spectral decomposition: First, the noisy input speech is decomposed into amplitude spectrum and phase spectrum through short-time Fourier transform, which are used as parallel inputs to the encoder; Encoder processing: The amplitude spectrum and phase spectrum are first processed by the encoder. The encoder uses SCConv to compress redundant features and uses FcaNet to weight the frequency domain features to enhance the model's focus on key frequency components. After encoding calculation without SCConv, features are further extracted and transformed. Decoder processing: First, the encoded features are split into N data parts, and N TS-Conformers computation units are generated, which work in parallel. The phase spectrum decoding and amplitude spectrum decoding modules in the decoder decode the data to capture the complex global and local dependencies in the sequence, achieving deep time-frequency feature fusion. Together, they generate enhanced speech. Calculate the loss function: Compare the generated enhanced speech with the real clean speech, and calculate the mixture loss function between the two to measure the difference between the actual output of the model and the real clean speech; Termination condition determination: During the iteration process, check whether the model loss has converged, i.e., whether the termination condition has been met. If it has converged, i.e., the PESQ decreases by less than 0.01 for 5 consecutive times, then stop training and output the generated enhanced speech; otherwise, update the model node parameters according to the hybrid loss function value, return to the encoder processing step, and proceed to the next iteration.

[0108] By introducing FcaNet, an embodiment of this invention, time-domain features are transformed into frequency-domain features through DCT, effectively capturing periodic variations and spectral features in speech signals, especially performing better in low signal-to-noise ratio scenarios. Compared to the pure convolution extraction method of MP-SENet, FcaNet provides an effective representation of global frequency information and local frequency components, enhancing the modeling ability of complex noise frequency features. Furthermore, FcaNet introduces a frequency-domain channel attention mechanism, which performs weighted operations based on the importance of features in different frequency bands, achieving adaptive feature compression and restoration. This dynamic feature weighting strategy not only effectively alleviates the feature loss problem in MP-SENet but also retains more low-frequency and key frequency band information during feature compression. Combining DCT and the attention mechanism, FcaNet improves feature representation capabilities, particularly in low-frequency noise suppression and complex spectrum recovery.

[0109] Furthermore, the baseline model is improved by employing SCConv, a flexible convolutional module capable of performing convolution operations in both the spatial and channel domains, enabling joint modeling of spatial and channel features. While extracting features, spatial and channel attention weights are generated to adaptively highlight information-dense feature channels. This mechanism effectively compensates for MP-SENet's failure in spatial feature extraction and poor multi-scale adaptability, improving robustness to noise in complex scenes.

[0110] This invention employs seven common evaluation metrics to objectively assess the speech enhancement effect of the speech enhancement model: Perceptual Speech Quality (PESQ), Short-Time Objective Intelligence (STOI), Segmental Signal-to-Noise Ratio (SSNR), Mean Opinion Score of Signal Distortion (CSIG), Mean Opinion Score of Background Noise (CBAK), Mean Opinion Score of Overall Quality (COVL), and Number of Model Parameters.

[0111] Among them, PESQ and STOI are the most commonly used core evaluation metrics for speech enhancement, reflecting the overall perceptual quality and intelligibility of enhanced speech, respectively. To further analyze the specific sources of improved perceptual quality, this invention introduces CSIG, CBAK, and COVL as supplementary metrics, providing a detailed explanation of PESQ changes from three dimensions: speech fidelity, background noise naturalness, and overall auditory experience. Additionally, SSNR measures noise suppression effectiveness from a signal processing perspective. Params are introduced to evaluate model size and memory requirements, forming a "performance-computational efficiency" evaluation system.

[0112] PESQ is an objective index with high similarity to subjective auditory scores, used to evaluate the quality of enhanced speech. PESQ scores are generally between 1 and 5; a higher score indicates that the enhanced speech is more similar to clean speech at the perceptual level. STOI is used to evaluate the intelligibility of enhanced speech, especially suitable for speech evaluation in noisy environments. The STOI calculation formula is shown in formula (19), where... These are the STFT coefficients of the reference signal. These are the STFT coefficients of the distorted signal. It is a time index. It is a frame index. It is a frequency band index. It is the duration of each frame. It refers to the number of bandwidths. The STOI score. The value is usually between 0 and 1; the higher the value, the better the intelligibility of the speech.

[0113]

[0114] SSNR is used to evaluate the degree of noise suppression. It is calculated by averaging the signal-to-noise ratio between speech frames. The calculation formulas are shown in formulas (20) and (21), where It is the first of the original speech signals n One sampling point, It is the first damaged speech signal n One sampling point, It is the total number of segments. This represents the number of sampling points per segment. A higher value indicates higher clarity and less noise in the damaged speech signal.

[0115]

[0116]

[0117] CSIG is an objective metric used to evaluate the performance of speech enhancement methods. It measures the quality of the enhanced speech signal relative to the original clean speech signal. CBAK primarily measures the degree of background noise distortion, reflecting the extent of background noise distortion after enhancement. COVL comprehensively considers multiple aspects of speech quality, including speech clarity, naturalness, and the effectiveness of background noise reduction, aiming to evaluate the overall quality of the enhanced speech. All three metrics range from 1 to 5 points. A higher CSIG value indicates that the enhanced speech is closer to the original signal and sounds more natural; a higher CBAK value indicates better performance in terms of background noise naturalness and noise distortion reduction; and a higher COVL value indicates higher overall quality of the enhanced speech signal and a better auditory experience.

[0118] Params refer to the total number of trainable parameters in a model, and are a key indicator for measuring model complexity and its computational resource requirements. More parameters mean higher model complexity and greater computational resource demands.

[0119] This invention employs spectrograms to further analyze speech enhancement effects. Spectrograms are three-dimensional speech analysis tools that visually present the dynamic characteristics of speech spectrum changes over time through three-dimensional information: a horizontal axis (time), a vertical axis (frequency), and the color intensity of coordinate points (i.e., speech energy intensity, typically an amplitude spectrum based on short-time Fourier transform). The magnitude of the energy value is represented by the color intensity; a darker color (higher decibel value) indicates stronger speech energy, and a lighter color (lower decibel value) indicates lower speech energy. The logarithmic amplitude spectrum (dB unit) better reflects human auditory perception. Spectrograms overcome the limitations of time-domain analysis (which cannot visually reflect frequency characteristics) and frequency-domain analysis (which cannot demonstrate time-varying relationships). Through joint time-frequency domain visualization, they provide an intuitive and quantitative means of evaluating effects, making them a core tool for optimizing and validating speech enhancement methods. The spectrograms of some samples from the VoiceBank+DEMAND dataset used in this invention are shown in Figures 6(a) and 7(b). Figures 6(a) and 6(b) are spectrograms of noisy samples from the VoiceBank+DEMAND dataset used in this invention, and Figures 7(a) and 7(b) are spectrograms of clean speech samples from the VoiceBank+DEMAND dataset used in this invention.

[0120] Analysis of experimental comparison results To verify the performance of the method of this invention, this invention selected the following models on the VoiceBank+DEMAND dataset for comparison: SEGAN (primarily based on adversarial learning), MetricGAN (based on adversarial learning), the improved MetricGAN+ model, PFPL (based on U-Net / convolutional modeling), speech enhancement methods combining DNN and convex optimization, DEMUCS-MRE, ForkNet, Transformer / Conformer methods that have recently emphasized long-range dependency and time-frequency modeling capabilities, TSTNN, SE-Conformer, NSE-CATNet, MTFAA, and the baseline model of this invention, MP-SENet. The experimental results are shown in Table 1.

[0121] Table 1. Experimental results of different speech enhancement methods on the VoiceBank+DEMAND dataset.

[0122] In Table 1, "-" indicates that the data was not provided in the original paper. As can be seen from Table 1, the method of this invention significantly improves PESQ, STOI, SSNR, and COVL compared to the comparison method, and achieves the highest scores in the table. CSIG and CBAK are also among the top, indicating that the speech enhanced by the method of this invention has advantages in subjective auditory quality (PESQ), speech intelligibility (STOI), noise suppression level (SSNR), and overall speech quality (COVL).

[0123] The method of this invention has a parameter count of 2.06 M, which is almost identical to the baseline model MP-SENet (2.05 M) (an increase of only 0.01 M, negligible), and far lower than other high-performance models such as SEGAN (43.18 M) and DEMUCS-MRE (33.53 M). This invention uses MP-SENet, currently known for its good speech enhancement performance, as the baseline model. Addressing the issues of insufficient frequency domain feature extraction, high feature redundancy in spatial and channel dimensions, and low computational efficiency inherent in MP-SENet, this invention introduces FcaNet and SCConv. While improving performance, the model complexity of this invention does not increase significantly, indicating that the SCConv feature compression module effectively removes redundancy. Compared to the baseline MP-SENet, this invention improves PESQ by 0.15, SSNR by 1.88, COVL by 0.20, and other metrics also show improvement, verifying the effectiveness of combining FcaNet and SCConv in this invention. FcaNet improves feature quality, while SCConv efficiently utilizes these high-quality features to enhance speech enhancement performance.

[0124] Traditional speech enhancement methods exhibit significant performance degradation in non-stationary noise and complex noise environments. Deep learning-based speech enhancement methods have become mainstream, demonstrating superior speech quality and intelligibility compared to traditional methods. However, while existing deep learning-based speech enhancement methods perform well in scenarios with known noise types and normal signal-to-noise ratios, their generalization ability declines significantly in real-world complex noise environments. The models are prone to overfitting to training data, leading to performance degradation when dealing with unknown noise types.

[0125] Other alternatives include adversarial learning-based models such as SEGAN, MetricGAN, and the improved model MetricGAN+; U-Net / convolutional modeling-based PFPL; speech enhancement methods combining DNN and convex optimization; DEMUCS-MRE; ForkNet; Transformer / Conformer-type methods that have recently emphasized long-range dependency and time-frequency modeling capabilities such as TSTNN, SE-Conformer, NSE-CATNet, and MTFAA; and the baseline model of this invention, MP-SENet.

[0126] As shown in Table 1, for the adversarial learning-based models SEGAN, MetricGAN, and the improved MetricGAN+, SEGAN has a parameter count as high as 43.18M, ​​nearly 21 times that of the method in this invention (2.06M). However, its core metrics such as PESQ (2.16) and SSNR (7.73) are all inferior to those of the method in this invention. MetricGAN+ outperforms MetricGAN, but its four objective evaluation metrics—PESQ, CSIG, CBAK, and COVL—are all inferior to those of the method in this invention. Models based on adversarial learning often have high computational costs, and their overall performance-computational efficiency lags behind the method in this invention.

[0127] Among the speech enhancement methods PFPL (based on U-Net / convolutional modeling), the combined DNN and convex optimization approach, DEMUCS-MRE, and ForkNet, considering the best-performing model overall, ForkNet has only 28% of the parameters (0.58M) of the method in this invention. However, its PESQ (3.18), CSIG (4.39), and COVL (3.81) scores are all lower than those of the method in this invention. It can be seen that models based on U-Net / convolutional modeling sacrifice necessary model capacity and performance in pursuit of lightweight design, thus facing a dilemma in balancing computational efficiency and performance.

[0128] For Transformer / Conformer methods such as TSTNN, SE-Conformer, NSE-CATNet, and MTFAA, which place greater emphasis on long-range dependencies and time-frequency modeling capabilities, considering the best-performing NSE-CATNet model, its parameter count (3.57M) is 173% of that of the method in this invention. However, its PESQ (3.19), CSIG (4.41), and COVL (3.82) metrics are basically the same as or slightly inferior to the method in this invention. This means that Transformer / Conformer methods, with significantly higher computational costs, only gain comparable or even slightly weaker performance, resulting in low performance-computation efficiency.

[0129] The baseline model MP-SENet of this invention has almost the same number of parameters (2.05M) as the method of this invention, but its PESQ (3.16), SSNR (16.34), and COVL (3.65) are all inferior to those of the method of this invention. Experimental results show that without the introduction of FcaNet and SCConv, the performance of the original MP-SENet model alone cannot be achieved.

[0130] Table 1 shows the experimental results of different speech enhancement methods on the VoiceBank+DEMAND dataset. The method of this invention, by combining the frequency domain discrimination capability of FcaNet and the feature extraction capability of SCConv, learns more universal and discriminative speech features. It achieves optimal performance on four objective evaluation metrics: PESQ, STOI, SSNR, and COVL, improving the overall quality of speech and effectively suppressing noise. This enhances the clarity, intelligibility, and robustness of speech enhancement in various real-world environments such as in-car, home, and public places. The method of this invention has only 2.06M parameters, verifying the effectiveness of SCConv in maximizing the preservation of discriminative information while reducing the number of model parameters and computational complexity.

[0131] Existing technologies are generally plagued by the contradiction that "improving performance requires increasing complexity, while reducing complexity sacrifices performance." The method of this invention achieves performance improvements with almost no increase in parameters (2.05M → 2.06M). The method of this invention demonstrates comprehensive superiority on the general dataset VoiceBank+DEMAND, particularly the simultaneous improvement in SSNR (speech noise reduction capability), PESQ, and STOI (speech quality), indicating that the method learns more fundamental and robust representations of speech and noise features. This generalization ability is the fundamental guarantee for its further adaptation to real-world complex environments and unknown marine noise environments.

[0132] In summary, the experimental data in Table 1 demonstrates that there is no existing alternative technology that can achieve comparable or better performance metrics with the same model complexity as this invention. Therefore, no other alternative can accomplish the same objective of this invention.

[0133] To more intuitively verify the effectiveness of the method of this invention, Figures 8(a) and 8(b) are spectrograms of the noise samples shown in Figures 6(a) and 6(b) after enhancement using the MP-SENet-based method, and Figures 9(a) and 9(b) are spectrograms of the noise samples shown in Figures 6(a) and 6(b) after enhancement using the speech enhancement method provided by this invention. It can be seen that Figures 8(a) and 8(b) contain many unevenly distributed, dotted or patchy dark areas, especially in the low-frequency region. This indicates that background noise has not been effectively suppressed, and there is a significant amount of residual noise. However, the formant structure (dark stripes) of the speech signal is clearly visible, indicating that the MP-SENet-based method basically preserves the speech content, but is always accompanied by significant background noise. Figures 9(a) and 9(b) have clean backgrounds, and the gray levels of non-speech areas are uniform and light in color, indicating that background noise has been strongly suppressed and the level of residual noise is low. The formant structure of the speech is well preserved, and the lines are clear and continuous. Within the speech segment, the valleys between harmonic structures are relatively clean, indicating that the method of the present invention can effectively distinguish and suppress noise coexisting with speech.

[0134] Cross-dataset experimental results and analysis To further evaluate the generalization ability of the method in this invention, in addition to the VoiceBank+DEMAND dataset, the DNS-Challenge dataset is also introduced for cross-dataset validation. DNS-Challenge provides a large-scale, diverse dataset containing clean speech, noise libraries, room impulse responses, etc. Feedback and strong transient noise are added to simulate real-world scenarios, making the acoustic environment far more complex than traditional datasets. The dataset selects high-quality speech and noise segments from public data sources such as Librivox and Audioset, and generates noisy speech pairs through synthesis techniques to create training and test datasets. The noise sources, mixing methods, and signal-to-noise ratio distribution of the DNS-Challenge dataset differ significantly from those of the VoiceBank+DEMAND dataset, effectively validating the robustness and generalization ability of the speech enhancement method under unknown noise conditions.

[0135] In the experimental setup, this invention selects the no-reverb subset of the DNS-Challenge synthetic test set as a supplementary test set. Under the premise that the network structure, model parameters, and training strategy are consistent with the above experiments, an objective evaluation of the MP-SENet-based method and the method of this invention is performed. The experimental results are shown in Table 2.

[0136] Table 2. Performance comparison results of different speech enhancement methods on the DNS-Challenge dataset.

[0137] Table 2 presents the objective evaluation results of different speech enhancement methods on the DNS-Challenge synthetic test set (no-reverb). As can be seen from Table 2, compared to the original noisy speech, both the MP-SENet-based method and the method of this invention achieve significant improvements in PESQ, STOI, CSIG, CBAK, and COVL metrics, indicating that the MP-SENet-based method and the method of this invention can effectively improve speech quality and intelligibility under complex noise conditions.

[0138] Compared with the baseline model MP-SENet, the method of this invention shows improvements in PESQ, STOI, CSIG, CBAK, and COVL. The parameter count of the method of this invention is 2.07 M, which is almost exactly the same as the parameter count of the baseline model MP-SENet (2.05 M) (an increase of only 0.02 M, which is negligible).

[0139] Traditional deep learning methods require increased model complexity (number of parameters) to improve performance, leading to a surge in computational costs. This invention introduces FcaNet to enhance the ability to discriminate noise frequency domain characteristics and uses SCConv to adaptively compress and refine features, achieving improvements in metrics such as PESQ, STOI, and COVL with only a slight increase in the number of parameters.

[0140] Marine environmental experiment results and analysis To further evaluate the adaptability and effectiveness of the method of this invention in complex marine acoustic environments, this invention designed a speech enhancement experiment for marine environmental noise data. Marine environmental noise is an underwater acoustic phenomenon formed by the combined effects of seawater movement, wind and waves, marine biological activity, ice movement, and human factors.

[0141] Since the VoiceBank+DEMAND dataset does not contain marine environmental noise, and the DNS-Challenge dataset is not designed to include marine environmental noise, its noise database may contain a very small number of relevant recordings, but these are completely insufficient for the study of marine environmental noise.

[0142] The marine environmental noise data selected in this invention comes from the publicly available DeepShip dataset. The DeepShip dataset is a large-scale real-world underwater recording dataset specifically designed for underwater acoustic research, including ship classification and noise detection. Collected from real marine environments, it covers different seasons and hydrological conditions. The signal composition includes ship acoustic signatures, natural background noise, marine mammal vocalizations, and noise sources from human activities. This dataset includes underwater recordings from four types of ships: tankers, tugboats, passenger ships, and cargo ships.

[0143] The clean speech data uses the AIShell-1 Chinese speech recognition dataset, which was recorded and compiled by Beijing AIShell. As the largest open speech library in the Chinese field, it covers various recording methods such as high-fidelity recordings and telephone recordings. The recorded texts cover 11 fields including smart homes, autonomous driving, and industrial production, and can effectively reproduce speech in real-world applications.

[0144] This invention first superimposes all clean audio signals from AIShell with noise. The superimposed SNR value is set to the lowest value in general research -5. The higher the SNR value, the clearer the speech signal is relative to the noise; the lower the value, the stronger the noise is relatively. The noise is randomly sampled from the DeepShip noise dataset to ensure its randomness. To ensure that the noise is perceived as louder than the clean audio, a 15dB gain is added to the noise before mixing. Real-world ocean-recorded sounds are added to the test dataset. The method of this invention and the baseline method (based on MP-SENet) are used to train all data, and the training process is recorded synchronously.

[0145] In marine environment speech enhancement tasks, the ultimate goal is to improve the clarity and comfort of speech perceived by the human ear, rather than simply pursuing numerical optimization at the signal level. PESQ comprehensively evaluates speech from multiple dimensions, including time-frequency envelope, spectral distortion, and auditory masking effects, more closely reflecting the actual auditory experience and better reflecting whether speech enhancement methods truly improve the "usability" of speech in complex real-world scenarios. Therefore, this invention selects PESQ as the sole metric for objectively evaluating the experimental results of marine environment speech enhancement. The correspondence between the evaluation metric PESQ and the number of training steps is shown below. Figure 10 .

[0146] from Figure 10As can be seen, the PESQ of the baseline method gradually increases from 2.30 points at 1000 steps, reaching its first peak of 2.85 points at 6000 steps, then fluctuating between 2.80 and 2.90 points before stabilizing at 2.85 points. The PESQ of the method of this invention continuously increases from 2.38 points at 1000 steps, with only slight dips at 5000 and 7000 steps, before continuing to climb, eventually reaching 2.99 points, close to 3.0 points. At each step, the score of the method of this invention is higher than or equal to that of the baseline method. The initial advantage of the PESQ of the method of this invention is 0.08 (2.38 points for this invention, 2.30 points for the baseline method), and the final advantage expands to 0.14 (2.99 points for this invention, 2.85 points for the baseline method), indicating that the method of this invention has a significantly clearer and more natural speech output.

[0147] In the early training phase (1000 to 3000 steps), the PESQ improvement of the baseline method and the method of this invention was comparable, with the method of this invention being slightly faster, indicating that the method of this invention can learn effective features more quickly in the early stages of training. In the later training phase (starting from 7000 steps), the PESQ of the method of this invention continued to improve steadily, while the growth of the baseline method slowed down significantly.

[0148] In the process of noise data processing, frequency components contain a lot of key information. Especially in complex marine environments, the spectral characteristics of various noises differ significantly. Experimental results show that the method of this invention has better PESQ evaluation results than the baseline model method. The method of this invention introduces FcaNet, which can accurately capture feature differences, separate speech signals and background noise better, and improve the perception quality of speech in complex marine environments.

[0149] To verify the enhancement effect of the method of this invention on real-world dialogues in complex environments, this invention uses a video diary filmed on a research vessel as the noisy speech. The video is 32 minutes and 48 seconds long and includes dialogues in various environments such as the cabin and deck, as well as everyday and sudden noises such as waves, the operation of shipboard equipment, and falling objects. The video involves dialogues between multiple people, encompassing both everyday conversations and scientific research discussions. Since there is no corresponding clean speech for reference, this invention uses the Mean Opinion on Score (MOS) as the evaluation metric to assess the speech enhancement quality. Twenty listeners were asked to independently rate the video, and the MOS results are shown in Table 3.

[0150] Table 3 Comparison of MOS values ​​between MP-SENet and the method of this invention in real complex environments.

[0151] The MOS scoring system uses a 5-point scale. 5 points represents Excellent (distortion is imperceptible); 4 points represents Good (distortion is perceptible but not annoying); 3 points represents Average (distortion is slightly annoying); 2 points represents Poor (distortion is annoying); and 1 point represents Very Poor (distortion is extremely annoying). MOS is a subjective evaluation that directly reflects the user's listening experience, which objective indicators (such as PESQ) cannot completely replace, especially in assessing the "naturalness" and "comfort" of speech.

[0152] As shown in Table 3, the performance of the method of this invention is more balanced and stable than MP-SENet. The average MOS of MP-SENet is 3.61, which is in the "average" to "good" range, leaning towards "average". The average MOS of the method of this invention is 3.70, also in this range, but closer to the lower limit of "good" (4.0). This indicates that the method of this invention is slightly better than MP-SENet in overall subjective speech quality, with an improvement of 0.09. In the MOS score, an improvement of about 0.1 is generally considered a perceptible improvement, indicating that ordinary listeners can perceive that the output speech quality of the method of this invention is slightly better. The maximum MOS of MP-SENet is 4.89, and the maximum MOS of the method of this invention is 4.92, which is slightly higher by 0.03. Both are close to the full score of 5, indicating that in the most favorable scenarios or for some noisy speech samples, both methods can produce near-distortion-free, high-quality speech. The minimum MOS of MP-SENet is 1.18, while the minimum MOS of the method of this invention is 1.32. Although both are in the worst range, they are significantly higher than MP-SENet (an improvement of 0.14). This improvement indicates that the method of this invention is more robust to the extreme challenges of real-world complex environments, avoids the worst-case scenario, and provides a relatively more tolerable (although still very poor) speech enhancement effect.

[0153] Although subjective evaluation is the most scientific standard for speech quality assessment, different factors such as age and the listening process can affect the results. This invention further enhances the speech of a conversation between three people on deck regarding their work experience. The spectrograms are shown in Figures 11(a) and 11(b). Figure 11(a) is the spectrogram enhanced using the MP-SENet method, and Figure 11(b) is the spectrogram enhanced using the method of this invention.

[0154] As can be seen from Figures 11(a) and 11(b), the MP-SENet-based method still exhibits patchy or dotted bright areas during speech pauses, indicating incomplete noise suppression. In contrast, the method of this invention presents most areas as dark during speech pauses, indicating effective suppression of background noise. The MP-SENet-based method, in its denoising efforts, may inadvertently damage weak speech components, leading to excessive energy attenuation in these areas (resulting in overly dark colors). The method of this invention produces continuous and clear harmonic structures and formant lines, remaining discernible even in weak speech segments, demonstrating a reasonable energy distribution.

[0155] The MP-SENet-based method did not enhance the interjections spoken by the main speaker wearing a microphone between 0 and 0.6 seconds, but the method of this invention preserves them. Before 2.2 seconds, the voices of other speakers were somewhat attenuated by both methods. Around 2.4 seconds, the main speaker begins to speak. Compared to the MP-SENet-based method, the method of this invention preserves the main speaker's voice earlier and more completely. Careful listening revealed that the MP-SENet-based method deleted half a word from the main speaker around 2.4 seconds, while the method of this invention completely preserved this part. This method of this invention can more accurately preserve the main speaker's voice while denoising, resulting in better-sounding speech enhancement in multi-person dialogue scenarios.

[0156] Since VoiceBank+DEMAND is a classic benchmark dataset widely used in the field of speech enhancement, providing an important foundation for method comparison and performance evaluation, this invention uses the VoiceBank+DEMAND dataset as an example to conduct ablation experiments to verify the effectiveness of the proposed method. The experimental results are shown in Table 4. All experiments were conducted on the VoiceBank+DEMAND dataset under the same training configuration. MP-SENet+FcaNet represents adding an FcaNet module to the baseline model MP-SENet, and MP-SENet+SCConv represents adding SCConv to the baseline model MP-SENet.

[0157] SSIM is used to measure the similarity between the original image and the distorted image. This invention introduces SSIM to evaluate the similarity of the enhanced speech to the original clean speech in terms of spectral structure. The calculation formula for SSIM is shown in formula (22). and These are windows representing the original clean speech and the enhanced speech, respectively. and Representing windows respectively and The average value, and Representing windows respectively and variance Display window and covariance, and The constant introduced to avoid a denominator of zero is usually taken as... and ,in L It refers to the dynamic range of the speech signal. and These are small constants, typically 0.01 and 0.03 respectively. The SSIM value ranges from -1 to 1, but in practical applications, the range of 0 to 1 is usually of interest. The closer the SSIM value is to 1, the higher the consistency of the spectral structure between the enhanced speech and the original clean speech, and the better the quality.

[0158] Table 4 Ablation experimental results on the VoiceBank+DEMAND dataset

[0159]

[0160] As shown in Table 4, the SSIM values ​​of all four methods are greater than or equal to 0.9970, indicating that all four methods perform exceptionally well in maintaining the overall spectral structure. Adding the FcaNet module to the baseline model MP-SENet (i.e., MP-SENet+FcaNet) significantly improves three metrics closely related to speech quality and perception: PESQ, CSIG, and COVL. This indicates that FcaNet, through its frequency domain channel attention mechanism, effectively focuses on frequency band features crucial to perceptual quality, enhancing the fidelity and naturalness of the speech signal. The improvements in STOI and SSIM are relatively small, suggesting that FcaNet primarily optimizes speech quality rather than speech intelligibility and spectral structure. In summary, utilizing the FcaNet module improves speech quality, intelligibility, and the consistency of the spectral structure. Adding SCConv to the baseline model MP-SENet (i.e., MP-SENet+SCConv) yields the best SSIM (0.9978) and the second best STOI, but PESQ decreases slightly (3.1243). This indicates that SCConv, through feature compression and reconstruction, effectively improves intelligibility while maintaining the integrity of the spectral structure. The enhanced speech signal is structurally closer to the original clean speech, making the speech sound clearer and more natural, but it has a slight impact on perceived speech quality.

[0161] The method of this invention (i.e., MP-SENet+ FcaNet+SCConv) adds two modules, FcaNet and SCConv. PESQ, CSIG, COVL, and STOI are all optimal, while SSIM is the second best. PESQ (3.3123) is significantly improved compared to adding only FcaNet (3.2154) and only adding only SCConv (3.1243), indicating a strong synergistic effect between FcaNet and SCConv in improving perceived quality. STOI (0.9550), CSIG (4.3612), and COVL (3.8538) simultaneously reach the highest values, while SSIM is the second best. This indicates that the method of this invention achieves optimization in terms of intelligibility, signal fidelity, overall auditory experience, and maintenance of the overall spectral structure, verifying the effectiveness of the synergistic enhancement of FcaNet and SCConv modules in this invention.

[0162] As can be seen from the ablation experiment results in Table 4, although adding SCConv to the baseline model MP-SENet slightly reduces PESQ, it significantly improves STOI and SSIM, indicating its unique advantages in speech intelligibility and spectral structure integrity; while the FcaNet module improves multiple indicators more evenly, reflecting its core role in frequency domain perception quality.

[0163] Based on the same inventive concept, this invention also provides an MP-SENet speech enhancement device based on frequency domain channel attention and feature compression, such as... Figure 12 As shown, the device 1200 may include: Acquisition module 1201 is used to acquire noisy speech; Processing module 1202 is used to perform short-time Fourier transform on noisy speech to obtain its amplitude spectrum and phase spectrum; stack the amplitude spectrum and the phase spectrum to form a two-dimensional time-frequency feature map; and use the two-dimensional time-frequency feature map as input features to the speech enhancement model. The speech enhancement model includes: The first convolutional block is used to extract features from the input features and use SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. FcaNet is used to segment the feature map F1 along the channel dimension, assign a 2D DCT frequency component basis to each group, extract multi-spectral vectors through DCT transformation, generate channel attention weights after learning, recalibrate and enhance the feature map F1 to suppress unimportant channels and enhance important channels, and output the feature map F2. DenseNet is expanded to take feature map F2 as input, and uses four convolutional layers with different expansion sizes to expand the receptive field on the time axis. Dense connections are used to avoid the gradient vanishing problem, and the output feature map F3 is produced. The second convolutional block is used to downsample the feature map F3 and output a low-resolution feature map F4. TS-Conformers is used to capture the long-range dependencies of feature map F4 in time and frequency dimensions in stages, fuse local and global information, and output feature map F5. An amplitude spectrum decoder is used to decode the feature map F5 to obtain an enhanced amplitude spectrum. A phase spectrum decoder is used to decode the feature map F5 to obtain an enhanced phase spectrum; The processing module is also used to generate enhanced speech by inverse short-time Fourier transform based on the enhanced amplitude spectrum and the enhanced phase spectrum.

[0164] Optionally, the first convolutional block includes: a 2D convolutional layer for extracting features from the input features to obtain first convolutional features; SCConv for performing joint feature filtering in the spatial and frequency domains on the first convolutional features to output first filtered features; a normalization layer for normalizing the first filtered features to output first normalized features; and a PReLU activation function for activating the first normalized features to output feature map F1.

[0165] Optionally, FcaNet is specifically used to: divide the feature map F1 into n groups along the channel dimension; assign a 2D DCT frequency component basis to each group based on the key frequency components of the speech signal; transform the feature map of each group using the assigned 2D DCT frequency component basis to obtain its spectral vector; concatenate the spectral vectors corresponding to the feature maps of the n groups to obtain a multi-spectral vector; input the multi-spectral vector into the weight learning module to generate a channel attention weight vector; and multiply the channel attention weight vector with the feature map F1 channel by channel to output the feature map F2.

[0166] Optionally, the weight learning module consists of a fully connected layer and a Sigmoid activation function.

[0167] An exemplary embodiment of the present invention also provides an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor. The memory stores a computer program executable by the at least one processor, the computer program being executed by the at least one processor to cause the electronic device to perform a method according to an embodiment of the present invention.

[0168] An exemplary embodiment of the present invention also provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a computer's processor, is used to cause the computer to perform a method according to an embodiment of the present invention.

[0169] An exemplary embodiment of the present invention also provides a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform a method according to an embodiment of the present invention.

[0170] refer to Figure 13 The present invention will now be described in the form of a structural block diagram of an electronic device 1300 that can serve as a server or client of the present invention, which is an example of a hardware device that can be applied to various aspects of the present invention. The term "electronic device" is intended to represent various forms of digital electronic computer devices, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0171] like Figure 13 As shown, the electronic device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of the electronic device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0172] Multiple components in electronic device 1300 are connected to I / O interface 1305, including: input unit 1306, output unit 1307, storage unit 1308, and communication unit 1309. Input unit 1306 can be any type of device capable of inputting information to electronic device 1300. Input unit 1306 can receive input digital or character information and generate key signal inputs related to user settings and / or function control of electronic device. Output unit 1307 can be any type of device capable of presenting information and may include, but is not limited to, a display, speaker, video / audio output terminal, vibrator, and / or printer. Storage unit 1308 may include, but is not limited to, disk and optical disk. Communication unit 1309 allows electronic device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks, and may include, but is not limited to, modems, network cards, infrared communication devices, wireless communication transceivers, and / or chipsets, such as Bluetooth devices, WiFi devices, WiMax devices, cellular communication devices, and / or the like.

[0173] The computing unit 1301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model methods, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above. For example, in some embodiments, the MP-SENet speech enhancement method based on frequency domain channel attention and feature compression can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1300 via ROM 1302 and / or communication unit 1309. In some embodiments, the computing unit 1301 may be configured, by any other suitable means (e.g., by means of firmware), to perform the MP-SENet speech enhancement method based on frequency domain channel attention and feature compression.

[0174] The program code used to implement the methods of the present invention can be written in any combination of one or more programming languages. This program code can be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code can be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0175] In the context of this invention, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0176] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0177] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) or liquid crystal display (LCD) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0178] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0179] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

Claims

1. An MP-SENet speech enhancement method based on frequency domain channel attention and feature compression, characterized in that, The method includes: Acquire noisy speech; Perform a short-time Fourier transform on the noisy speech to obtain its amplitude spectrum and phase spectrum; The amplitude spectrum and the phase spectrum are stacked to form a two-dimensional time-frequency feature map; The two-dimensional time-frequency feature map is used as the input feature to input the speech enhancement model; The first convolutional block of the speech enhancement model extracts features from the input features, and uses the spatial and channel reconstruction convolution SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1. The feature map F1 is segmented along the channel dimension by the frequency domain channel attention network FcaNet of the speech enhancement model. A two-dimensional DCT frequency component basis is assigned to each group. Multi-spectral vectors are extracted through DCT transformation. After learning, channel attention weights are generated. The feature map F1 is recalibrated and enhanced to suppress unimportant channels and enhance important channels, and the feature map F2 is output. The DenseNet network of the speech enhancement model takes the feature map F2 as input, uses four convolutional layers with different dilation sizes to expand the receptive field on the time axis, and uses dense connections to avoid the gradient vanishing problem, outputting feature map F3. The second convolutional block of the speech enhancement model downsamples the feature map F3 to output a low-resolution feature map F4. The two-stage convolutional enhancement transformer TS-Conformers of the speech enhancement model capture the long-range dependencies of the feature map F4 in the time and frequency dimensions in stages, fuse local and global information, and output feature map F5; The feature map F5 is decoded by the amplitude spectrum decoder of the speech enhancement model to obtain the enhanced amplitude spectrum, and the feature map F5 is decoded by the phase spectrum decoder to obtain the enhanced phase spectrum. Based on the enhanced amplitude spectrum and the enhanced phase spectrum, the enhanced speech is generated by inverse short-time Fourier transform.

2. The method according to claim 1, characterized in that, The first convolutional block consists of a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function. The first convolutional block extracts features from the input features and uses the spatial and channel reconstruction convolution SCConv to perform joint feature filtering in the spatial and frequency domains, outputting a compressed feature map F1, including: The 2D convolutional layer extracts features from the input features to obtain the first convolutional features; The SCConv performs joint feature filtering in the spatial and frequency domains on the first convolutional feature and outputs the first filtered feature. The normalization layer normalizes the first selected feature and outputs the first normalized feature. The PReLU activation function activates the first normalized feature and outputs a feature map F1.

3. The method according to claim 2, characterized in that, The SCConv includes a spatial reconstruction unit (SRU) and a channel reconstruction unit (CRU). The SRU is used to separate input features into informative features and redundant features using a learnable scaling factor, and to suppress spatial redundancy through a cross-reconstruction operation, outputting spatially refined features. The CRU is used to employ segmentation, transformation, and fusion strategies on the spatial refined features. By combining grouped convolution and point convolution, as well as feature reuse, channel redundancy is reduced, and channel refined features are output.

4. The method according to claim 1, characterized in that, FcaNet segments the feature map F1 along the channel dimension, assigns a two-dimensional DCT frequency component basis to each group, extracts multi-spectral vectors through DCT transform, generates channel attention weights after learning, and recalibrates and enhances the feature map F1 to suppress unimportant channels and enhance important channels, outputting feature map F2, including: The FcaNet divides the feature map F1 into n groups along the channel dimension; Two-dimensional DCT frequency component basis is assigned to each group based on the key frequency components of the speech signal; For the feature map of each group, the spectral vector is obtained by transforming it using the two-dimensional DCT frequency component basis assigned to it. By concatenating the spectral vectors corresponding to the feature maps of n groups, a multi-spectral vector is obtained; The multi-spectral vector is input into the weight learning module to generate a channel attention weight vector; The channel attention weight vector is multiplied channel by channel with the feature map F1 to output the feature map F2.

5. The method according to claim 4, characterized in that, The weight learning module consists of a fully connected layer and a Sigmoid activation function.

6. An MP-SENet speech enhancement device based on frequency domain channel attention and feature compression, characterized in that, The device includes: The acquisition module is used to acquire noisy speech; The processing module is used to perform a short-time Fourier transform on the noisy speech to obtain its amplitude spectrum and phase spectrum; stack the amplitude spectrum and the phase spectrum to form a two-dimensional time-frequency feature map; and use the two-dimensional time-frequency feature map as input features to the speech enhancement model. The speech enhancement model includes: The first convolutional block is used to extract features from the input features and to perform joint feature filtering in the spatial and frequency domains using the spatial and channel reconstruction convolution SCConv, outputting a compressed feature map F1. The frequency domain channel attention network FcaNet is used to segment the feature map F1 along the channel dimension, assign a two-dimensional DCT frequency component basis to each group, extract multi-spectral vectors through DCT transformation, generate channel attention weights after learning, recalibrate and enhance the feature map F1 to suppress unimportant channels and enhance important channels, and output feature map F2. The DenseNet network is used to expand the receptive field on the time axis using four convolutional layers with different dilation sizes as input to the feature map F2, and dense connections are used to avoid the gradient vanishing problem, outputting feature map F3. The second convolutional block is used to downsample the feature map F3 and output a low-resolution feature map F4. Two-stage convolution-enhanced transformers (TS-Conformers) are used to capture the long-range dependencies of the feature map F4 in the time and frequency dimensions in stages, fuse local and global information, and output feature map F5. An amplitude spectrum decoder is used to decode the feature map F5 to obtain an enhanced amplitude spectrum; A phase spectrum decoder is used to decode the feature map F5 to obtain an enhanced phase spectrum; The processing module is further configured to generate enhanced speech by inverse short-time Fourier transform based on the enhanced amplitude spectrum and the enhanced phase spectrum.

7. The apparatus according to claim 6, characterized in that, The first convolutional block consists of a 2D convolutional layer, SCConv, a normalization layer, and a PReLU activation function, wherein: The 2D convolutional layer is used to extract features from the input features to obtain the first convolutional features; The SCConv is used to perform joint feature filtering in the spatial and frequency domains on the first convolutional features and output the first filtered features. The normalization layer is used to normalize the first screening feature and output the first normalized feature. The PReLU activation function is used to activate the first normalized feature and output feature map F1.

8. The apparatus according to claim 6, characterized in that, The FcaNet is specifically used for: The feature map F1 is divided into n groups along the channel dimension; Two-dimensional DCT frequency component basis is assigned to each group based on the key frequency components of the speech signal; For the feature map of each group, the spectral vector is obtained by transforming it using the two-dimensional DCT frequency component basis assigned to it. By concatenating the spectral vectors corresponding to the feature maps of n groups, a multi-spectral vector is obtained; The multi-spectral vector is input into the weight learning module to generate a channel attention weight vector; The channel attention weight vector is multiplied channel by channel with the feature map F1 to output the feature map F2.

9. The apparatus according to claim 8, characterized in that, The weight learning module consists of a fully connected layer and a Sigmoid activation function.

10. An electronic device, comprising: processor; as well as Stored program memory, The program includes instructions that, when executed by the processor, cause the processor to perform the method according to any one of claims 1-5.

Images