A cascade mask network speech enhancement method and related device integrating auditory process

Abstract

This invention provides a cascaded masking network speech enhancement method and related device that integrates auditory processes, relating to the field of speech signal processing technology. Noisy speech is acquired, and coarse-grained features are extracted from the noisy speech using a time-domain coarse-grained masking network to generate a preliminary enhanced speech waveform. This preliminary enhanced speech waveform is then subjected to a short-time Fourier transform, and the target speaker embedded information is combined using a frequency-domain fine-grained masking network to generate the final enhanced speech. This method improves the auditory quality and clarity of the desired speech in noisy environments.

Description

Technical Field

[0001] The present application relates to the technical field of speech signal processing, and in particular to a cascade mask network speech enhancement method and related devices integrating auditory processes. Background Art

[0002] In recent years, with the rapid development of deep learning technology, speech enhancement has gradually evolved into a supervised learning task based on deep learning. By inputting noisy speech into the network and learning the nonlinear mapping relationship between noisy speech and clean speech, deep learning-based speech enhancement methods have a stronger ability to suppress non-stationary noise than traditional statistical signal processing methods. To improve the speech enhancement performance of the network, deep learning-based speech enhancement methods are moving towards the use of more powerful neural network basic modules and more effective structural design. Based on the different training objectives, they can be divided into two categories: mask estimation-based and mapping-based methods. Compared with mask estimation, mapping methods are not stable in regulating detailed numerical values, while mask methods learn a limited dynamic range and converge faster. Therefore, mask estimation-based enhancement methods are superior to mapping-based enhancement methods and have attracted more research attention. In terms of network architecture design, the decoupled speech enhancement framework decouples the original complex spectrum estimation into two subtasks: amplitude and phase optimization. This mitigates the two-way compensation effect between amplitude and phase in single-stage complex-domain enhancement methods. Consequently, many time-frequency domain methods adopt the decoupled dual-path framework and similar architectures. Existing decoupled or single-stage speech enhancement frameworks are designed based on various characteristics of speech signals, such as time and frequency domain characteristics. They generally perform better under relatively high signal-to-noise ratio (SNR). However, when the SNR is extremely low, they suffer from incomplete noise removal and over-suppression of the target speaker's speech, resulting in poor final speech enhancement results. In terms of basic module design, existing methods typically improve speech enhancement performance by increasing the number of learnable parameters or deepening the neural network hierarchy. This results in excessively large model parameters and high computational complexity. Furthermore, typical speech enhancement models are limited to eliminating ambient noise and cannot effectively remove human voice noise. Therefore, a new speech enhancement method is urgently needed.

[0003] Various types of interference, such as ambient sound, human voices, music, and room reverberation, can severely degrade the performance of speech processing applications such as automatic speech recognition, online audio and video calls, and hearing aids in real-world scenarios. Therefore, improving the perceived quality and clarity of desired speech in noisy environments has become a pressing technical challenge. Summary of the Invention

[0004] In order to improve the auditory quality and clarity of desired speech in a noisy environment, the present application provides a cascade mask network speech enhancement method and related devices that integrate the auditory process.

[0005] In the first aspect, the present application provides a cascade mask network speech enhancement method that integrates the auditory process and adopts the following technical solutions:

[0006] A cascade mask network speech enhancement method integrating auditory processes, comprising:

[0007] Acquire noisy speech, and extract coarse-grained features from the noisy speech using a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform;

[0008] Performing a short-time Fourier transform on the initially enhanced speech waveform, and combining it with target speaker embedding information through a frequency-domain fine-grained mask network to generate a final enhanced speech;

[0009] The time-domain coarse-grained mask network includes an encoder, an enhancement layer, a mask network, and a decoder, and the frequency-domain fine-grained mask network includes an encoder, a dual-path sparse attention module, a speaker embedding fusion module, and a decoder.

[0010] The dual-path sparse attention module models time-frequency distribution through sparse multi-head self-attention, temporal convolutional attention, and frequency convolutional attention. The speaker embedding fusion module performs cross-attention calculation on the target speaker embedding and intermediate features to optimize the enhancement effect.

[0011] Optionally, the encoder of the time-domain coarse-grained mask network includes a two-dimensional convolution block and a dense dilated convolution block, which are used to extract high-level features of the input speech; the enhancement layer uses grouped bidirectional GRU and unidirectional GRU to model the dependencies within and between frames respectively; the mask network generates a coarse-grained mask through a parallel two-dimensional grouped convolution path, and multiplies it element-by-element with the encoder output to reconstruct a preliminary enhanced speech waveform through a decoder, and the decoder uses dense dilated convolution blocks and sub-pixel convolution to achieve waveform reconstruction.

[0012] Optionally, the grouped bidirectional GRU of the enhancement layer groups the input features into two parts along the channel dimension, and processes them in parallel through bidirectional GRU respectively. After the output is spliced, the features are enhanced by one-dimensional group convolution and group normalization, and a residual connection is performed with the features output by the encoder to alleviate gradient disappearance.

[0013] Optionally, the dual-path sparse attention module of the frequency-domain fine-grained mask network includes:

[0014] Sparse multi-head self-attention mechanism, which uses local window and dilated attention along the frequency axis and time axis respectively to reduce computational complexity;

[0015] The time convolution attention module and the frequency convolution attention module capture the long-range correlation of the time axis and frequency axis through compressed temporal convolutional networks, respectively, and generate attention maps to multiply with the input features;

[0016] Among them, the time convolution attention module performs global average pooling along the frequency axis, and the frequency convolution attention module performs global average pooling along the time axis.

[0017] Optionally, in the sparse multi-head self-attention mechanism, the attention calculation along the time axis adopts causal mask and double mask settings, wherein the forward mask in the double mask allows the model to integrate limited future context information, and the backward mask ensures that the attention calculation only focuses on the information of local adjacent frames.

[0018] Optionally, the speaker embedding fusion module fuses the target speaker embedding vector with the intermediate features through cross attention calculation, including:

[0019] Fixed speaker embedding vector and cross attention calculation of intermediate features to generate adaptive speaker embedding;

[0020] Adaptive speaker embedding is concatenated with intermediate features and integrated through two-dimensional point-by-point convolution to form fused features.

[0021] The target speaker embedding vector is generated by a speaker embedding extraction network and processed in the compressed ERB domain. The speaker embedding extraction network is composed of a stack of recalibrated encoders, which include two-dimensional gated linear units and U-Net blocks, and optimizes the distinguishability of the embedding through AAM softmax additive angular margin loss.

[0022] Optionally, the framing parameters of the time-domain coarse-grained mask network are a frame length of 640 samples and a frame shift of 320 samples;

[0023] The framing parameters of the frequency domain fine-grained mask network are a frame length of 512 samples and a frame shift of 256 samples, and a square root Hanning window is used for short-time Fourier transform.

[0024] In a second aspect, the present application proposes a cascade mask network speech enhancement device that integrates the auditory process, and executes the method described above, including:

[0025] A speech acquisition module is used to acquire noisy speech and extract coarse-grained features from the noisy speech through a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform;

[0026] The speech enhancement module is used to perform short-time Fourier transform on the initially enhanced speech waveform, and generate the final enhanced speech by combining the target speaker embedding information through a frequency domain fine-grained mask network.

[0027] In a third aspect, the present application provides a computer device, comprising: a memory and a processor, wherein the processor executes the method described above when running computer instructions stored in the memory.

[0028] In a fourth aspect, the present application provides a computer-readable storage medium comprising instructions, which, when executed on a computer, enable the computer to execute the method described above.

[0029] In summary, this application obtains noisy speech, extracts coarse-grained features from it using a coarse-grained time-domain masking network to generate a preliminary enhanced speech waveform, performs a short-time Fourier transform on the preliminary enhanced speech waveform, and combines it with the target speaker embedding information using a fine-grained frequency-domain masking network to generate the final enhanced speech. This achieves the effect of improving the auditory quality and clarity of the desired speech in noisy environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 It is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application;

[0031] Figure 2 This is a flow chart of the first embodiment of the cascade mask network speech enhancement method integrating the auditory process of the present application;

[0032] Figure 3 This is the overall network structure diagram of the cascade mask network speech enhancement method of the present application that integrates the auditory process;

[0033] Figure 4 This is a schematic diagram of the internal structure of the dual-path sparse attention module of this application;

[0034] Figure 5 This is a schematic diagram of the internal structure of the time convolution attention and frequency convolution attention modules of this application;

[0035] Figure 6 This is a structural block diagram of the first embodiment of the cascade mask network speech enhancement device that integrates the auditory process of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solutions and advantages of this application more clearly understood, the present application is further described in detail below through the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0037] Reference Figure 1 , Figure 1 This is a schematic diagram of the computer device structure of the hardware operating environment involved in the embodiment of the present application.

[0038] like Figure 1As shown, the computer device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display and an input unit, such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a wireless fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk storage device. The memory 1005 may also be a storage device independent of the processor 1001.

[0039] Those skilled in the art will understand that Figure 1 The structure shown in the figure does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0040] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a cascade mask network speech enhancement program that integrates the auditory process.

[0041] exist Figure 1 In the computer device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in this application can be set in the computer device, and the computer device calls the cascade mask network voice enhancement program of the fusion auditory process stored in the memory 1005 through the processor 1001, and executes the cascade mask network voice enhancement method of the fusion auditory process provided in the embodiment of the present application.

[0042] The embodiment of the present application provides a cascade mask network speech enhancement method integrating the auditory process, referring to Figure 2 , Figure 2 This is a flow chart of the first embodiment of the cascade mask network speech enhancement method integrating the auditory process of this application.

[0043] In this embodiment, the cascade mask network speech enhancement method integrating the auditory process includes the following steps:

[0044] Step S10: Acquire noisy speech, and extract coarse-grained features from the noisy speech through a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform.

[0045] The encoder of the time-domain coarse-grained mask network includes a two-dimensional convolution block and a dense dilated convolution block, which are used to extract high-level features of the input speech; the enhancement layer uses a grouped bidirectional GRU and a unidirectional GRU to model the dependencies within and between frames respectively; the mask network generates a coarse-grained mask through a parallel two-dimensional grouped convolution path, and after element-by-element multiplication with the encoder output, the decoder reconstructs the preliminary enhanced speech waveform, and the decoder uses a dense dilated convolution block and sub-pixel convolution to achieve waveform reconstruction.

[0046] It can be understood that the grouped bidirectional GRU of the enhancement layer groups the input features into two parts along the channel dimension, and processes them in parallel through the bidirectional GRU respectively. After the output is spliced, the features are enhanced by one-dimensional group convolution and group normalization, and the residual connection is performed with the features output by the encoder to alleviate the gradient disappearance.

[0047] In the specific implementation, this embodiment constructs a time-domain coarse-grained mask network. In order to simulate the process of "rough listening" of the human ear in a noisy environment in the first stage, the time-domain frame length is longer than the frequency-domain frame length in the second stage. Specifically, the noisy speech undergoes coarse-grained sequence framing preprocessing before being input into the encoder. Each frame covers more time samples, and each mask value is calculated based on signal information over a longer period of time, thereby achieving a coarser-grained analysis. Although the time resolution is reduced, the model is able to capture signal features over a longer period of time, such as the overall energy changes of the speech signal, the speech envelope, etc., reflecting the overall trend and dynamic changes of the speech signal. The first stage of the time-domain coarse-grained mask network includes three modules: encoder, enhancement layer, mask network and decoder. The encoder uses two-dimensional convolution blocks and dense dilated convolution blocks to learn the coded representation of the input speech signal. The enhancement layer uses two grouped DPRNNs to model the dependencies within and between frames. The mask network estimates the optimal mask based on the features learned by the enhancement layer to enhance noisy speech, including two two-dimensional convolutions and two parallel two-dimensional grouped convolution paths. The decoder uses dense dilated convolution blocks, sub-pixel convolutions and two-dimensional point-by-point convolutions to reconstruct speech features, and reconstructs the preliminary enhanced speech waveform through overlap-addition.

[0048] In a specific implementation, building a time-domain coarse-grained mask network includes:

[0049] The input long speech sequence is coarsely divided into frames, and the sequence frames are stacked to form a 3D tensor input to the encoder;

[0050] Use the encoder to obtain the learned signal representation S of the input sequence frames;

[0051] Use enhancement layers to model intra-frame dependencies.

[0052] The enhancement layer models the dependencies between frames. The process is similar to that of the enhancement layer modeling the dependencies within frames. The difference lies in the dimension transformation and the use of a unidirectional GRU as the basic modeling module. That is, when flattening the tensor, dimensions B and F are merged to prepare for inter-frame RNN processing.

[0053] The mask network uses two-dimensional convolution to double the output of the enhancement layer along the channel dimension to obtain features , dual-path two-dimensional group convolution is combined with different activation functions, and the advantages of the two activation functions are combined to obtain the feature ;

[0054] feature Through two-dimensional group convolution and ReLU activation function processing, nonlinear transformation is used to extract and learn the feature representation of the target speaker, and then a coarse-grained mask of the target speaker is generated;

[0055] Combine the obtained mask with the encoder output The decoder is used to obtain the preliminary enhanced features of the target speaker by element-by-element multiplication, and the overlap-addition is used to reconstruct the preliminary enhanced speech waveform.

[0056] The step of using the enhancement layer to model the dependency relationship within the frame specifically further includes:

[0057] Rearrange and reshape the learning signal representation S, that is, rearrange the last two dimensions of the 3D tensor, the number of frames N and the frame length F, before the channel dimension C, and then flatten the tensor into (B×N, F, C);

[0058] Use grouped bidirectional GRU for intra-frame modeling, and reshape the encoding representation Grouping along the last dimension C yields , , after segmentation and They correspond to different subsets of the input features, which helps reduce the computational burden of a single GRU layer and allows the model to process different feature groups in parallel while hiding the state After being initialized to an all-zero tensor, the hidden state Split into and , corresponding to the initial hidden states of rnn1 and rnn2, respectively, and the splitting method is the same as the input Same, that is, along the last dimension Divide into two parts;

[0059] Use rnn1 to process input and , and get the output and the new hidden state ;

[0060] Using rnn2 to process input and , and get the output and the new hidden state The two GRU layers can process different feature groups in parallel, thereby improving computational efficiency;

[0061] Will and Concatenate along the last dimension, the new hidden state and Splicing to get the final output and the final hidden state ;

[0062] The output after intra-frame modeling After rearrangement, one-dimensional group convolution is used to perform linear transformation and feature extraction on the rearranged output to enhance the feature expression ability, and then the linearly transformed features are extracted. Transpose to restore the original feature dimension;

[0063] The features after convolution Perform group normalization to stabilize the training process and accelerate model convergence;

[0064] After group normalization and the rearranged input encoding representation Perform residual connections to promote information flow and alleviate the gradient disappearance problem.

[0065] The temporal coarse-grained mask network described in this embodiment is set as follows: the frame length and frame shift for segmenting long speech sequences are 640 (40ms) and 320 (20ms) respectively, the sampling rate of noisy speech is 16000Hz, the encoder uses two-dimensional point-by-point convolution to double the channels to 64, and uses dense dilated convolution blocks with dilation sizes of 1, 2, 4, and 8 to gradually expand the receptive field and extract high-level features. The dense dilated convolution blocks introduce depthwise separable convolution to replace standard convolution for pruning, and the decoder uses depthwise separable convolution to downsample the frame length F, with a convolution kernel size of (1, 3) and a stride of (1, 2). Sub-pixel convolution in the decoder implements upsampling and restores the dimension to F. The input features of the enhancement layer are , grouped bidirectional GRU models local dependencies, After rearrangement and reshaping before input, the features are obtained , the specific formula is:

[0066]

[0067] in is the output of the grouped bidirectional GRU, and They are the mapping functions defined for bidirectional GRU, and for Features grouped along the last dimension, and Hidden Grouped features, hidden states Initialized as a full zero tensor, the one-dimensional group convolution Perform linear transformation and feature extraction, group normalization divides the channels into 4 groups, and calculates the mean and variance in each group for normalization. The specific formula is:

[0068]

[0069]

[0070] in and is the mean and variance of all elements in the group; is a small constant used to prevent division by zero errors; is the scaling factor; is the offset; is the input to the group normalized feature. The global dependency modeling between frames is implemented based on the unidirectional GRU. The whole process is the same as the local dependency modeling. When the input features are rearranged and reshaped, the features obtained are The coarse-grained target speaker mask generated by the mask network is element-wise multiplied with the encoder output.

[0071] Step S20: performing a short-time Fourier transform on the initially enhanced speech waveform, and combining the target speaker embedding information through a frequency domain fine-grained mask network to generate the final enhanced speech;

[0072] Among them, the time domain coarse-grained mask network includes an encoder, an enhancement layer, a mask network and a decoder, and the frequency domain fine-grained mask network includes an encoder, a dual-path sparse attention module, a speaker embedding fusion module and a decoder.

[0073] In its implementation, this embodiment constructs a frequency-domain fine-grained mask network. In the second stage, a shorter frame length is used for short-time Fourier transform (SFT). Before entering the encoder, an equivalent rectangular bandwidth filter bank is used to reduce input feature redundancy. The encoder and decoder consist of two two-dimensional convolutions and three grouped temporal convolution blocks. The grouped temporal convolution blocks consist of two two-dimensional point-by-point convolutions and a two-dimensional depth-wise dilated convolution. Skip connections are used between the encoder and decoder to reduce information loss. The enhancement layer uses a dual-path sparse attention module consisting of sparse multi-head self-attention, temporal convolution attention, and frequency convolution attention to model the time-frequency distribution. The fine-grained mask generated by the decoder further refines the signal based on the initial enhancement. In addition to the encoder, enhancement layer, and decoder, the second stage introduces a speaker fusion module after the enhancement layer to fuse the target speaker embedding with the intermediate features of the fine-grained mask network. The target speaker embedding is output by the speaker embedding extraction network after processing the target speaker's registered speech.

[0074] In a specific implementation, the steps of constructing a frequency domain fine-grained mask network include:

[0075] Use short-time Fourier transform to obtain the spectrum Z of the preliminary enhanced speech. The frame length is shorter than that in the first stage. Z is decoupled to obtain the original complex spectrum Y and the original amplitude spectrum Y0. The amplitude, real part, and imaginary part are stacked into a new feature tensor and input into the frequency domain fine-grained mask network.

[0076] An ERB filter is used to merge features in high-frequency bands above 2kHz, effectively reducing the computational burden of the model while ensuring performance;

[0077] The encoder extracts features from the feature map, doubles the number of channels from 16 to 32, and downsamples in the frequency direction;

[0078] The dual-path sparse attention module models the spectral pattern within a single frame and the long-distance dependencies between frames output by the encoder along the frequency axis and time axis respectively, enhancing the model's ability to model the entire time-frequency representation.

[0079] The speaker embedding fusion module combines the output of the dual-path sparse attention module Fusion with the target speaker embedding, the fixed speaker embedding vector is Cross-attention calculations are performed, with attention weights reflecting the relevance of each time-frequency unit with a fixed speaker embedding. The cross-attention output is residually concatenated with the original speaker embedding to generate an adaptive speaker embedding. The residual connection is used to aggregate the adaptive embeddings, which are then directly concatenated with intermediate features for fusion. The speaker embedding extraction network consists of stacked recalibrated encoders, which scale features while minimizing information loss. The entire network is executed in the compressed ERB domain, which not only consumes less computational resources but also generates speaker embeddings optimized for speech enhancement tasks.

[0080] The decoder upsamples the frequency dimension layer by layer, and the number of channels is restored from 32 to 16, which corresponds to the downsampling operation of the encoder in the step "the encoder extracts features from the feature map, doubles the channels from 16 to 32, and downsamples them along the frequency direction", completing the reconstruction of the features.

[0081] The ERB filter is used to split the decoder output into frequency bands to restore the original resolution, which corresponds to the downsampling operation in the step "using the ERB filter to merge the features of the high-frequency band above 2kHz, effectively reducing the computational burden of the model while ensuring performance."

[0082] The dual-path sparse attention module models time-frequency distribution through sparse multi-head self-attention, temporal convolutional attention, and frequency convolutional attention. The speaker embedding fusion module performs cross-attention calculation on the target speaker embedding and intermediate features to optimize the enhancement effect.

[0083] The dual-path sparse attention module models the spectral pattern within a single frame and the long-distance dependencies between frames output by the encoder along the frequency axis and time axis, respectively, enhancing the model's ability to model the entire time-frequency representation. Specifically, the following steps are involved:

[0084] First, the spectral pattern within the frame is modeled along the frequency axis. Sparse multi-head self-attention only focuses on the adjacent frequency bands of a specific frequency within the local window size, and uses expansion attention outside the window. The expansion attention step size is , the interval outside the window is and The model can effectively filter or emphasize specific frequency components, reduce redundant calculations in the full multi-head attention mechanism, and reduce model complexity.

[0085] The features of the sparse multi-head self-attention mechanism's output are linearly transformed through a linear layer to enhance feature expression. The output of the linear layer is reshaped and rearranged, and the mean and variance are independently calculated along the channel dimension through an instance normalization layer for normalization to stabilize the training process.

[0086] The normalized features are added to the encoder output to form a residual connection, which helps the model better transfer gradients and alleviates the gradient vanishing problem in deep networks.

[0087] The output after the residual connection is rearranged and input into the frequency convolution attention module. The frequency convolution attention module is composed of a compressed temporal convolutional network (S-TCM) to capture long-distance correlations. Global average pooling is performed along the time axis before inputting the S-TCM. In the S-TCM, the initial one-dimensional convolution is used for channel mixing, and the dilated convolution is used to explore long-distance sequence relationships. The final one-dimensional convolution projects the number of output channels back to the original size. A gating branch is added in parallel to the main dilated convolution branch. The gating branch uses the sigmoid function to adjust its value, which helps the flow of gradients during backpropagation and alleviates the gradient disappearance problem. After applying the sigmoid activation function to generate the attention map, the attention map is copied along the time axis and multiplied with the output after the residual connection to obtain the feature. ;

[0088] Causal sparse multi-head self-attention is used to model long-distance dependencies between frames along the time axis. The forward mask allows the first layer of attention to incorporate limited future contextual information. The backward mask ensures that the attention calculation only focuses on the information of local adjacent frames. The forward mask is canceled in the remaining layers, and a causal mask is set to prevent further expansion of the receptive field in the future. Compared with the full multi-head self-attention mechanism, this ensures that the model runs in a streaming manner and can focus on local temporal features, improving efficiency and relevance.

[0089] The output after the residual connection is rearranged and input into the time convolution attention module. The time convolution attention module has the same structure as the frequency convolution attention module, but global average pooling is performed along the frequency axis before inputting into the S-TCM. Finally, after applying the sigmoid activation function to generate the attention map, the attention map is copied along the frequency axis and multiplied with the output after the residual connection to obtain the feature , generating an attention map to guide the model to focus on “which time frame”, effectively modeling the energy distribution on the time axis.

[0090] In this embodiment, the human auditory process is integrated into the speech enhancement framework. A coarse-grained masking network in the time domain analyzes the overall waveform shape, temporal structure, and energy variations of the speech signal to generate a preliminary enhanced signal dominated by the target speaker. In the frequency domain enhancement stage, speaker embedding is introduced as prior information, and the resulting fine-grained mask performs more refined enhancement. The overall framework design follows a "coarse-to-fine" auditory process, improving existing speech enhancement methods' shortcomings of incomplete noise removal and over-suppression of the target speaker's speech at low signal-to-noise ratios. It not only eliminates background noise but also effectively suppresses other interfering speakers. A novel dual-path sparse attention module is proposed within the frequency domain fine-grained masking network. The temporal convolutional attention module and the frequency convolutional attention module effectively model the energy distribution along the time and frequency axes. Compared to the full attention mechanism, the sparse multi-self-attention mechanism enhances the model's expressiveness while significantly reducing the number of parameters and computational overhead. The dual-masking design along the time axis also ensures the model's execution in a streaming manner. This overall module setup makes the model more efficient, lightweight, and suitable for resource-constrained environments. This paper provides a new perspective for the construction of a speech enhancement framework. The overall model improves the effect of speech enhancement while maintaining a small number of parameters and computational complexity.

[0091] It should be noted that the dual-path sparse attention module of the frequency domain fine-grained mask network includes: a sparse multi-head self-attention mechanism, which uses local windows and dilated attention along the frequency axis and time axis respectively to reduce computational complexity; a time convolution attention module and a frequency convolution attention module, which capture the long-range correlation of the time axis and frequency axis respectively through a compressed temporal convolutional network, and generate an attention map to multiply the input features; among them, the time convolution attention module performs global average pooling along the frequency axis, and the frequency convolution attention module performs global average pooling along the time axis.

[0092] In a specific implementation, in the sparse multi-head self-attention mechanism, the attention calculation along the time axis adopts causal mask and double mask settings, where the forward mask in the double mask allows the model to integrate limited future context information, and the backward mask ensures that the attention calculation only focuses on the information of local adjacent frames.

[0093] It should be noted that the speaker embedding fusion module fuses the target speaker embedding vector with the intermediate features through cross-attention calculation, including: cross-attention calculation of the fixed speaker embedding vector and the intermediate features to generate an adaptive speaker embedding; the adaptive speaker embedding is concatenated with the intermediate features and integrated through two-dimensional point-by-point convolution to form a fusion feature; wherein, the target speaker embedding vector is generated by the speaker embedding extraction network and processed in the compressed ERB domain. The speaker embedding extraction network is composed of a stack of recalibrated encoders, which include a two-dimensional gated linear unit and a U-Net block, and the distinguishability of the embedding is optimized through the AAM softmax additive angular boundary loss.

[0094] In a specific implementation, the framing parameters of the time domain coarse-grained mask network are a frame length of 640 samples and a frame shift of 320 samples; the framing parameters of the frequency domain fine-grained mask network are a frame length of 512 samples and a frame shift of 256 samples, and a square root Hanning window is used for short-time Fourier transform.

[0095] The overall structure of this embodiment is shown in FIG. Figure 3 As shown in the figure, the model includes three parts: a time domain coarse-grained mask network, a frequency domain fine-grained mask network, and a speaker embedding extraction network. The time domain coarse-grained mask network includes four modules: an encoder, an enhancement layer, a mask network, and a decoder. The encoder uses a two-dimensional convolution block and a dense dilated convolution block to learn the coded representation of the input speech signal. The enhancement layer uses two grouped DPRNNs to model the dependency relationship between frames within and between frames. The mask network estimates the optimal mask based on the features learned by the enhancement layer to enhance the noisy speech, including two two-dimensional convolutions and two parallel two-dimensional grouped convolution paths. The decoder applies dense dilated convolution blocks, sub-pixel convolutions, and two-dimensional point-by-point convolutions to reconstruct speech features, and reconstructs the preliminary enhanced speech waveform by overlapping and adding; the frequency domain fine-grained mask network includes an encoder, an enhancement layer, a speaker embedding fusion module Block and decoder, the encoder and decoder consist of two two-dimensional convolution blocks and three grouped temporal convolution blocks. The grouped temporal convolution block consists of two two-dimensional point-by-point convolutions and two-dimensional depth-expanded convolutions. Skip connections are made between the encoder and decoder to reduce information loss. The enhancement layer uses a dual-path sparse attention module containing sparse multi-head self-attention, temporal convolution attention, and frequency convolution attention to model the time-frequency distribution. The speaker embedding fusion module calculates the cross-attention of the fixed speaker embedding vector and the intermediate features, and the generated target speaker adaptive embedding is fused with the intermediate features; the speaker embedding extraction network consists of stacked recalibrated encoders. Through a combination of continuous downsampling and upsampling operations, it can effectively utilize information of different scales without generating excessive computational load.

[0096] In a specific implementation, the specific enhancement process of the frequency domain fine-grained mask network in this embodiment is as follows:

[0097] The time-frequency spectrum of the first-stage time-domain preliminary enhanced signal is obtained by short-time Fourier transform. The frame length and frame shift are 512 (32ms) and 256 (16ms) respectively. The window function is the root Hanning window. The real and imaginary parts are extracted from the input spectrum data. The amplitude, real and imaginary parts are stacked together to form a new feature tensor input network. The second stage of fine-grained enhancement is performed on a compressed Execute within the domain, using The filter compresses the frequency band above 2kHz. The filter converts Hertz to The specific formula for scale conversion is:

[0098]

[0099] In the formula is the center frequency of the filter (kHz), is the bandwidth of the filter (Hz).

[0100] The first two layers of 2D convolution blocks in the encoder include 2D convolution, batch normalization, and PReLU activation functions. The 2D convolution kernel size is (1, 5), downsampling in the width direction, and increasing the number of channels from 9 to 32. The last three layers of grouped temporal convolution blocks gradually expand the receptive field through point-by-point convolution and depth-wise separable convolution, thereby capturing longer-range temporal dependencies and enhancing feature representation capabilities.

[0101] The dual-path sparse attention module models the spectral pattern within a single frame and the long-distance dependency between frames along the frequency axis and time axis respectively. The internal structure of the dual-path sparse attention module is as follows: Figure 4 Encoder output The dimensions are B×C×T×F, which are rearranged and reshaped into Then input into the sparse multi-head self-attention module with the dimension of BT×F×C, and define three matrices 、 、 respectively Perform three linear transformations to get the new vector 、 、 The obtained vectors are concatenated into a large matrix, which is recorded as 、 、 , which correspond to the query matrix, key matrix and value matrix respectively. Multiply the query vector with the key matrix in turn, and get the relevant attention weights. Apply sparse attention mask to the attention weights, and get the values , and then multiply them with the corresponding value vectors. The specific formula is:

[0102]

[0103] In the formula Is the adjustment factor. Sparse multi-head self-attention mask value 0 or 1, the mask value is 0 time frame Focus Frame And calculate the corresponding attention score. If the mask value is 1, the attention score is blocked or discarded. The specific formula is:

[0104]

[0105] In sparse multi-head self-attention, local attention is used to control each frame to focus on the frames within the context window, and the expanded attention is used to focus on the frames outside the local context window. The mask is applied to the similarity matrix, assuming For the local attention mask, when it exceeds the index distance, it is set to 0, so that the weight of this part becomes extremely small and hardly affects the result. The specific formula is:

[0106]

[0107] For the dilated attention mask, the index distance is The attention calculation is valid when it is a multiple of , and the specific formula is:

[0108]

[0109] A linear layer is used to perform a linear transformation on the output of the sparse multi-head attention module. The output of the linear layer is normalized by independently calculating the mean and variance along the channel dimension by the instance normalization layer. The normalized features are residually connected with the encoder output.

[0110] The frequency convolution attention module models the energy distribution in the frequency dimension. The internal structures of the frequency convolution attention module and the time convolution attention module are as follows: Figure 5 As shown. The frequency convolution attention will perform global average pooling along the time axis, that is, after the model "scans" the entire time axis, it will obtain the overall statistical characteristics of the frequency dimension. The specific formula is:

[0111]

[0112] in is the frequency energy representation, Indicates the number of channels, represents the time step, The frequency dimension is represented by the resulting frequency energy, which is then used by the frequency convolutional attention module to capture long-range correlations. The frequency convolutional attention module sets the dilation rate of the dilated convolution in the S-TCM to 5. The normalization layer uses instantaneous layer normalization, which performs normalization independently on the channel dimension for each time step. After applying the sigmoid activation function to generate the frequency-dimensional attention map, the attention map is replicated along the time axis and multiplied with the input of the frequency convolutional attention module.

[0113] Sparse multi-head self-attention adds a dual mask setting to the first layer of attention calculation. That is, the forward mask allows the model to integrate limited future context information, and the backward mask ensures that the attention calculation only focuses on the information of local adjacent frames. The remaining layers set causal masks and cancel the forward mask to prevent the fixed forward mask from causing the model to rely on the number of future frames to increase linearly as the number of attention calculation layers increases. In addition, the frequency convolution attention module is replaced with the time convolution attention module. The specific formula for global average pooling along the frequency axis is:

[0114]

[0115] in To represent the temporal energy, the temporal convolution attention module sets the dilation rate of the dilated convolution in S-TCM to 2. After applying the sigmoid activation function to generate the time dimension attention map, the attention map is copied along the frequency axis and multiplied with the input of the temporal convolution attention module.

[0116] The speaker embedding fusion module performs cross-attention calculation on the fixed speaker embedding vector and the intermediate features generated by the frequency domain fine-grained mask network, where and is the intermediate feature, For speaker embedding, a single speaker embedding is calculated with the intermediate features of each time step to calculate the dot product, and an attention score vector with a length of the sequence is obtained, which represents the similarity between the fixed speaker embedding and the intermediate features of each time step. According to the degree of match between the intermediate features of each time step and the speaker embedding, the model dynamically adjusts the feature representation of the time step. The time step features that are highly correlated with the speaker embedding will be given a higher weight. The output features after the cross-attention calculation are residually connected with the original speaker embedding to obtain an adaptive speaker embedding. The specific formula is:

[0117]

[0118] In the formula is the intermediate feature, is the speaker embedding vector, is a learnable parameter that controls the cross-attention correction term for the speaker embedding vector The generated adaptive embedding The intermediate features of the original backbone network are concatenated and fused. The specific formula is:

[0119]

[0120] In the formula Represents 2D point-wise convolution, effectively integrating intermediate features with adaptive embedding.

[0121] The decoder adds the feature map of the encoder to the output of the current decoder layer through a jump connection with the corresponding layer of the encoder. The grouped temporal convolution blocks of the first three layers use point-by-point convolution and depth-wise separable convolution with different expansion rates (5, 2, and 1, respectively) to gradually expand the receptive field for feature reconstruction, and use transposed convolution to achieve upsampling of the feature map. The last two layers of two-dimensional convolution blocks use two-dimensional convolution with a convolution kernel size of (1, 5), reducing the number of channels from 32 to 16 and then from 16 to 2, respectively, and upsampling with a stride of 2 in the frequency dimension.

[0122] In this embodiment, the speaker embedding extraction network consists of a four-layer stacked recalibrated encoder. The registered speaker speech undergoes a short-time Fourier transform to obtain a time-frequency spectrum. The stacked real and imaginary parts, along with the amplitude, form a new feature tensor and are input into the speaker embedding extraction network. The input spectral features are downsampled by an ERB filter and then passed through a recalibrated encoder, which includes a two-dimensional gated linear unit and a U-Net block, to obtain a speaker embedding. The recalibrated encoder encodes key information about a specific speaker. Therefore, in addition to calculating the loss between clean and enhanced speech, a classification task loss function (AAM softmax additive angular margin loss) is also implemented to enhance the distinguishability of the embedding. This loss function addresses the significant ambiguity at the decision boundary caused by softmax loss by maximizing inter-class distance and minimizing intra-class distance.

[0123] In specific implementation, the beneficial effects that can be ultimately achieved by this embodiment through the above description are:

[0124] 1. Improve speech enhancement in low signal-to-noise ratio environments

[0125] Technical means: The coarse-grained mask network in the time domain captures the overall energy changes and envelope characteristics of the speech signal through long-frame framing preprocessing (40ms frame length), and the fine-grained mask network in the frequency domain uses short-frame framing (32ms frame length) combined with a dual-path sparse attention module to refine the modeling of time-frequency distribution.

[0126] Logical deduction: The coarse-grained stage initially suppresses noise and enhances the dominant signal of the target speech, while the fine-grained stage further eliminates residual noise and optimizes speech details. The cascade of the two alleviates the problems of residual noise and over-suppression of the target speech under low signal-to-noise ratio.

[0127] 2. Effectively suppress background noise and human voice interference

[0128] Technical means: The speaker embedding fusion module introduces the target speaker embedding information and dynamically adjusts the time-frequency unit weights through cross-attention to enhance the ability to focus on specific speakers.

[0129] Logical deduction: Traditional methods have difficulty distinguishing target speech from interfering voices. Speaker embedding, as prior information, guides the model to focus on the target sound source. Combined with refined processing of fine-grained masks in the frequency domain, it significantly suppresses cross-interference in multi-voice environments.

[0130] 3. Reduce computational complexity and support streaming processing

[0131] Technical means: The dual-path sparse attention module combines local windows with dilated attention, and sparse multi-head self-attention reduces redundant calculations; the temporal convolutional attention module introduces causal masks to limit future frame dependencies.

[0132] Logical deduction: The sparse attention mechanism reduces computational complexity from O(n²) to O(n), and the causal mask ensures that the model relies only on historical information, meeting the needs of real-time streaming processing and being suitable for scenarios such as online audio and video calls.

[0133] 4. Lightweight model design adapted to resource-constrained environments

[0134] Technical means: Grouped bidirectional GRU, depthwise separable convolution, ERB filter compression of high-frequency redundant features and other technologies are used to optimize the number of parameters; the speaker embedding extraction network uses compressed ERB domain processing to reduce the computational load.

[0135] Logical deduction: The modular and lightweight design reduces memory usage and computing resource consumption while ensuring performance, allowing the model to be deployed on mobile devices or embedded systems (such as hearing aids).

[0136] 5. Improving the robustness and generalization of speech enhancement

[0137] Technical means: Residual connection, group normalization, instance normalization and other technologies stabilize the training process; AAM softmax loss function optimizes the inter-class discrimination of speaker embedding.

[0138] Logical deduction: Normalization and residual connections alleviate the vanishing gradient problem and improve the model's convergence stability; the high discriminability of speaker embedding enhances the model's adaptability to diverse noise and speaker changes.

[0139] This embodiment obtains noisy speech and extracts coarse-grained features from it using a coarse-grained time-domain masking network to generate a preliminary enhanced speech waveform. This waveform is then subjected to a short-time Fourier transform and combined with target speaker embedding information using a fine-grained frequency-domain masking network to generate the final enhanced speech. This improves the perceived quality and clarity of the desired speech in noisy environments.

[0140] In addition, an embodiment of the present application also proposes a computer-readable storage medium, which stores a program for cascade mask network speech enhancement that integrates auditory processes. When the program for cascade mask network speech enhancement that integrates auditory processes is executed by a processor, the steps of the method for cascade mask network speech enhancement that integrates auditory processes as described above are implemented.

[0141] Reference Figure 6 , Figure 6 This is a structural block diagram of the first embodiment of the cascade mask network speech enhancement device that integrates the auditory process of this application.

[0142] like Figure 6 As shown, the cascade mask network speech enhancement device integrating the auditory process proposed in the embodiment of the present application includes:

[0143] The speech acquisition module 10 is used to acquire noisy speech and extract coarse-grained features from the noisy speech through a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform;

[0144] The speech enhancement module 20 is used to perform short-time Fourier transform on the initially enhanced speech waveform, and generate the final enhanced speech by combining the target speaker embedding information through a frequency domain fine-grained mask network.

[0145] It should be understood that the above is only an example and does not constitute any limitation to the technical solution of the present application. In specific applications, technicians in this field can make settings as needed, and the present application does not impose any restrictions on this.

[0146] This embodiment obtains noisy speech and extracts coarse-grained features from it using a coarse-grained time-domain masking network to generate a preliminary enhanced speech waveform. This waveform is then subjected to a short-time Fourier transform and combined with target speaker embedding information using a fine-grained frequency-domain masking network to generate the final enhanced speech. This improves the perceived quality and clarity of the desired speech in noisy environments.

[0147] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In actual applications, technicians in this field can select part or all of it according to actual needs to achieve the purpose of this embodiment scheme, and no restrictions are imposed here.

[0148] In addition, for technical details not fully described in this embodiment, please refer to the method for cascade mask network speech enhancement integrating auditory processes provided in any embodiment of the present application, which will not be repeated here.

[0149] In addition, it should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or system comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or system. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or system comprising the element.

[0150] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0151] Through the above description of the embodiments, those skilled in the art will clearly understand that the above-mentioned embodiments and methods can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, a magnetic disk, or an optical disk) and includes several instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of this application. The above are only preferred embodiments of this application and do not limit the scope of the patent application. Any equivalent structure or equivalent process transformation made using the contents of this application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the scope of patent protection of this application.

Claims

1. A cascade mask network speech enhancement method integrating auditory processes, characterized in that: include: Acquire noisy speech, and extract coarse-grained features from the noisy speech using a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform; Performing a short-time Fourier transform on the initially enhanced speech waveform, and combining it with target speaker embedding information through a frequency domain fine-grained mask network to generate a final enhanced speech; The time-domain coarse-grained mask network includes an encoder, an enhancement layer, a mask network, and a decoder, and the frequency-domain fine-grained mask network includes an encoder, a dual-path sparse attention module, a speaker embedding fusion module, and a decoder. The dual-path sparse attention module models time-frequency distribution through sparse multi-head self-attention, temporal convolutional attention, and frequency convolutional attention. The speaker embedding fusion module performs cross-attention calculation on the target speaker embedding and intermediate features to optimize the enhancement effect.

2. The method according to claim 1, characterized in that The encoder of the time-domain coarse-grained mask network includes a two-dimensional convolution block and a dense dilated convolution block, which are used to extract high-level features of the input speech; the enhancement layer uses a grouped bidirectional GRU and a unidirectional GRU to model the dependencies within and between frames respectively; the mask network generates a coarse-grained mask through a parallel two-dimensional grouped convolution path, and after element-by-element multiplication with the encoder output, the decoder reconstructs the preliminary enhanced speech waveform, and the decoder uses a dense dilated convolution block and sub-pixel convolution to achieve waveform reconstruction.

3. The method according to claim 2, characterized in that The grouped bidirectional GRU of the enhancement layer groups the input features into two parts along the channel dimension, and processes them in parallel through bidirectional GRU respectively. After the output is spliced, it is subjected to one-dimensional group convolution and group normalization for feature enhancement, and a residual connection is performed with the features output by the encoder to alleviate the gradient disappearance.

4. The method according to claim 1, wherein The dual-path sparse attention module of the frequency domain fine-grained mask network includes: Sparse multi-head self-attention mechanism, which uses local window and dilated attention along the frequency axis and time axis respectively to reduce computational complexity; The time convolution attention module and the frequency convolution attention module capture the long-range correlation of the time axis and frequency axis through compressed temporal convolutional networks, respectively, and generate attention maps to multiply with the input features; Among them, the time convolution attention module performs global average pooling along the frequency axis, and the frequency convolution attention module performs global average pooling along the time axis.

5. The method according to claim 4, characterized in that In the sparse multi-head self-attention mechanism, the attention calculation along the time axis adopts causal mask and double mask settings, where the forward mask in the double mask allows the model to integrate limited future context information, and the backward mask ensures that the attention calculation only focuses on the information of local adjacent frames.

6. The method according to claim 1, characterized in that The speaker embedding fusion module fuses the target speaker embedding vector with the intermediate features through cross attention calculation, including: Fixed speaker embedding vector and cross attention calculation of intermediate features to generate adaptive speaker embedding; Adaptive speaker embedding is concatenated with intermediate features and integrated through two-dimensional point-by-point convolution to form fused features. The target speaker embedding vector is generated by a speaker embedding extraction network and processed in the compressed ERB domain. The speaker embedding extraction network is composed of a stack of recalibrated encoders, which include two-dimensional gated linear units and U-Net blocks, and optimizes the distinguishability of the embedding through AAM softmax additive angular margin loss.

7. The method according to claim 1, characterized in that The framing parameters of the time domain coarse-grained mask network are a frame length of 640 samples and a frame shift of 320 samples; The framing parameters of the frequency domain fine-grained mask network are a frame length of 512 samples and a frame shift of 256 samples, and a square root Hanning window is used for short-time Fourier transform.

8. A cascade mask network speech enhancement device integrating auditory processes, characterized in that: Executing the method according to claim 1, comprising: A speech acquisition module is used to acquire noisy speech and extract coarse-grained features from the noisy speech through a time-domain coarse-grained mask network to generate a preliminary enhanced speech waveform; The speech enhancement module is used to perform short-time Fourier transform on the initially enhanced speech waveform, and generate the final enhanced speech by combining the target speaker embedding information through a frequency domain fine-grained mask network.

9. A computer device, characterized in that: The device comprises: a memory and a processor, wherein the processor executes the method according to any one of claims 1 to 7 when running computer instructions stored in the memory.

10. A computer-readable storage medium, characterized in that The method comprises instructions which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 7.

Images