Multitask speech enhancement method and device

Abstract

The embodiment of the invention provides a multi-task speech enhancement method and device, and the method comprises the steps: carrying out the feature extraction of a microphone signal and a reference signal through a speech enhancement model, and carrying out the adaptive time alignment of the two signals through a dynamic time delay alignment module. The features extracted by the two signal branches are fused by using an adaptive gating module, multi-scale time features and frequency domain features are extracted from time and frequency dimensions, and the accuracy of noise feature, echo feature and voice feature extraction is improved. According to the invention, noise suppression and echo cancellation tasks can be jointly executed, collaborative optimization of multi-task voice enhancement is realized, voice call quality is improved, required computing resources are reduced, lightweight deployment can be realized, and the method can be suitable for resource-limited application scenarios.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of speech processing technology, and in particular to a multi-task speech enhancement method and device. Background Art

[0002] With the development of mobile communication technology, users have increasingly higher requirements for voice quality. Speech enhancement technology improves voice quality by processing voice signals to suppress interference factors such as noise and echo. Existing speech enhancement methods can use one neural network model to remove noise or another neural network model to cancel echo. Using two models for speech signal processing is not only complex and resource-intensive, but also difficult to handle the various nonlinear interferences that may arise from processing noise and echo separately. This results in poor speech enhancement results and is unsuitable for resource-constrained devices. Summary of the Invention

[0003] In view of this, the purpose of the embodiments of the present application is to propose a multi-task speech enhancement method and apparatus to solve the multi-task speech enhancement problem of simultaneously eliminating noise and echo.

[0004] Based on the above objectives, the present invention provides a multi-task speech enhancement method, including:

[0005] Using a first encoding module to upsample the microphone time-frequency signal to obtain a first microphone time-frequency signal feature; using a second encoding module to upsample the first microphone time-frequency signal feature to obtain a second microphone time-frequency signal feature; using a first bottleneck layer to compress and reduce the dimension of the second microphone time-frequency signal feature to obtain a third microphone time-frequency signal feature;

[0006] The reference time-frequency signal is upsampled using a third encoding module to obtain a first reference time-frequency signal feature; the first microphone time-frequency signal feature and the first reference time-frequency signal feature are soft-aligned using a dynamic delay alignment module to obtain an aligned reference time-frequency signal; the aligned reference time-frequency signal is upsampled using a fourth encoding module to obtain a second reference time-frequency signal feature; the second reference time-frequency signal feature is compressed and reduced in dimension using a second bottleneck layer to obtain a third reference time-frequency signal feature;

[0007] Using the adaptive gating module to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused time-frequency signal features;

[0008] The fused time-frequency signal features are downsampled using a decoding module to obtain key time-frequency signal features; and an enhanced speech signal is reconstructed based on the key time-frequency signal features.

[0009] Optionally, using a dynamic delay alignment module to perform echo soft alignment on the first microphone time-frequency signal feature and the first reference time-frequency signal feature to obtain an aligned reference time-frequency signal, including:

[0010] In the microphone signal processing branch, the first microphone time-frequency signal feature is processed by one convolution layer to obtain a query vector, the first microphone time-frequency signal feature is processed by two convolution layers to obtain a key vector, the query vector and the key vector are matrix multiplied to obtain an attention score, the attention score is normalized, and the normalized attention score and the first microphone time-frequency signal feature are matrix multiplied to obtain a first feature portion;

[0011] In the reference signal processing branch, the first reference time-frequency signal feature is processed by a sliding window to obtain a value vector, the first reference time-frequency signal feature is processed by a convolution layer and a sliding window to obtain a key vector, the key vector and the query vector are matrix multiplied to obtain an attention score, the attention score is normalized, the normalized attention score and the value vector are matrix multiplied, and then weighted fusion is performed to obtain a second feature part;

[0012] The first characteristic part and the second characteristic part are spliced ​​together to obtain the aligned reference time-frequency signal.

[0013] Optionally, using an adaptive gating module to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused time-frequency signal features, including:

[0014] Performing feature fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused features;

[0015] Performing multi-scale time domain feature extraction on the fused features using a time domain multi-scale dilated convolutional layer to obtain multi-time scale features;

[0016] Performing multi-scale frequency domain feature extraction on the fused features using a frequency domain multi-scale dilated convolutional layer to obtain multi-frequency domain scale features;

[0017] Based on the multi-time scale features and the multi-frequency scale features, generating a first weight through a gating function;

[0018] Performing matrix multiplication calculation on the third reference time-frequency signal feature and the first weight to obtain a first calculation result;

[0019] Performing matrix multiplication calculation on the time-frequency signal feature of the third microphone and the second weight to obtain a second calculation result; wherein the sum of the first weight and the second weight is 1;

[0020] The first calculation result and the second calculation result are subjected to feature superposition, and feature fusion is performed through a convolution layer to obtain the fused time-frequency signal feature.

[0021] Optionally, the time-domain multi-scale dilated convolution layer uses multi-level dilation rate adjustable convolution to extract echo features and noise features at different time scales;

[0022] The frequency domain multi-scale dilated convolution layer uses independent convolution kernels combined with multi-level dilation to separate the harmonic components of high-frequency noise and low-frequency echo.

[0023] Optionally, before upsampling the microphone time-frequency signal by using the first encoding module, the method further includes:

[0024] Performing short-time Fourier transform on the input microphone signal to obtain a transformed microphone time-frequency signal;

[0025] Performing frequency band compression processing on the transformed microphone time-frequency signal to obtain a compressed microphone time-frequency signal;

[0026] The up-sampling of the microphone time domain signal by using the first encoding module includes: up-sampling the compressed microphone time domain signal by using the first encoding module.

[0027] Optionally, before upsampling the reference time-frequency signal by using the third encoding module, the method further includes:

[0028] Performing short-time Fourier transform on the input reference signal to obtain a transformed reference time-frequency signal;

[0029] Performing frequency band compression processing on the transformed reference time-frequency signal to obtain a compressed reference time-frequency signal;

[0030] The up-sampling of the reference time-frequency signal by using the third encoding module includes: up-sampling the compressed reference time-frequency signal by using the third encoding module.

[0031] Optionally, the performing frequency band compression processing includes:

[0032] The low-frequency components less than or equal to 4 kHz are retained, and the high-frequency components greater than 4 kHz are compressed and merged.

[0033] Optionally, reconstructing the enhanced speech signal based on the key time-frequency signal features includes:

[0034] Performing frequency band decompression processing on the key time-frequency signal features to obtain a decompressed time-frequency signal;

[0035] A complex ratio mask generated by the mask layer is used for speech reconstruction on the decompressed time-frequency signal to obtain a reconstructed time-frequency signal.

[0036] An inverse short-time Fourier transform is performed on the reconstructed time-frequency signal to obtain a transformed speech signal.

[0037] Optionally, the method is implemented based on a speech enhancement model, and a loss function of the speech enhancement model is:

[0038]

[0039] wherein, MSE is a mean square error, SISNR is a scale invariant signal-to-noise ratio, is a model-predicted speech signal, S is a clean speech signal, S_noisy is a speech signal containing noise and echo, and a and β are weights.

[0040] Embodiments of the present application also provide a multi-task speech enhancement device, comprising:

[0041] A microphone signal processing module is configured to up-sample a microphone time-frequency signal by using a first encoding module to obtain a first microphone time-frequency signal feature; up-sample the first microphone time-frequency signal feature by using a second encoding module to obtain a second microphone time-frequency signal feature; and compress and reduce dimensions of the second microphone time-frequency signal feature by using a first bottleneck layer to obtain a third microphone time-frequency signal feature.

[0042] A reference signal processing module is configured to up-sample a reference time-frequency signal by using a third encoding module to obtain a first reference time-frequency signal feature; perform soft alignment on the first microphone time-frequency signal feature and the first reference time-frequency signal feature by using a dynamic time delay alignment module to obtain an aligned reference time-frequency signal; up-sample the aligned reference time-frequency signal by using a fourth encoding module to obtain a second reference time-frequency signal feature; and compress and reduce dimensions of the second reference time-frequency signal feature by using a second bottleneck layer to obtain a third reference time-frequency signal feature.

[0043] A signal fusion processing module is configured to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature by using an adaptive gating module to obtain a fused time-frequency signal feature.

[0044] A signal reconstruction module is configured to down-sample the fused time-frequency signal feature by using a decoding module to obtain a key time-frequency signal feature; and reconstruct an enhanced speech signal based on the key time-frequency signal feature.

[0045] As can be seen from the above, the multi-task speech enhancement method and device provided in the embodiment of the present application utilizes a speech enhancement model to perform feature extraction processing on the microphone signal and the reference signal respectively, utilizes a dynamic time delay alignment module to adaptively time-align the two signals, utilizes an adaptive gating module to fuse the features extracted from the two signal branches, extracts multi-scale time features and frequency domain features from the time and frequency dimensions, and improves the accuracy of noise feature, echo feature, and speech feature extraction. The present application can jointly perform noise suppression and echo cancellation tasks, realize collaborative optimization of multi-task speech enhancement, improve voice call quality, reduce the required computing resources, realize lightweight deployment, and be suitable for resource-constrained application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0047] Figure 1 This is a schematic diagram of the method flow of an embodiment of the present application;

[0048] Figure 2 This is a schematic diagram of the speech enhancement model training process in an embodiment of the present application;

[0049] Figure 3 This is a structural block diagram of the speech enhancement model according to an embodiment of the present application;

[0050] Figure 4 This is a structural block diagram of the dynamic delay alignment module in an embodiment of the present application;

[0051] Figure 5 This is a structural block diagram of the adaptive gating module according to an embodiment of the present application;

[0052] Figure 6 This is a block diagram of the device structure of an embodiment of the present application;

[0053] Figure 7 This is a structural block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.

[0055] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the present application should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in the embodiments of the present application do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0056] like Figures 1-3 As shown, the embodiment of the present application provides a multi-task speech enhancement method, including:

[0057] S101: Using a first encoding module to upsample the microphone time-frequency signal to obtain a first microphone time-frequency signal feature; using a second encoding module to upsample the first microphone time-frequency signal feature to obtain a second microphone time-frequency signal feature; using a first bottleneck layer to compress and reduce the second microphone time-frequency signal feature to obtain a third microphone time-frequency signal feature;

[0058] In this embodiment, a speech enhancement model is used to simultaneously process the microphone signal and the reference signal, wherein the microphone signal contains noise, echo and normal speech signal, and the reference signal contains echo and normal speech signal. The speech enhancement model is divided into two branches to process the input microphone signal and the reference signal respectively, extract noise features, echo features and speech features, and mask the noise and echo based on the extracted features to reconstruct a clean normal speech signal.

[0059] The microphone signal processing branch performs a short-time Fourier transform on the input microphone signal to obtain a transformed microphone time-frequency signal. This transformed microphone time-frequency signal is then subjected to frequency band compression to obtain a compressed microphone time-frequency signal. This involves converting the time-domain microphone signal into a microphone time-frequency signal through a short-time Fourier transform, and then performing frequency band compression on the microphone time-frequency signal to reduce computational complexity.

[0060] In some methods, an equivalent rectangular bandwidth merging module is used to downsample the microphone time-frequency signal, retaining the original frequency band in the micro-low frequency range to obtain the fundamental frequency and main harmonics of the speech, retaining the low-frequency components less than or equal to 4 kHz, accurately capturing the fundamental frequency and harmonics, ensuring the naturalness of the speech, and compressing and merging the high-frequency components greater than 4 kHz into several equivalent rectangular bandwidth (ERB) bands to simulate the wideband perception characteristics of the human ear. This can significantly reduce the computational complexity and achieve the effect of achieving the greatest possible frequency band compression within the range of human ear perception.

[0061] The frequency-band-compressed microphone time-frequency signal is input into the first encoding module, and the first encoding module is used to perform preliminary upsampling on the input microphone time-frequency signal to obtain the first microphone time-frequency signal feature; the second encoding module is used to upsample the first microphone time-frequency signal feature to obtain the second microphone time-frequency signal feature. Both the first encoding module and the second encoding module are composed of multiple hierarchical convolutional layers, batch normalization layers, and activation functions, and use a layer-by-layer stacking method for deep feature learning. Each encoding module contains point convolution and multiple groups of dilated convolution operations to capture the time and frequency features of the signal. Dilated convolution can effectively expand the receptive field and capture echo path features with long-range time dependence without significantly increasing the number of model parameters, reducing computational complexity.

[0062] The time-frequency signal features of the second microphone are input into the first bottleneck layer, and the first bottleneck layer is used to compress and reduce the dimension of the time-frequency signal features of the second microphone to obtain the time-frequency signal features of the third microphone, reducing the computational complexity and removing redundant information, thereby improving the generalization ability of the model.

[0063] S102: Using a third encoding module to upsample the reference time-frequency signal to obtain a first reference time-frequency signal feature; using a dynamic delay alignment module to soft-align the first microphone time-frequency signal feature and the first reference time-frequency signal feature to obtain an aligned reference time-frequency signal; using a fourth encoding module to upsample the aligned reference time-frequency signal to obtain a second reference time-frequency signal feature; using a second bottleneck layer to compress and reduce the dimension of the second reference time-frequency signal feature to obtain a third reference time-frequency signal feature;

[0064] In this embodiment, for the reference signal processing branch, the input reference signal is subjected to a short-time Fourier transform to obtain a transformed reference time-frequency signal; the transformed reference time-frequency signal is subjected to a frequency band compression process to obtain a compressed reference time-frequency signal. That is, the time domain reference signal is converted into a reference time-frequency signal through a short-time Fourier transform, and then the reference time-frequency signal is subjected to frequency band compression to reduce computational complexity. Frequency band compression also uses an equivalent rectangular wide-band merging module to downsample the reference time-frequency signal, retaining low-frequency components less than or equal to 4 kHz, and compressing and merging high-frequency components greater than 4 kHz, so as to achieve the effect of achieving as much frequency band compression as possible within the range of human ear perception.

[0065] The reference time-frequency signal after frequency band compression is input into the third encoding module, and the third encoding module is used to perform preliminary up-sampling on the input reference time-frequency signal to obtain a first reference time-frequency signal feature.

[0066] In some embodiments, taking into account the dynamic delay between the microphone signal and the reference signal, the first microphone time-frequency signal characteristics and the first reference time-frequency signal characteristics are input into a dynamic delay alignment module, and the dynamic delay alignment module is used to soft-align the first microphone time-frequency signal characteristics and the first reference time-frequency signal characteristics to obtain an aligned reference time-frequency signal.

[0067] like Figure 4 As shown, the dynamic time delay alignment module is used to model the time-frequency correlation between the microphone signal and the reference signal. The dynamic time delay alignment module includes a microphone signal processing branch and a reference signal processing branch. In the microphone signal processing branch, the first microphone time-frequency signal feature is processed by a convolution layer to obtain the query vector Q mic The first microphone time-frequency signal feature is processed by two convolutional layers to obtain a key vector, the query vector and the key vector are matrix multiplied to obtain an attention score, the attention score is normalized by the activation function, and the normalized attention score is matrix multiplied with the first microphone time-frequency signal feature to obtain the first feature part of the microphone signal processing branch output.

[0068] In the reference signal processing branch, the first reference time-frequency signal feature is processed by a sliding window to obtain a value vector V sw , the first reference time-frequency signal feature is processed by a convolution layer and a sliding window to obtain a key vector, the key vector K sw The query vector Q of the microphone signal processing branch mic After matrix multiplication, the attention score is obtained, the timing offset is dynamically aligned, and the attention score is normalized by the activation function to obtain the normalized attention score. (The unsqueeze() function is used to align the vector dimensions), the normalized attention score and the value vector V sw After matrix multiplication, weighted fusion is performed to obtain the second characteristic part of the reference signal processing branch output

[0069] The first feature component output by the microphone signal processing branch and the second feature component output by the reference signal processing branch are then concatenated to produce a delay-aligned reference time-frequency signal. Optionally, a point convolution operation is performed on the feature components of the two branches to achieve feature fusion. The fused features contain the aligned speech time-frequency features and echo time-frequency features, and are dimensionally aligned with the fourth encoding module.

[0070] In some embodiments, a soft alignment mechanism using sliding window attention is employed in the reference signal processing branch, employing a sliding window operation to capture features by increasing the time dimension. The window size of the sliding window is a learnable parameter, enabling the model to adaptively adjust the delay range and identify the optimal delay parameter to achieve delay alignment with minimal computational resources. This parameter is converted into a discrete integer delay parameter through a discretization operation. The learnable delay parameter is used to dynamically constrain the window range, dividing the reference signal into multiple sliding windows in the time dimension, generating delay-sensitive key-value pairs, and extracting the time domain features of the reference signal through the sliding window. The dynamic delay alignment module dynamically weights features of different time delays, increasing attention to important time frames and improving the time alignment accuracy of the signal, thereby facilitating the processing of linear and nonlinear echo components.

[0071] The aligned reference time-frequency signal is input into the fourth encoding module, which then upsamples it to obtain the second reference time-frequency signal features. The second bottleneck layer compresses and reduces the dimensionality of the second reference time-frequency signal features to obtain the third reference time-frequency signal features. This reduces computational complexity, removes redundant information, and improves the model's generalization capabilities. Both the third and fourth encoding modules consist of multiple layers of convolutional layers, batch normalization layers, and activation functions, using a layer-by-layer stacking approach for deep feature learning.

[0072] S103: Using an adaptive gating module to perform feature extraction and fusion processing on the time-frequency signal features of the third microphone and the third reference time-frequency signal features to obtain fused time-frequency signal features;

[0073] In this embodiment, the adaptive gating module is used to extract and fuse the third microphone time-frequency signal features and the third reference time-frequency signal features obtained after the processing branch of the microphone signal is processed, to obtain the fused time-frequency signal features, including:

[0074] Performing feature fusion processing on the time-frequency signal features of the third microphone and the third reference time-frequency signal features to obtain fused features;

[0075] Performing multi-scale time domain feature extraction on the fused features using a time domain multi-scale dilated convolutional layer to obtain multi-time scale features;

[0076] The frequency domain multi-scale dilated convolution layer is used to extract multi-scale frequency domain features from the fused features to obtain multi-scale frequency domain features;

[0077] Based on the multi-time scale features and the multi-frequency scale features, a first weight is generated through a gating function;

[0078] Performing matrix multiplication calculation on the third reference time-frequency signal feature and the first weight to obtain a first calculation result;

[0079] Performing matrix multiplication calculation on the time-frequency signal feature of the third microphone and the second weight to obtain a second calculation result; wherein the sum of the first weight and the second weight is 1;

[0080] The first calculation result and the second calculation result are superimposed on each other, and feature fusion is performed through a convolution layer to obtain the fused time-frequency signal features.

[0081] like Figure 5 As shown, the adaptive gating module is used to unify the echo cancellation and noise suppression tasks, using a gating mechanism to achieve synergistic enhancement of echo cancellation and noise suppression, avoiding the error accumulation problem caused by cascade processing. The adaptive gating module includes a time-domain multi-scale dilated convolution layer and a frequency-domain multi-scale dilated convolution layer. The input third microphone time-frequency signal features and the third reference time-frequency signal features are first fused. The fused features are then processed by the time-domain multi-scale dilated convolution layer and the frequency-domain multi-scale dilated convolution layer respectively. The time-domain multi-scale dilated convolution layer is used to extract multi-scale time-domain features from the input fused features. That is, multi-scale dilated convolution uses convolution kernels with different dilation rates in the time dimension to capture short-term echo noise, medium-term echo noise, and long-term echo noise features while avoiding feature redundancy. The frequency-domain multi-scale dilated convolution layer is used to extract multi-scale frequency-domain features from the input fused features. That is, multi-scale dilated convolution uses independent convolution kernels in the frequency dimension to model harmonic correlation and separate overlapping echo noise.

[0082] After extracting multi-time-scale and multi-frequency-scale features, a gating function (e.g., a Sigmoid activation function) dynamically generates a first weight G. Matrix multiplication is then performed on the third reference time-frequency signal feature and the first weight. The third microphone time-frequency signal feature is then matrix multiplied with the second weight (1-G). The results of the two paths are then combined and fused, and a convolutional layer is used to fully fuse the features to produce the fused time-frequency signal features. By dynamically generating weights, the module accurately selects features for echo cancellation and noise suppression paths, improving system robustness, enhancing adaptability to various interference types, and enhancing signal clarity and audio quality.

[0083] In some implementations, the time-domain multi-scale dilated convolution layer employs multi-level dilation with adjustable convolution rates. By adjusting the dilation rate through adaptive parameters, the layer dynamically adapts to the time scale requirements of different scenarios and extracts echo and noise features at different time scales. For example, dilation rates of 1, 3, and 5 are used to capture the dependencies between short-term localized echo noise, medium-term periodic echo noise, and long-term far-field reflection echo noise, respectively. The frequency-domain multi-scale dilated convolution layer utilizes independent convolution kernels in the frequency dimension, combined with multi-level dilation (for example, dilation rates of 1, 2, and 4), to separate the harmonic components of high-frequency noise and low-frequency echoes.

[0084] S104: down-sampling the fused time-frequency signal features using a decoding module to obtain key time-frequency signal features; and reconstructing the enhanced speech signal based on the key time-frequency signal features.

[0085] In this embodiment, the time-frequency signal features obtained after processing by the adaptive gating module are input to the decoding module, which performs a deconvolution operation on the input time-frequency signal features to achieve a downsampling restoration process, and extracts the key time-frequency signal features after downsampling. The decoding module adopts a symmetrical structure to gradually restore the frequency resolution to ensure that important information is not lost during the encoding process. Figure 3 As shown, jump connections are used between the first encoding module and the decoding module, and between the fourth encoding module and the decoding module. Through cross-layer information fusion, the degradation problem of the deep network is alleviated and the information loss in the encoding process is reduced.

[0086] Each encoding module upsamples the compressed high-frequency time-frequency signal, and the decoding module combines deconvolution with skip connections to restore high-frequency details. Taking into account the human ear's sensitivity to different frequency bands, this reduces model complexity while preserving the key features and information of the microphone and reference signals. By dividing the spectral features of the microphone and reference signals into low-frequency and high-frequency components, the model can more efficiently extract and learn important information from the speech signal, reducing the computational burden of subsequent network layers.

[0087] In some embodiments, reconstructing an enhanced speech signal based on key time-frequency signal features includes:

[0088] Perform frequency band decompression processing on key time-frequency signal features to obtain decompressed time-frequency signals;

[0089] The decompressed time-frequency signal is reconstructed using the complex ratio mask generated by the mask layer to obtain a reconstructed time-frequency signal;

[0090] Perform inverse short-time Fourier transform on the reconstructed time-frequency signal to obtain the transformed speech signal.

[0091] In this embodiment, the key time-frequency signal features output by the decoding module are subjected to frequency band decompression processing to obtain a decompressed time-frequency signal. The decompressed time-frequency signal is then subjected to speech reconstruction using a complex ratio mask generated by a mask layer to obtain a reconstructed time-frequency signal. The reconstructed time-frequency signal is then subjected to an inverse short-time Fourier transform to obtain a transformed speech signal. An equivalent rectangular bandwidth separation module is used to perform frequency band decompression processing on the key time-frequency signal features, and band separation restoration is used to restore the key time-frequency signal features to the original resolution of the input signal. The complex ratio mask generated by the mask layer is used to mask the noise and echo portions of the decompressed time-frequency signal, retaining only the clean time-frequency signal portion. The clean time-frequency signal portion is then subjected to an inverse short-time Fourier transform to restore the speech signal after speech enhancement.

[0092] like Figure 2 As shown, in some embodiments, the training method of the speech enhancement model includes:

[0093] A speech, echo, and noise dataset was obtained, and data augmentation methods were used to generate microphone signals containing noise and echo, clean speech signals, and a reference signal dataset containing echo. The dataset combined data resources from the ICASSP Echo Cancellation Challenge and the Noise Suppression Challenge. Several hours of synthetic samples of remote single calls were obtained from the Echo Cancellation Challenge. In the Noise Suppression Challenge, for each noisy audio, an echo signal from the Echo Cancellation Challenge was randomly selected and mixed with the noisy audio to construct a new microphone signal. In this process, the signal-to-noise ratio of the echo and noise was evenly distributed between predetermined thresholds. To further enrich the dataset, several hours of dual-talk scenario data were constructed, and a Chinese speech dataset recorded using professional equipment was used for further training in specific scenarios to improve the versatility and robustness of the model.

[0094] The microphone signal samples and reference signal samples in the dataset are input into the speech enhancement model for model training. The speech enhancement model processes the microphone signal samples and reference signal samples and outputs a speech signal. Based on the output speech signal and the corresponding clean speech signal in the dataset, a loss value is calculated according to the loss function. The model then performs backpropagation based on this loss value and continues forward reasoning. Following this process, after multiple rounds of training, the loss value gradually decreases and eventually converges, completing model training and obtaining the trained speech enhancement model. During training, the weights of the adaptive gating module dynamically adjust the gradient return path, prioritizing the optimization of task-related features.

[0095] In some implementations, the time-domain scale-invariant signal-to-noise ratio and the frequency-domain amplitude spectrum loss are combined to balance speech quality and spectral fidelity. The designed loss function is:

[0096]

[0097] Among them, MSE is the mean square error, SISNR is the scale-invariant signal-to-noise ratio, is the speech signal predicted by the model, S is the clean speech signal, S_noisy is the speech signal containing echo and noise, α and β are weights, which can be dynamically adjusted according to task requirements.

[0098] The multi-task speech enhancement method provided in the embodiment of the present application utilizes a speech enhancement model to perform feature extraction processing on the microphone signal and the reference signal respectively, which can improve the performance of signal feature fusion and enhancement, and adopt appropriate upsampling and downsampling to retain the key features in the echo cancellation and noise suppression process; utilizes a dynamic delay alignment module to adaptively time-align the two signals, which can adapt to nonlinear echo path changes and dynamic environmental interference to achieve more accurate signal alignment; utilizes an adaptive gating module to fuse the features extracted from the two signal branches, extracts multi-scale time features and frequency domain features from the time and frequency dimensions, and improves the accuracy of noise feature, echo feature, and speech feature extraction. The method of the present application can jointly perform noise suppression and echo cancellation tasks and achieve high-level performance while consuming very little computing resources, realize collaborative optimization of multi-task speech enhancement, improve voice call quality, and achieve lightweight deployment, which is suitable for resource-constrained application scenarios.

[0099] It should be noted that the method of the embodiment of the present application can be performed by a single device, such as a computer or server. The method of this embodiment can also be applied in a distributed scenario and performed by multiple devices working together. In such a distributed scenario, one of the multiple devices may only perform one or more steps of the method of the embodiment of the present application, and the multiple devices will interact with each other to complete the method.

[0100] It should be noted that the foregoing description of this specification is based on specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0101] like Figure 6 As shown, the embodiment of the present application provides a multi-task speech enhancement device, comprising:

[0102] a microphone signal processing module configured to upsample the microphone time-frequency signal using the first encoding module to obtain a first microphone time-frequency signal feature; upsample the first microphone time-frequency signal feature using the second encoding module to obtain a second microphone time-frequency signal feature; and compress and reduce the second microphone time-frequency signal feature using the first bottleneck layer to obtain a third microphone time-frequency signal feature;

[0103] a reference signal processing module configured to upsample the reference time-frequency signal using the third encoding module to obtain a first reference time-frequency signal feature; soft-align the first microphone time-frequency signal feature and the first reference time-frequency signal feature using the dynamic delay alignment module to obtain an aligned reference time-frequency signal; upsample the aligned reference time-frequency signal using the fourth encoding module to obtain a second reference time-frequency signal feature; and compress and reduce the second reference time-frequency signal feature using the second bottleneck layer to obtain a third reference time-frequency signal feature;

[0104] a signal fusion processing module, configured to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature using the adaptive gating module to obtain fused time-frequency signal features;

[0105] The signal reconstruction module is used to downsample the fused time-frequency signal features using the decoding module to obtain key time-frequency signal features; and reconstruct the enhanced speech signal based on the key time-frequency signal features.

[0106] For the convenience of description, the above devices are described as being functionally divided into various modules. Of course, when implementing the embodiments of the present application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0107] The apparatus of the above embodiment is used to implement the corresponding method in the above embodiment and has the beneficial effects of the corresponding method embodiment, which will not be described in detail here.

[0108] Figure 7 10 is a schematic diagram showing a more specific hardware structure of an electronic device provided in this embodiment. The device may include: a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are communicatively connected to each other within the device via the bus 1050.

[0109] The processor 1010 can be implemented using a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0110] The memory 1020 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage devices, dynamic storage devices, etc. The memory 1020 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1020 and is called and executed by the processor 1010.

[0111] The input / output interface 1030 is used to connect input / output modules to implement information input and output. The input / output modules can be configured as components within the device (not shown in the figure) or can be externally connected to the device to provide corresponding functions. Input devices may include a keyboard, mouse, touch screen, microphone, various sensors, etc., and output devices may include a display, speaker, vibrator, indicator light, etc.

[0112] The communication interface 1040 is used to connect to a communication module (not shown) to enable communication between the device and other devices. The communication module can communicate via a wired method (such as USB, network cable, etc.) or a wireless method (such as mobile network, WiFi, Bluetooth, etc.).

[0113] The bus 1050 comprises a pathway for transmitting information between the various components of the device (eg, the processor 1010 , the memory 1020 , the input / output interface 1030 , and the communication interface 1040 ).

[0114] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040, and the bus 1050, in a specific implementation, the device may also include other components necessary for normal operation. In addition, it will be understood by those skilled in the art that the above device may only include the components necessary to implement the embodiments of this specification, and does not necessarily include all the components shown in the figure.

[0115] The electronic devices of the above embodiments are used to implement the corresponding methods in the above embodiments and have the beneficial effects of the corresponding method embodiments, which will not be described in detail here.

[0116] The computer-readable media of this embodiment include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0117] Those skilled in the art should understand that the discussion of any of the above embodiments is merely illustrative and is not intended to imply that the scope of the present disclosure (including the claims) is limited to these examples. Within the scope of the present disclosure, the above embodiments or technical features in different embodiments may be combined, the steps may be implemented in any order, and there are many other variations of the different aspects of the embodiments of the present application as described above, which are not provided in detail for the sake of simplicity.

[0118] In addition, to simplify the description and discussion, and in order not to make the embodiment of the present application difficult to understand, the known power supply / ground connection with the integrated circuit (IC) chip and other components may or may not be shown in the accompanying drawings provided. In addition, the device can be shown in the form of a block diagram to avoid making the embodiment of the present application difficult to understand, and this also takes into account the following fact, that is, the details of the implementation method of these block diagram devices are highly dependent on the platform to be implemented in the embodiment of the present application (that is, these details should be fully within the scope of understanding of those skilled in the art). When specific details (for example, circuits) are set forth to describe exemplary embodiments of the present disclosure, it will be apparent to those skilled in the art that the embodiment of the present application can be implemented without these specific details or when these specific details are changed. Therefore, these descriptions should be considered to be illustrative rather than restrictive.

[0119] Although the present disclosure has been described in conjunction with specific embodiments thereof, many alternatives, modifications, and variations of these embodiments will be apparent to those skilled in the art based on the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed.

[0120] The embodiments of the present application are intended to cover all such substitutions, modifications, and variations that fall within the broad scope of the appended claims. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this disclosure.

Claims

1. A multi-task speech enhancement method, characterized in that: include: Upsampling the microphone time-frequency signal using the first encoding module to obtain a first microphone time-frequency signal feature; Upsampling the first microphone time-frequency signal feature using a second encoding module to obtain a second microphone time-frequency signal feature, and compressing and reducing the second microphone time-frequency signal feature using a first bottleneck layer to obtain a third microphone time-frequency signal feature; Upsampling the reference time-frequency signal using a third encoding module to obtain a first reference time-frequency signal feature; Using a dynamic time delay alignment module, soft-align the time-frequency signal characteristics of the first microphone and the first reference time-frequency signal characteristics to obtain an aligned reference time-frequency signal; Using a fourth encoding module to upsample the aligned reference time-frequency signal to obtain a second reference time-frequency signal feature; using a second bottleneck layer to compress and reduce the dimension of the second reference time-frequency signal feature to obtain a third reference time-frequency signal feature; Using the adaptive gating module to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused time-frequency signal features; The fused time-frequency signal features are downsampled using a decoding module to obtain key time-frequency signal features; and an enhanced speech signal is reconstructed based on the key time-frequency signal features.

2. The method according to claim 1, characterized in that Using a dynamic delay alignment module to perform echo soft alignment on the first microphone time-frequency signal characteristics and the first reference time-frequency signal characteristics to obtain an aligned reference time-frequency signal, including: In the microphone signal processing branch, the first microphone time-frequency signal feature is processed by one convolution layer to obtain a query vector, the first microphone time-frequency signal feature is processed by two convolution layers to obtain a key vector, the query vector and the key vector are matrix multiplied to obtain an attention score, the attention score is normalized, and the normalized attention score and the first microphone time-frequency signal feature are matrix multiplied to obtain a first feature portion; In the reference signal processing branch, the first reference time-frequency signal feature is processed by a sliding window to obtain a value vector, the first reference time-frequency signal feature is processed by a convolution layer and a sliding window to obtain a key vector, the key vector and the query vector are matrix multiplied to obtain an attention score, the attention score is normalized, the normalized attention score and the value vector are matrix multiplied, and then weighted fusion is performed to obtain a second feature part; The first characteristic part and the second characteristic part are spliced ​​together to obtain the aligned reference time-frequency signal.

3. The method according to claim 1, characterized in that The adaptive gating module is used to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused time-frequency signal features, including: Performing feature fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature to obtain fused features; Performing multi-scale time domain feature extraction on the fused features using a time domain multi-scale dilated convolutional layer to obtain multi-time scale features; Performing multi-scale frequency domain feature extraction on the fused features using a frequency domain multi-scale dilated convolutional layer to obtain multi-frequency domain scale features; Based on the multi-time scale features and the multi-frequency scale features, generating a first weight through a gating function; Performing matrix multiplication calculation on the third reference time-frequency signal feature and the first weight to obtain a first calculation result; Performing matrix multiplication calculation on the time-frequency signal feature of the third microphone and the second weight to obtain a second calculation result; wherein the sum of the first weight and the second weight is 1; The first calculation result and the second calculation result are subjected to feature superposition, and feature fusion is performed through a convolution layer to obtain the fused time-frequency signal feature.

4. The method according to claim 3, characterized in that The time domain multi-scale dilated convolution layer adopts multi-level dilation rate adjustable convolution to extract echo features and noise features at different time scales; The frequency domain multi-scale dilated convolution layer uses independent convolution kernels combined with multi-level dilation to separate the harmonic components of high-frequency noise and low-frequency echo.

5. The method according to claim 1, wherein Before up-sampling the microphone time-frequency signal by the first encoding module, the method further includes: Performing short-time Fourier transform on the input microphone signal to obtain a transformed microphone time-frequency signal; Performing frequency band compression processing on the transformed microphone time-frequency signal to obtain a compressed microphone time-frequency signal; The up-sampling of the microphone time domain signal by using the first encoding module includes: up-sampling the compressed microphone time domain signal by using the first encoding module.

6. The method according to claim 1, characterized in that Before up-sampling the reference time-frequency signal by the third encoding module, the method further includes: Performing short-time Fourier transform on the input reference signal to obtain a transformed reference time-frequency signal; Performing frequency band compression processing on the transformed reference time-frequency signal to obtain a compressed reference time-frequency signal; The up-sampling of the reference time-frequency signal by using the third encoding module includes: up-sampling the compressed reference time-frequency signal by using the third encoding module.

7. The method according to claim 5 or 6, characterized in that The frequency band compression processing includes: The low-frequency components less than or equal to 4 kHz are retained, and the high-frequency components greater than 4 kHz are compressed and merged.

8. The method according to claim 1, characterized in that Reconstructing an enhanced speech signal based on the key time-frequency signal features includes: Performing frequency band decompression processing on the key time-frequency signal features to obtain a decompressed time-frequency signal; Performing speech reconstruction on the decompressed time-frequency signal using a complex ratio mask generated by the mask layer to obtain a reconstructed time-frequency signal; Performing inverse short-time Fourier transform on the reconstructed time-frequency signal to obtain a transformed speech signal.

9. The method according to claim 1, characterized in that The method is implemented based on a speech enhancement model, and the loss function of the speech enhancement model is: Among them, MSE is the mean square error, SISNR is the scale-invariant signal-to-noise ratio, is the speech signal predicted by the model, S is the clean speech signal, S_noisy is the speech signal containing noise and echo, and α and β are weights.

10. A multi-task speech enhancement device, characterized in that: include: a microphone signal processing module, configured to upsample the microphone time-frequency signal using the first encoding module to obtain a first microphone time-frequency signal feature; Upsampling the first microphone time-frequency signal feature using a second encoding module to obtain a second microphone time-frequency signal feature, and compressing and reducing the second microphone time-frequency signal feature using a first bottleneck layer to obtain a third microphone time-frequency signal feature; a reference signal processing module, configured to upsample the reference time-frequency signal using the third encoding module to obtain a first reference time-frequency signal feature; Using a dynamic time delay alignment module, soft-align the time-frequency signal characteristics of the first microphone and the first reference time-frequency signal characteristics to obtain an aligned reference time-frequency signal; Using a fourth encoding module to upsample the aligned reference time-frequency signal to obtain a second reference time-frequency signal feature; using a second bottleneck layer to compress and reduce the dimension of the second reference time-frequency signal feature to obtain a third reference time-frequency signal feature; a signal fusion processing module, configured to perform feature extraction and fusion processing on the third microphone time-frequency signal feature and the third reference time-frequency signal feature using the adaptive gating module to obtain fused time-frequency signal features; The signal reconstruction module is used to downsample the fused time-frequency signal features using the decoding module to obtain key time-frequency signal features; and reconstruct the enhanced speech signal based on the key time-frequency signal features.

Images