Speech enhancement method, training method, and device

Abstract

The application provides a speech enhancement method, a training method and equipment. A constructed speech enhancement model comprises a first stage model for obtaining noise removal and a second stage model for retaining only early reverberation. Decoupling of noise removal and removal of reverberation except early reverberation is achieved, so that the network of each stage can focus on noise removal or reverberation removal. The model parameter quantity is reduced, and the model complexity is reduced while ensuring the effect.

Description

Technical Field

[0001] This application relates to the field of speech technology, and in particular to a speech enhancement method, training method and device. Background Technology

[0002] Speech enhancement (SE) is a widely used technique in speech processing to remove interference noise (such as environmental noise, equipment noise, reverberation, etc.) from speech signals, thereby improving speech intelligibility (STOI, Short-Time Objective Intelligence) and signal-to-noise ratio (SNR). Speech enhancement technology is widely used in speech recognition, speech translation, smart speakers, call noise reduction, and many other fields, and its effectiveness is particularly significant in noisy environments and low SNR scenarios.

[0003] Existing speech enhancement methods are mainly divided into traditional signal processing methods and deep learning-based methods. Traditional methods, such as spectral subtraction and Wiener filtering, while effective in handling simple noise, show significant performance degradation in low signal-to-noise ratio and complex noise environments, making them difficult to effectively address. In contrast, deep learning-based speech enhancement techniques, especially neural network models, demonstrate better noise reduction performance in complex conditions such as non-stationary noise and mixed noise scenarios.

[0004] However, these deep learning models typically involve high computational complexity and a large number of parameters, posing significant challenges to deployment on resource-constrained hardware platforms such as embedded devices. Furthermore, existing neural network models largely focus on noise processing. While breakthroughs have been made in noise reduction, significant limitations remain in reverberation removal, particularly in tasks requiring simultaneous noise and reverberation reduction. This increases the difficulty of model training and makes it difficult to balance performance. Noise reduction and reverberation reduction are two complex sub-tasks, typically processing different aspects of the signal. Solving both problems simultaneously is even more complex, and existing models generally perform poorly in this area. On the other hand, training existing models often relies on pre-prepared datasets with limited representational space, which can easily lead to overfitting during training and an inability to adapt well to changes in different environments. Summary of the Invention

[0005] This application proposes a speech enhancement method, training method, and device that can solve one of the problems existing in the background art.

[0006] To achieve the above objectives, this application adopts the following technical solution:

[0007] Firstly, a training method for a speech enhancement model is provided, the training method comprising:

[0008] Obtain training data, the training data including: first speech data, second speech data, and third speech data, wherein the first speech data contains noise and full reverberation, the second speech data contains full reverberation, and the third speech data contains early reverberation, the early reverberation being extracted from the full reverberation; and

[0009] The speech enhancement model is trained using the training data.

[0010] The speech enhancement model includes: a first-stage model for obtaining the noise removal and a second-stage model for retaining only the early reverberation.

[0011] The first-stage model includes, in sequence: a first time-frequency domain transformation module for performing time-frequency domain transformation on training data to obtain a first spectrogram; a first encoder for encoding the first spectrogram to obtain a first frequency domain feature; a first decoder for decoding the first frequency domain feature to obtain a mask; and a first dot product module for obtaining a second spectrogram from the first spectrogram and the mask.

[0012] The second stage model includes, in sequence: a second encoder for encoding the second spectrogram to obtain a second frequency domain feature, a second decoder for decoding the second frequency domain feature to obtain a third spectrogram, and a second time-frequency conversion module for performing frequency-time conversion on the third spectrogram to obtain enhanced speech.

[0013] Based on the above technical solution, the constructed speech enhancement model includes: a first-stage model for obtaining the noise removal and a second-stage model for retaining only the early reverberation. This decouples the noise removal and the removal of other reverberations besides the early reverberation, allowing each stage of the network to focus on noise removal or reverberation removal. While ensuring the effect, the number of model parameters is reduced and the model complexity is decreased.

[0014] In one possible design approach of the first aspect, obtaining training data specifically includes:

[0015] Obtain clean speech, noisy audio, and impulse response signals;

[0016] Using the clean speech and the impulse response signal, a plurality of second speech data and the third speech data are randomly generated; and

[0017] Using the second speech data and the noise audio, several sets of the first speech data are randomly generated.

[0018] In one possible design of the first aspect, obtaining training data further includes: adjusting the length, volume, and / or signal-to-noise ratio of the first speech data, the second speech data, and the third speech data.

[0019] Based on the above technical solutions, more data can be synthesized, the data space can be expanded, model overfitting can be avoided, and generalization ability can be improved.

[0020] In one possible design of the first aspect, the first stage model further includes: a first enhancer disposed between the first encoder and the first decoder for enhancing the first frequency domain features by utilizing the contextual relationships between the captured features; the second stage model further includes: a second enhancer disposed between the second encoder and the second decoder for enhancing the second frequency domain features by utilizing the contextual relationships between the captured features.

[0021] In one possible design of the first aspect, the first enhancer and the second enhancer employ a dual-path cyclic speech separation neural network.

[0022] Based on the above technical solution, by utilizing the contextual relationships between captured features, adjacent frequency components can be used to assist in the repair, making the model more stable.

[0023] In one possible design of the first aspect, the first encoder and the second encoder adopt the same structure as follows: a first branch, a second branch, and a second dot product module. The first branch includes, in sequence, a first convolutional neural network layer, a first normalization layer, and a first activation layer. The second branch includes, in sequence, a second convolutional neural network layer and a second normalization layer. The first convolutional neural network layer and the second convolutional neural network layer are used as inputs, and the outputs of the first activation layer and the second normalization layer are used as inputs to the second dot product module.

[0024] In one possible design of the first aspect, the first decoder and the second decoder adopt the same and sequentially arranged structure as follows: a third convolutional neural network layer and several stacked layers with the same input and output directions, the stacked layers including: a third normalization layer, a second activation layer and an upsampling layer.

[0025] Based on the above technical solutions, a codec with low computational complexity is proposed, which reduces the number of model parameters and the operating load.

[0026] In one possible design approach of the first aspect, the first-stage model and the second-stage model use the same loss function L as follows:

[0027]

[0028] L = 0.7 × L mag +0.3×L ri

[0029] in, Let S represent the output of each stage, S represent the training objective of each stage, real(·) represents the operation of finding the real part, imag(·) represents the operation of finding the imaginary part, ‖·‖2 represents the L2 norm, and ‖·‖ represents the L1 norm.

[0030] Secondly, a speech enhancement method is provided, the speech enhancement method comprising:

[0031] Obtain the speech data to be enhanced; and

[0032] The speech enhancement model trained above is used to process the speech data to be enhanced, and the speech enhancement result is obtained.

[0033] Thirdly, an electronic device is provided, comprising: a processor, and a memory coupled to the processor, the memory for storing a computer program; the processor for executing the computer program stored in the memory to cause the electronic device to perform the training method as described in any possible implementation of the first aspect, or to perform the speech enhancement method as described in the second aspect.

[0034] In this application, the electronic device described in the third aspect may be a terminal device or a network device, or a chip (system) or other component or assembly disposed within the terminal device or network device.

[0035] Fourthly, a computer-readable storage medium is provided, including a computer program or instructions that, when executed on a computer, cause the computer to perform the training method as described in any possible implementation of the first aspect, or to perform the speech enhancement method as described in the second aspect.

[0036] Fifthly, a computer program product is provided, comprising: a computer program or instructions that, when executed on a computer, cause the computer to perform the training method as described in any possible implementation of the first aspect, or to perform the speech enhancement method as described in the second aspect. Attached Figure Description

[0037] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0038] Figure 1 This is a flowchart of the training method for the speech enhancement model provided in Embodiment 1 of this application;

[0039] Figure 2 This is a schematic diagram of the speech enhancement model provided in Embodiment 1 of this application;

[0040] Figure 3 This is a diagram showing the overall structure of the two-stage model provided in Embodiment 2 of this application;

[0041] Figure 4 This is a diagram of the encoder substructure provided in Embodiment 2 of this application;

[0042] Figure 5 This is a diagram of the decoder substructure provided in Embodiment 2 of this application;

[0043] Figure 6 This is a comparison diagram of the street noise enhancement effect before (left) and after (right) enhancement provided in Embodiment 2 of this application;

[0044] Figure 7 This is a comparison diagram of the effects of outdoor noise and indoor reverberation audio enhancement after (left) and before (right) enhancement, provided in Embodiment 2 of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0048] Example 1

[0049] like Figure 1 As shown in the figure, this embodiment provides a training method for a speech enhancement model, the training method including:

[0050] Step S101: Obtain training data, which includes: first speech data, second speech data, and third speech data. The first speech data contains noise and full reverberation, the second speech data contains full reverberation, and the third speech data contains early reverberation, which is extracted from the full reverberation.

[0051] Step S102: Train the speech enhancement model using the training data.

[0052] like Figure 2 As shown, the speech enhancement model 200 includes: a first-stage model 201 for obtaining the noise removal and a second-stage model 301 for retaining only the early reverberation.

[0053] The first-stage model 201 includes, in sequence: a first time-frequency domain transformation module 210 for performing time-frequency domain transformation on training data to obtain a first spectrum; a first encoder 211 for encoding the first spectrum to obtain a first frequency domain feature; a first decoder 212 for decoding the first frequency domain feature to obtain a mask; and a first dot product module 213 for obtaining a second spectrum from the first spectrum and the mask.

[0054] The second stage model 301 includes, in sequence: a second encoder 310 for encoding the second spectrogram to obtain a second frequency domain feature, a second decoder 311 for decoding the second frequency domain feature to obtain a third spectrogram, and a second time-frequency conversion module 312 for performing frequency-time conversion on the third spectrogram to obtain enhanced speech.

[0055] In one implementation, step S101 may specifically include:

[0056] Obtain clean speech, noisy audio, and impulse response signals;

[0057] Using the clean speech and the impulse response signal, a plurality of second speech data and the third speech data are randomly generated; and

[0058] Using the second speech data and the noise audio, several sets of the first speech data are randomly generated.

[0059] Specifically, clean speech refers to raw speech data in languages ​​such as Mandarin, English, and German, which is largely free of noise or reverberation effects from various scenarios. Clean speech can originate from open-source datasets on the internet, freely downloadable public audio resources, or legally collected audio data using self-built recording equipment.

[0060] Noise audio can be sourced from publicly available noise databases or from specific noise recorded based on actual application scenarios.

[0061] The impact response signal superimposed on the speech can be used to simulate reverberation in different scenarios, such as indoor and forest environments.

[0062] In one implementation, obtaining training data further includes: adjusting the length, volume, and / or signal-to-noise ratio of the first speech data, the second speech data, and the third speech data.

[0063] Specifically, length adjustment refers to adjusting all speech data to a uniform length L to facilitate later training. For example, speech data exceeding L may be truncated, or speech data less than L may be repeatedly supplemented.

[0064] Volume adjustment can include: voice amplitude adjustment, and decibel adjustment relative to full scale. Voice amplitude adjustment refers to adjusting the voice amplitude using an adjustment factor to make the volume level meet the requirements. Decibels adjustment relative to full scale can meet the pickup needs at different distances, and uses an adjustment factor similar to an adjustment coefficient.

[0065] Signal-to-noise ratio adjustment is mainly used to adjust the noise amplitude.

[0066] By making the above adjustments multiple times, more data can be synthesized, the data space can be expanded, model overfitting can be avoided, and generalization ability can be improved.

[0067] In one implementation, the first stage model further includes: a first enhancer disposed between the first encoder and the first decoder for enhancing the first frequency domain features by utilizing the contextual relationships between the captured features; the second stage model further includes: a second enhancer disposed between the second encoder and the second decoder for enhancing the second frequency domain features by utilizing the contextual relationships between the captured features.

[0068] In one implementation, the first enhancer and the second enhancer employ a dual-path cyclic speech separation neural network.

[0069] Specifically, the spectrum is obtained by speech transformation. The frequency magnitudes form a harmonic structure from low to high. There is a close relationship between adjacent frequencies. If the current frequency position is interfered with or lost by noise, adjacent frequency components can be used to help repair it. At the same time, speech has a strong temporal sequence, which is reflected in the spectrum as the current frame signal is related to the historical frame signal, and the closer the signal is, the greater the correlation between them.

[0070] By using a dual-path cyclic speech separation neural network, the contextual relationships of feature frequencies and time can be modeled separately. By capturing these contextual relationships, frequency domain features can be enhanced.

[0071] In one implementation, the first encoder and the second encoder adopt the same structure as follows: a first branch, a second branch, and a second dot product module. The first branch includes, in sequence, a first convolutional neural network layer, a first normalization layer, and a first activation layer. The second branch includes, in sequence, a second convolutional neural network layer and a second normalization layer. The first convolutional neural network layer and the second convolutional neural network layer are used as inputs, and the outputs of the first activation layer and the second normalization layer are used as inputs to the second dot product module.

[0072] The first decoder and the second decoder adopt the same structure as the second decoder and are arranged in sequence as follows: a third convolutional neural network layer and several stacked layers with the same input and output directions. The stacked layers include: a third normalization layer, a second activation layer and an upsampling layer.

[0073] Based on the above technical solutions, a codec with low computational complexity is proposed, which reduces the number of model parameters and the operating load.

[0074] In one implementation, the first-stage model and the second-stage model use the same loss function L as follows:

[0075]

[0076] L = 0.7 × L mag +0.3×L ri

[0077] in, Let S represent the output of each stage, S represent the training objective of each stage, real(·) represents the operation of finding the real part, imag(·) represents the operation of finding the imaginary part, ‖·‖2 represents the L2 norm, and ‖·‖ represents the L1 norm.

[0078] Of course, it is understood that the above loss function is only an example, and the coefficients 0.3 and 0.7 are also only examples and not the only choice for implementing this embodiment.

[0079] This embodiment also provides a speech enhancement method, the speech enhancement method including:

[0080] Obtain the speech data to be enhanced; and

[0081] The speech enhancement model trained above is used to process the speech data to be enhanced, and the speech enhancement result is obtained.

[0082] The functions and processing of the speech enhancement model in the above speech enhancement methods are similar to those involved in the above training methods, and will not be repeated here.

[0083] Example 2

[0084] This embodiment proposes a two-stage speech enhancement method combining masking and mapping, along with dynamic data synthesis, aiming to construct a model that effectively removes noise and handles reverberation, with low algorithmic complexity and a small number of parameters. The model employs an innovative two-stage structure, separating the noise reduction and reverberation removal processes to reduce task complexity. Furthermore, by introducing a lightweight neural network design and a dynamic training data synthesis method, the model's robustness is improved, enabling real-time operation on resource-constrained platforms.

[0085] Overall architecture introduction:

[0086] A two-stage speech enhancement method involving joint masking and mapping, comprising:

[0087] Part 1: Data Collection, Preprocessing, and Dynamic Dataset Synthesis;

[0088] Part Two: Building and Training the Two-Stage Model of Joint Masking and Mapping;

[0089] Part 3: Model Streaming Inference Transformation.

[0090] Part 1: Data Collection, Preprocessing, and Dynamic Dataset Synthesis

[0091] 1. Collect raw audio data in multiple languages ​​such as Mandarin, English, German, and French. Data sources include, but are not limited to, open-source datasets on the Internet, freely downloadable public audio resources, and legal audio data collected through self-built recording equipment.

[0092] 2. Preprocess the speech data to remove speech segments with a duration of less than 1 second; filter out audio with a sampling rate (fs) lower than 16,000 Hz and a quantization bit depth lower than 16 bits; use the Voice Activity Detection (VAD) algorithm to cut out independent speech segments and remove excessively long silences; classify the processed speech into different folders according to different speakers, and label the different speaker folders with numerical order from smallest to largest.

[0093] 3. Collect and process noisy audio. Noise sources include publicly available noise databases (such as AudioSet, FSD50K, and ESC-50) and specific noise recorded according to actual application scenarios. Remove silent parts from the noisy audio using energy decision, and filter audio with a duration between 1 and 10 seconds.

[0094] 4. Collect impact response signals. The collected impact response signals are used in subsequent step 6 to generate speech data with reverberation effects, simulating different indoor scenarios. In actual deployment, using this reverberation data, the model can adapt to the speech enhancement needs of various indoor environments.

[0095] 5. Split the speech data of different languages ​​into validation and training sets in a 9:1 ratio according to the number of files; split all types of noise audio into training and validation sets in an 8:2 ratio.

[0096] 6. Dynamic dataset synthesis

[0097] (1) Set the length of the mixed audio to L, select a speaker in numerical order, randomly select voice audio in the folder, and insert a silent segment between the two audios to simulate a pause in speech.

[0098] (2) The length of the silent section is 1 to 5 seconds, and the silent part does not exceed 80% of the total length L.

[0099] (3) After obtaining the speech that meets the length, adjust the speech rate to 0.8 to 1.2 times the original speech rate with a 50% probability, and then trim the audio length to L.

[0100] (4) Reverberation is added to the clean speech s with a 40% probability. There are two types of reverberation: full reverberation and early reverberation. In speech enhancement tasks, noisy speech input with reverberation does not result in completely reverberation-free speech, but rather speech with shortened reverberation time. Since early reverberation is beneficial, the early reverberated speech is used as the final output target. The two-stage model proposed in this embodiment requires the fully reverberated clean speech s1 as the first-stage optimization target, and the early reverberated clean speech s2 as the second-stage optimization target.

[0101]

[0102] Where s(t) is the speech sound, and r full (t) is the impulse response audio, r early (t) represents the early reverberation, obtained by truncating r full (t) is obtained from the first 30 to 80 ms.

[0103] (5) Adjust the speech amplitude to make its volume -25dB. The adjustment coefficient scale is calculated as follows:

[0104]

[0105] Among them, T dbfs It is the target volume level, s MAX It is the maximum value that can represent the amplitude of speech, s MAX =2 bits-1 Bits represents the quantization bit depth, commonly ranging from 8 to 16 to 32. In practice, audio is usually normalized first, so s MAX=1 is the default value. P{s(t)} is the average power of the signal.

[0106] (6) Randomly select a noisy audio file. If the audio length is greater than L, trim the audio; if it is less than L, repeat the audio multiple times until the length is L. Then, uniformly generate N signal-to-noise ratio (SNR) values ​​(N is a positive integer) within the range of -5dB to 15dB. For example, if N = 10, the generated SNR values ​​include -5dB, 0dB, 5dB, 10dB, 15dB, etc. This method ensures that the model can be effectively trained under various different SNR conditions, thereby improving its adaptability to complex noise environments. Adjust the amplitude of the corresponding noise according to the selected SNR value. Specifically, the noise amplitude is adjusted using the following formula.

[0107]

[0108] Where, p s This represents the effective power of the signal.

[0109] This formula effectively mixes speech and noise signals, ensuring the mixed signal-to-noise ratio (SNR) reaches the desired value. The adjusted noise signal is then mixed with the speech signal to generate noisy speech samples. These noisy samples serve as the training dataset for the model, enabling it to learn denoising capabilities under different SNR conditions, thereby improving its performance in practical applications.

[0110] (7) Adjust the dBFS of the noise frequency relative to full scale. In real-world scenarios, changes in the distance between the sound source and the microphone will cause changes in the signal audio. Therefore, it is necessary to adjust the dBFS value of the audio to simulate different distances for sound pickup. Statistical data shows that the dBFS value range for digital audio signals is -40 to -10 dB. The adjustment factor required to obtain the target dBFS can be obtained using formula (3). At the same time, the target speech also needs to be multiplied by the same adjustment factor.

[0111] (8) Repeat steps (1)-(7) continuously to synthesize various training data. Each set of data includes: noisy reverb input, fully reverb clean speech, and early reverb clean speech. The number of repetitions is set to iters. By default, iters is equal to the number of clean speech files in the training set to ensure that all speech is fully utilized in each round of training.

[0112] (9) The method of synthesizing the validation dataset is the same as that of the training set. However, in order to ensure that the same validation set is used for each validation, the audio of the validation set is generated in advance before training, while the audio of the training set is dynamically synthesized during training.

[0113] Part Two: Building and Training the Two-Stage Model of Joint Masking and Mapping

[0114] 1. Perform a Short-Time Fourier Transform (STFT) on each group of training audio. During the transform, the number of points per frame (n_fft) is 512, the frame shift number (hop_length) is 256, and the window function is a Hamming window. The transformed image is a complex spectrum. Then, the real and imaginary parts are extracted to form a dual-channel spectrum. The spectrum dimension is... This represents rounding down. For example, when the sampling rate fs = 16,000, the audio length is 10 seconds, L = 160,000, and the dimension of the input spectrogram is (2, 257, 624). For clarity, we use (C, F, T) to represent the dimension, representing the channel, frequency, and frame number, respectively. Finally, we obtain the input spectrogram, the fully reverberated clean speech spectrogram, and the early reverberated clean speech spectrogram (Y, S1, S2).

[0115] 2. Use an encoder to compress the spectral image and extract spectral features to remove redundancy. The encoder can be a stacked CNN, MLP, or other structure.

[0116] This embodiment designs a 3-layer causal gated CNN layer as the encoder, and each sub-module is as follows: Figure 4 As shown, the feature input from the upper layer goes through a CNN and a Batch Normalization (BN) layer to both the left and right branches. The left branch ends with a Sigmoid unit to obtain the gating factor, while the right branch outputs the features. The outputs from the left and right branches are multiplied together to obtain the gated feature output. To ensure causality, each CNN has a kernel size of (3,1) and a sliding size of (2,1), meaning that after each CNN layer, the frequency dimension is reduced to half the size of the previous layer, while the number of frames remains unchanged. The number of channels in each CNN layer is 4, 8, and 16, respectively.

[0117] 3. The enhancer captures the contextual relationships between features. The spectrogram is obtained from speech transformation, with frequencies ranging from low to high, forming a harmonic structure. Adjacent frequencies are closely related. If the current frequency is interfered with or lost by noise, adjacent frequency components can be used to help repair it, making the model more stable. Furthermore, speech has strong temporal characteristics, reflected in the spectrogram as the correlation between the current frame signal and historical frame signals; the closer the signals, the stronger the correlation. Therefore, this embodiment uses a Dual Path Recurrent Neural Network (DPRNN) to model the contextual relationships of feature frequencies and time. The enhancer stacks three DPRNN structures. Each DPRNN contains both bidirectional and unidirectional gated recurrent unit (GRU) structures. The bidirectional GRU has an input dimension of 32 and an output dimension of 36, while the unidirectional GRU has an input dimension of 36 and an output dimension of 72.

[0118] 4. The decoder is used to estimate the denoised mask or map to obtain the dual-channel spectrum. The decoder uses transposed convolutional layers to continuously recover the frequency dimension. The decoder substructure is as follows: Figure 5 As shown, the features are first processed by a separable convolution with a kernel size of (2,1), a sliding size of (1,1), and a grouping number equal to the number of input channels. This is followed by Batch Normalization (BN) and PreLU activation functions. Then, the features are upsampled by a transposed convolution with a kernel size of (3,1) and a sliding size of (2,1). The entire decoder stacks three identical substructures, upsampling the feature dimensions to restore them to (C,F,T), resulting in an imaginary / real dual-channel spectral mask or a direct mapping to an imaginary / real spectral map. The number of channels for the three decoders' convolutions and transposed convolutions are (16,8), (8,4), and (4,2), respectively. Finally, the output channels are 2 corresponding to the imaginary / real part spectra.

[0119] 5. Two-stage network combination and training, such as Figure 3 As shown, the network structure is the same in both stages. The input to the model in the first stage is a noisy, reverberant audio spectrogram. After encoding, decoding, and enhancement, the output is an imaginary / real part spectrogram mask, which is then multiplied by the input spectrogram to obtain a denoised spectrogram, resulting in a reverberant spectrogram. The training objective for dynamic synthesis is now the clean speech spectrum S1 with reverberation. The network is optimized using the loss function L, as follows:

[0120]

[0121] L = 0.7 × L mag +0.3×L ri ……(8)

[0122] After the first phase of training converges, the output is... As input to the second-stage network, after encoding, decoding, and enhancement, the imaginary / real part spectrum is obtained. At this point, the training target for dynamic synthesis is the clean speech spectrum S2 with early reverberation. The same loss function as in the first stage is used for optimization to obtain a model with déverberation effect.

[0123] The third part is the streaming derivation of the model.

[0124] Most speech enhancement algorithms require real-time processing. As shown above, during training, L = 160,000, and each input consists of 624 frames. However, real-time processing can only process one frame at a time. To achieve this, the trained network is modified. The initial state input of the enhancer's unidirectional GRU is the hidden state output of the previous frame's signal after passing through the GRU. The network has six unidirectional GRUs, so it needs to cache the six hidden states {h1, h2, ... h6} of the previous frame. After each frame passes through the network, a denoised and de-reverberated complex spectrum is obtained. The frequency domain data is then restored to the time domain using the Inverse Short Time Fourier Transform (ISTFT). Finally, the signals from different frames are concatenated using an overlap-addition method. This embodiment provides some processed audio examples as follows: Figure 6 and Figure 7 As shown.

[0125] Figure 6 The image shows the effect of speech enhancement in a street noise scene, with the left and right sides displaying the audio spectrograms before and after enhancement, respectively. Figure 6 As shown, in the original audio (right), human voices are severely mixed with street noise, and the noise energy even masks the characteristics of human voices, resulting in unclear human voice structure in the spectrum. After noise reduction processing (left), the noise is effectively filtered out, and the characteristics of human voices in the spectrum are clearly visible.

[0126] Figure 7 This demonstrates the speech enhancement effect in a scenario where outdoor noise is superimposed on indoor reverberation. The left and right sides correspond to the audio before and after enhancement. In the original audio (right), the reverberation effect is more significant, causing the human voice to become aliased and blurred. After decoupling processing (separating noise reduction and reverberation) (left), the speech clarity is significantly improved.

[0127] Advantages explained:

[0128] 1. This embodiment proposes a method for dynamically synthesizing data, which can synthesize training data online, reducing machine storage costs during training. At the same time, since the signal-to-noise ratio, noise type, impulse response type, and volume of each training data set are randomly selected, the dynamic synthesis method can perform more data augmentation, expand the data space, avoid model overfitting, and improve generalization ability.

[0129] 2. This embodiment also proposes a two-stage speech enhancement method that combines masking and mapping, decoupling noise reduction and reverberation reduction. Staged training allows each stage of the network to focus on either noise reduction or reverberation reduction, thereby reducing the number of model parameters and model complexity.

[0130] 3. This embodiment also proposes an encoding / decoding submodule with low computational complexity, which reduces the number of model parameters and the operating load.

[0131] This application also provides an electronic device, including: a processor, and a memory coupled to the processor, the memory being used to store a computer program; the processor being used to execute the computer program stored in the memory, so that the electronic device performs the method as described in any of the above embodiments.

[0132] Electronic devices can be computing devices such as desktop computers, laptops, handheld computers, and cloud servers. These electronic devices may include, but are not limited to, processors and memory.

[0133] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the device via various interfaces and lines.

[0134] The memory can be used to store the computer program, and the processor implements various functions of the electronic device by running or executing the computer program stored in the memory and calling the data stored in the memory.

[0135] The memory may primarily include a program storage area and a data storage area. The program storage area may store the operating system, applications required for at least one function, etc.; the data storage area may store data created based on the use of the mobile phone, etc. Furthermore, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disks, RAM, plug-in hard disks, smart media cards (SMC), secure digital cards (SD cards), flash cards, at least one disk storage device, flash memory device, or other volatile solid-state storage devices.

[0136] This application embodiment also provides a storage medium, which is a computer-readable storage medium. The computer program is stored in the computer-readable storage medium, and when executed by a processor, the computer program can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0137] This application also provides a computer program product, including: a computer program or instructions that, when the computer program or instructions are run on a computer, cause the computer to perform any of the above possible implementation methods.

[0138] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A training method for a speech enhancement model, characterized in that, The training method includes: Obtain training data, the training data including: first speech data, second speech data, and third speech data, wherein the first speech data contains noise and full reverberation, the second speech data contains full reverberation, and the third speech data contains early reverberation, the early reverberation being extracted from the full reverberation; and The speech enhancement model is trained using the training data. The speech enhancement model includes: a first-stage model for removing the noise and a second-stage model for retaining only the early reverberation. The first-stage model includes, in sequence: a first time-frequency domain transformation module for performing time-frequency domain transformation on training data to obtain a first spectrogram; a first encoder for encoding the first spectrogram to obtain a first frequency domain feature; a first decoder for decoding the first frequency domain feature to obtain a mask; and a first dot product module for obtaining a second spectrogram from the first spectrogram and the mask. The second-stage model includes, in sequence: a second encoder for encoding the second spectrogram to obtain a second frequency domain feature, a second decoder for decoding the second frequency domain feature to obtain a third spectrogram, and a second time-frequency conversion module for performing frequency-time conversion on the third spectrogram to obtain enhanced speech; The specific training data obtained includes: Obtain clean speech, noisy audio, and impulse response signals; Using the clean speech and the impulse response signal, a plurality of second speech data and the third speech data are randomly generated; and Using the second speech data and the noise audio, several sets of the first speech data are randomly generated; The first stage model further includes: a first enhancer disposed between the first encoder and the first decoder for enhancing the first frequency domain features by utilizing the contextual relationships between the captured features; the second stage model further includes: a second enhancer disposed between the second encoder and the second decoder for enhancing the second frequency domain features by utilizing the contextual relationships between the captured features. The first encoder and the second encoder adopt the same structure as follows: a first branch, a second branch, and a second dot product module. The first branch includes, in sequence, a first convolutional neural network layer, a first normalization layer, and a first activation layer. The second branch includes, in sequence, a second convolutional neural network layer and a second normalization layer. The first convolutional neural network layer and the second convolutional neural network layer are used as inputs, and the outputs of the first activation layer and the second normalization layer are used as inputs to the second dot product module.

2. The training method as described in claim 1, characterized in that, Obtaining training data also includes: adjusting the length, volume, and / or signal-to-noise ratio of the first speech data, the second speech data, and the third speech data.

3. The training method as described in claim 1, characterized in that, The first enhancer and the second enhancer employ a dual-path cyclic speech separation neural network.

4. The training method as described in claim 1, characterized in that, The first decoder and the second decoder adopt the same structure as the second decoder and are arranged in sequence as follows: a third convolutional neural network layer and several stacked layers with the same input and output directions. The stacked layers include: a third normalization layer, a second activation layer and an upsampling layer.

5. The training method as described in claim 1, characterized in that, The first-stage model and the second-stage model use the same loss function L as follows: in, This represents the output of each stage. This indicates the training objectives for each stage. This indicates the operation of finding the real part. This represents the operation of finding the imaginary part. Represents the L2 norm. This represents the L1 norm.

6. A speech enhancement method, characterized in that, The speech enhancement method includes: Obtain the speech data to be enhanced; and The speech enhancement model trained according to any one of claims 1-5 is used to process the speech data to be enhanced, and the speech enhancement result is obtained.

7. An electronic device, characterized in that, The electronic device includes: a processor, and a memory coupled to the processor. The memory is used to store computer programs; and The processor is configured to execute the computer program stored in the memory, such that the electronic device performs the training method as described in any one of claims 1-5, or performs the speech enhancement method as described in claim 6.

Images