A method and system for speech enhancement

Abstract

The present application relates to a speech enhancement method and system, belonging to the field of speech enhancement technology, the speech enhancement method includes constructing a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator and a decoder connected in sequence; obtaining audio data to be processed, and inputting the audio data into the speech enhancement network, extracting L time domain features of different dimensions from the audio data through the encoder; extracting the L time domain features respectively through the sound-noise separator, and mapping the extraction results to a preset feature set; fusing the highest dimensional time domain feature and each extraction result in the corresponding preset feature set through the decoder to obtain denoised enhanced speech and / or background sound. In an unstable noise environment, the speech enhancement network can quickly adapt to changes in noise characteristics, perform noise suppression and speech enhancement in real time, and the present application has the effect of improving the stability and performance of speech enhancement.

Description

Technical Field

[0001] The present invention relates to the technical field of speech enhancement, and in particular to a speech enhancement method and system. Background Art

[0002] Speech enhancement refers to extracting the purest original speech from noisy speech to improve speech quality and reduce the negative impact of noise. Speech enhancement has important applications in speech recognition, speech coding and other fields, and is the most advanced preprocessing module in speech interaction systems.

[0003] At present, the frequency domain analysis method is usually used for speech enhancement, that is, the fast Fourier transform (FFT) of the speech signal is performed to convert the time domain signal into the frequency domain signal, and the frequency domain signal is analyzed and processed, such as filtering the signal of a specific frequency band through a filter to improve the quality of the speech signal. However, the method of filtering the signal of a specific frequency band makes restrictive assumptions about the characteristics of the noise signal, and the performance will be significantly reduced when processing some non-stationary noise, resulting in the enhancement of noise residue and the aggravation of speech distortion. How to improve the performance of speech enhancement is currently an urgent problem to be solved. Summary of the invention

[0004] In order to improve the performance of speech enhancement, the present application provides a speech enhancement method and system.

[0005] In the first aspect, the present application provides a method for speech enhancement, which adopts the following technical solution:

[0006] A speech enhancement method, comprising:

[0007] Constructing a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator, and a decoder connected in sequence;

[0008] The audio data to be processed is obtained and input into a speech enhancement network. L time domain features of different dimensions are extracted from the audio data by an encoder. The L time domain features are extracted separately by a sound-noise separator, and the extraction results are mapped to a preset feature set. The highest-dimensional time domain feature and each extraction result in the corresponding preset feature set are fused by a decoder to obtain denoised and enhanced speech and / or background sound.

[0009] By adopting the above technical solution, the encoder first accurately extracts L time domain features from the audio data. The time domain features cover the multi-dimensional information of the audio. These time domain features are then separated and extracted by the sound-noise separator. The extracted features are mapped to a preset feature set. Finally, the decoder fuses the highest-dimensional time domain features with the corresponding extraction results in the set to obtain denoised enhanced speech and / or background sound. Effective processing of noisy speech is achieved through the encoder, sound-noise separator and decoder, so that in an unstable noise environment, the speech enhancement network can quickly adapt to changes in noise characteristics, perform noise suppression and speech enhancement in real time, thereby improving the stability and performance of speech enhancement.

[0010] Optionally, the encoder includes L first convolution modules connected in sequence, each of which includes a one-dimensional convolution layer, a normalization layer, an activation function layer, and a downsampling pooling layer;

[0011] The first convolution module is used to input audio data, the output of each first convolution module is used as the input of the next first convolution module connected thereto, each first convolution module outputs a time domain feature, and the number of channels of the one-dimensional convolution layer in each first convolution module increases successively from the input end of the encoder to the output end of the encoder, so as to obtain time domain features with increasing dimensions;

[0012] The output end of each of the first convolution modules is also connected to the sound and noise separator to output L time domain features to the sound and noise separator.

[0013] By adopting the above technical solution, by setting a one-dimensional convolution layer, a normalization layer, an activation function layer and a downsampling pooling layer in the encoder, preliminary feature extraction of the audio data is achieved, and the number of channels of each first convolution module increases successively from the input end to the output end to extract time domain features of different dimensions, so that the encoder can capture richer and more diverse feature information from the audio data.

[0014] Optionally, the sound-noise separator includes L separation modules, each of which includes a plurality of second convolution modules connected in sequence and at least one attention module connected to the output end of the last second convolution module; wherein each of the attention modules corresponds to a preset feature set; the preset feature set includes a speech feature set and / or a noise feature set;

[0015] The input end of the first second convolution module in each separation module is correspondingly connected to the output end of the first convolution module to obtain time domain features, and the second convolution module is used to extract the time domain features to obtain extraction results;

[0016] Each of the attention modules maps the extraction results to a corresponding preset feature set;

[0017] Each of the second convolution modules includes a one-dimensional convolution layer, a normalization layer and an activation function.

[0018] By adopting the above technical solution, multiple second convolution modules respectively process the input time domain features to obtain extraction results, and then through the attention module, the extraction results can be accurately mapped to the preset speech feature set or noise feature set, thereby achieving the effect of separating speech and noise in a non-stable noise environment.

[0019] Optionally, the attention module includes a self-attention layer or a multi-head self-attention layer.

[0020] By adopting the above technical solution, the self-attention layer is simpler than the multi-head self-attention layer and can be flexibly selected according to actual computing resources.

[0021] Optionally, the decoder includes a speech decoder and / or a noise decoder, and the speech decoder and the noise decoder each include L third convolution modules connected in sequence and an output module connected to the third convolution module at the end; each of the third convolution modules is connected to an attention module, and the first third convolution module is also connected to an encoder to obtain the highest dimensional time domain features;

[0022] The first third convolution module fuses and adds the time domain feature with the highest dimension and the extraction result in the corresponding preset feature set to obtain a fused feature, and outputs the fused feature to the next third convolution module connected thereto;

[0023] Each of the remaining third convolution modules fuses and adds the fused features output by the previous third convolution module and the extraction results in the corresponding preset feature set, and outputs the fused features output by the tail third convolution module to the output module, so that the output module outputs denoised and enhanced speech and / or background sound.

[0024] By adopting the above technical solution, the time domain features with the highest dimension are first obtained from the encoder, and the attention module is used to extract more detailed feature information. The first third convolution module fuses and adds the time domain features with the highest dimension and the extraction results in the corresponding preset feature set to obtain representative fused features. Each third convolution module is then used to continue to fuse the fused features output by the previous module with the extraction results in the preset feature set to further enhance the feature expression. Finally, the output module is used to convert these fused features into clear denoising and enhanced speech and / or background sound, thereby achieving the decoder's restoration effect on speech and noise.

[0025] Optionally, the third convolution module includes a channel fusion layer, an upsampling layer, a one-dimensional convolution layer, a normalization layer and an activation function layer connected in sequence; the number of channels of the one-dimensional convolution layer in each of the third convolution modules decreases from the input end to the output end of the decoder; the output module includes a one-dimensional convolution layer, a normalization layer and an activation function layer, and the number of channels of the one-dimensional convolution layer in the output module is 1.

[0026] By adopting the above technical solution, the channel fusion layer, upsampling layer, one-dimensional convolution layer, normalization layer and activation function layer in the third convolution module are connected in sequence, so that feature extraction and dimensionality transformation can be carried out normally. The number of channels of the one-dimensional convolution layer in each third convolution module decreases from the input end to the output end of the decoder, which helps to gradually reduce the feature dimension and adapt to the needs of the output module. Finally, the output module is used to output the audio data with the same dimension as the input.

[0027] Optionally, the constructing of the speech enhancement network includes a training step of training a preset neural network model to obtain the speech enhancement network;

[0028] The training steps include:

[0029] Acquire a training data set; the training data set includes a noisy original speech signal, an ideal noise-free speech signal and an ideal noise;

[0030] The noisy original speech signal is input into the encoder, and L time domain features of different dimensions are extracted from the audio data through the encoder;

[0031] The L time domain features are extracted respectively by a sound-noise separator, and the extraction results are mapped to a preset feature set; wherein the preset feature set also includes a noise feature set;

[0032] The first decoder performs a fusion process on each extraction result in the time domain feature with the highest dimension and the speech feature set to obtain actual denoised speech;

[0033] The second decoder performs a fusion process on each extraction result in the time domain feature with the highest dimension and the noise feature set to obtain actual noise;

[0034] According to the comparison results of the actual denoised speech and the ideal noise-free speech signal and the comparison results of the actual noise and the ideal noise, the model parameters of the preset neural network model are iteratively updated to obtain the speech enhancement network.

[0035] By adopting the above technical solution, a noisy original speech signal, an ideal noise-free speech signal and an ideal noise are obtained from a training data set, and the noisy original speech signal is input into an encoder. After being processed by the encoder, L time domain features of different dimensions are successfully extracted from the audio data. The audio-noise separator further processes these features and maps them to a preset feature set. The first decoder is then used to fuse the time domain features with the highest dimension with the extraction results in the speech feature set to generate actual denoised speech. At the same time, the second decoder is used to generate actual noise. The actual denoised speech and the actual noise are compared with the corresponding ideal signals to update the parameters of the neural network model. After multiple iterative updates, a speech enhancement network with better performance can be obtained.

[0036] In a second aspect, the present application provides a speech enhancement system, which adopts the following technical solution:

[0037] A speech enhancement system, comprising:

[0038] A network construction unit, used to construct a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator and a decoder connected in sequence;

[0039] The speech enhancement unit is used to input the audio data to be processed into the speech enhancement network, extract L time domain features of different dimensions from the audio data through the encoder; extract the L time domain features respectively through the sound-noise separator, and map the extraction results to the preset feature set; and fuse the highest-dimensional time domain feature and each extraction result in the corresponding preset feature set through the decoder to obtain denoised and enhanced speech and / or background sound.

[0040] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0041] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to any one of the above methods.

[0042] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0043] A computer-readable storage medium includes a computer program that can be loaded by a processor and executed in any of the above methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] Figure 1 It is a flow chart of a speech enhancement method according to one embodiment of the present application.

[0045] Figure 2 It is a structural diagram of an encoder according to one embodiment of the present application.

[0046] Figure 3 It is a structural diagram of a separation module in one embodiment of the present application.

[0047] Figure 4 It is a structural diagram of a speech decoder in one embodiment of the present application.

[0048] Figure 5 This is a flow chart of a training method for a speech enhancement network according to one embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0050] The present application embodiment discloses a method for speech enhancement. Figure 1 , a speech enhancement method, comprising:

[0051] Step S101: constructing a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator, and a decoder connected in sequence;

[0052] Step S102: Obtain the audio data to be processed, and input the audio data into the speech enhancement network, extract L time domain features of different dimensions from the audio data through the encoder; extract the L time domain features separately through the sound-noise separator, and map the extraction results to the preset feature set; fuse the highest-dimensional time domain feature and each extraction result in the corresponding preset feature set through the decoder to obtain denoised and enhanced speech and / or background sound.

[0053] It should be understood that the speech enhancement network is a time domain analysis network and does not involve the frequency domain analysis process. There is no need for Fourier transform. On the one hand, it reduces the amount of data to be processed. On the other hand, time domain analysis can better adapt to non-stable background sounds to better enhance speech.

[0054] In the above implementation, the encoder first accurately extracts L time domain features from the audio data, and the time domain features cover the multi-dimensional information of the audio. These time domain features are then separated and extracted by a sound-noise separator, and the extracted features are mapped to a preset feature set. Finally, the decoder fuses the highest-dimensional time domain features with the corresponding extraction results in the set to obtain denoised enhanced speech and / or background sound. Effective processing of noisy speech is achieved through the encoder, sound-noise separator and decoder, so that in an unstable noise environment, the speech enhancement network can quickly adapt to changes in noise characteristics, perform noise suppression and speech enhancement in real time, thereby improving the stability and performance of speech enhancement.

[0055] Reference Figure 2 As an implementation of the encoder, the encoder includes L first convolution modules connected in sequence, each of which includes a one-dimensional convolution layer, a normalization layer, an activation function layer, and a downsampling pooling layer connected in sequence;

[0056] The first convolution module is used to input audio data, and the output of each first convolution module is used as the input of the next first convolution module connected thereto, and each first convolution module outputs a time domain feature, and the number of channels of the one-dimensional convolution layer in each first convolution module increases successively from the input end of the encoder to the output end of the encoder, so as to obtain time domain features with increasing dimensions;

[0057] The output end of each of the first convolution modules is also connected to the sound and noise separator to output L time domain features to the sound and noise separator.

[0058] It should be understood that each first convolution module outputs a corresponding time domain feature.

[0059] Among them, the number of channels of the one-dimensional convolution layer in each first convolution module increases from the input end of the encoder to the output end of the encoder, that is, the number of channels of the one-dimensional convolution layer from the input end of the encoder to the encoder is {𝑁, 2 ×𝑁, ⋯, 𝐿× 𝑁} respectively. In this embodiment, the convolution kernel of the first convolution module is 𝑘 𝑒𝑛𝑐 , the step size is 𝑠 𝑒𝑛𝑐 , downsampling spacing d 𝑒𝑛𝑐 And the number of the first convolution modules, the size of the convolution kernel, the step size, and the downsampling interval can be set according to the actual situation. For example, it can be set to 𝐿 = 6, 𝑁 = 64, 𝑘 𝑒𝑛𝑐 = 21, 𝑠 𝑒𝑛𝑐 = 1, d 𝑒𝑛𝑐 =2.

[0060] Specifically, combined Figure 2 , the first convolution module is set to L, namely CONV a1、CONV a2 ,……,CONV aL ; Each first convolution module includes a one-dimensional convolution layer, a normalization layer, an activation function layer, and a downsampling pooling layer connected in sequence, that is, a one-dimensional convolution layer {𝐶𝑜𝑛𝑣1, 𝐶𝑜𝑛𝑣2, ⋯ , 𝐶𝑜𝑛𝑣 is provided in the encoder. 𝐿}, normalization layer {𝐵𝑁1,𝐵𝑁2, ⋯ , 𝐵𝑁 𝐿}, {𝑅𝑒𝐿𝑈1, 𝑅𝑒𝐿𝑈2, ⋯ , 𝑅𝑒𝐿𝑈 𝐿} and down-sampling pooling layers {𝐷𝑆1, 𝐷𝑆2, ⋯, 𝐷𝑆 𝐿}, the audio data is input from the one-dimensional convolution layer of the first convolution module, passes through the normalization layer and the activation function layer in sequence, and then outputs the time domain features 𝑒𝑛𝑐 by the downsampling pooling layer v 1 , the number of channels of the one-dimensional convolution layer of the first convolution module is N, then the output time domain feature 𝑒𝑛𝑐 v 1 The dimension is N; then the time domain feature 𝑒𝑛𝑐 v 1 Input to the second first convolution module CONV a2 , the same reason as the second first convolution module downsamples the pooling layer to output the time domain feature 𝑒𝑛𝑐 v 2 , the number of channels of the one-dimensional convolution layer of the second first convolution module is 2N, so the dimension of the output time domain feature is 2N. By analogy, the L time domain features output by the encoder are {𝑒𝑛𝑐 v 1 , 𝑒𝑛𝑐 v 2 , ⋯ ,𝑒𝑛𝑐 v L}, and the corresponding dimensions are {𝑁, 2 × 𝑁, ⋯ , 𝐿× 𝑁}, thereby achieving the effect of the encoder outputting time domain features of different dimensions.

[0061] It should also be understood that the more channels there are, the richer the time domain features that can be extracted, but at the same time the encoder is prone to overfitting problems. By combining time domain features of different dimensions, richer features can be obtained for speech enhancement while retaining the time domain features of a smaller number of channels, reducing the possibility of overfitting.

[0062] In the above implementation, by setting a one-dimensional convolution layer, a normalization layer, an activation function layer and a downsampling pooling layer in the encoder, preliminary feature extraction of the audio data is achieved, and the number of channels of each first convolution module increases successively from the input end to the output end to extract time domain features of different dimensions, so that the encoder can capture richer and more diverse feature information from the audio data.

[0063] Reference Figure 3 As an implementation of the sound-noise separator, the sound-noise separator includes L separation modules, each of which includes a plurality of second convolution modules connected in sequence and at least one attention module connected to an output end of the last second convolution module;

[0064] The input end of the first second convolution module in each separation module is correspondingly connected to the output end of the first convolution module to obtain time domain features, and the second convolution module is used to extract the time domain features to obtain extraction results;

[0065] Each attention module maps the extraction results to the corresponding preset feature set;

[0066] Wherein, each attention module corresponds to a preset feature set; the preset feature set includes a speech feature set and / or a noise feature set;

[0067] Among them, the attention module includes a self-attention layer or a multi-head self-attention layer.

[0068] Among them, each second convolution module includes a one-dimensional convolution layer, a normalization layer and an activation function, and the second convolution module does not have a pooling layer to keep the dimensions of the input and output consistent.

[0069] It should be understood that the number of separation modules should be consistent with the number of first convolution modules in the encoder, and each separation module corresponds to a first convolution module.

[0070] Specifically, combined Figure 3 , the separation module is 𝑆𝐸𝑃 𝑖 , then the L separation modules are {𝑆𝐸𝑃1, 𝑆𝐸𝑃2,⋯ , 𝑆𝐸𝑃 𝐿}, each separation module is equipped with M second convolution modules and two attention modules, and the M second convolution modules are CONV b1 、CONV b2 ,……,CONV bM In this embodiment, the attention module is provided with two speech attention modules 𝑆𝐴 𝑖 s and the noisy attention module 𝑆𝐴 𝑖 n, then the audio-noise separator is equipped with L speech attention modules, namely {𝑆𝐴 1 s , 𝑆𝐴 2 s , ⋯ , 𝑆𝐴 n s}, the preset feature set corresponding to these L speech attention modules is the speech feature set, and the extraction result output by each speech attention module is an element 𝑠𝑒𝑝 in the speech feature set 𝑠 , then the speech feature set obtained by mapping all speech attention modules in the audio-noise separator is {𝑠𝑒𝑝 1 s , 𝑠𝑒𝑝 2 s , ⋯ , 𝑠𝑒𝑝 L s}. Similarly, the extraction result output by each noise attention module is an element 𝑠𝑒𝑝 in the noise feature set 𝑛 , then the noise feature set obtained by mapping all the noise attention modules in the audio-noise separator is {𝑠𝑒𝑝 1 n , 𝑠𝑒𝑝 2 n , ⋯ ,𝑠𝑒𝑝 L n}.

[0071] It should be noted that the number and type of attention modules can be set according to actual needs and are not limited here. For example, multiple voice attention modules can be provided to separate different human voices, and the type of attention module can also be a music attention module to separate background music.

[0072] In the above implementation, multiple second convolution modules respectively process the input time domain features to obtain extraction results, and then through the attention module, the extraction results can be accurately mapped to a preset speech feature set or noise feature set, thereby achieving the effect of separating speech and noise in a non-stable noise environment.

[0073] Reference Figure 4 As an implementation of the decoder, the decoder includes a speech decoder and / or a noise decoder, and the speech decoder and the noise decoder each include L third convolution modules connected in sequence and an output module connected to the third convolution module at the end; each of the third convolution modules is connected to an attention module, and the first third convolution module is also connected to an encoder to obtain the highest dimensional time domain features;

[0074] The first third convolution module fuses and adds the time domain feature with the highest dimension and the extraction result in the corresponding preset feature set to obtain a fused feature, and outputs the fused feature to the next third convolution module connected thereto;

[0075] Each of the remaining third convolution modules fuses and adds the fused features output by the previous third convolution module and the extraction results in the corresponding preset feature set, and outputs the fused features output by the tail third convolution module to the output module, so that the output module outputs denoised and enhanced speech and / or background sound.

[0076] Specifically, combined Figure 4 , the third convolution module is set to L, namely CONV c1 、CONV c2 ,……,CONV cL , the input of the first third convolution module is the time domain feature 𝑒𝑛𝑐 output by the Lth first convolution module v L The extracted feature 𝑠𝑒𝑝 in the speech feature set obtained by mapping with the Lth speech attention module L s Additive fusion, the input of the other 𝑖∈ [2, 𝐿]th third convolutional module is the output of the 𝑖− 1th layer third convolutional module and the extracted features 𝑠𝑒𝑝 in the speech feature set of the 𝐿+ 1 − 𝑖th layer s L+1-i Obtained by fusion and addition.

[0077] In the above implementation, the time domain features with the highest dimension are first obtained from the encoder, and the attention module is used to extract more detailed feature information. The first third convolution module fuses and adds the time domain features with the highest dimension and the extraction results in the corresponding preset feature set to obtain representative fused features. Each third convolution module is then used to continue to fuse the fused features output by the previous module with the extraction results in the preset feature set to further enhance the feature expression. Finally, the output module is used to convert these fused features into clear denoising and enhanced speech and / or background sound, thereby achieving the decoder's restoration effect on speech and noise.

[0078] As an implementation of the third convolution module, the third convolution module includes a channel fusion layer, an upsampling layer, a one-dimensional convolution layer, a normalization layer, and an activation function layer connected in sequence; the number of channels of the one-dimensional convolution layer in each of the third convolution modules decreases from the input end to the output end of the decoder; the output module includes a one-dimensional convolution layer, a normalization layer, and an activation function layer, and the number of channels of the one-dimensional convolution layer in the output module is 1.

[0079] Specifically, the number of channels of the one-dimensional convolution layer in the third convolution module is {𝐿× 𝑁, (𝐿− 1)× 𝑁, ⋯ ,𝑁}, that is, the decreasing method of the number of channels of the one-dimensional convolution layer in the third convolution module is opposite to the increasing method of the number of channels of the one-dimensional convolution layer in the first convolution module, and the convolution kernel is 𝑘 d𝑒𝑐 , the step size is 𝑠 d𝑒𝑐 , upsampling is doubled, the parameters can be customized, such as setting 𝐿 = 6, 𝑁 = 64, 𝑘 d𝑒𝑐 = 7, 𝑠 d𝑒𝑐 = 1.

[0080] Specifically, combined Figure 4 , taking the speech decoder as an example, the third convolution module is set to L, namely CONV c1 、CONV c2 ,……,CONV cL ; The voice decoder is provided with a channel fusion layer

[0081] {𝑈𝑆1, 𝑈𝑆2, ⋯ , 𝑈𝑆 𝐿}, one-dimensional convolutional layer {𝐶𝑜𝑛𝑣1, 𝐶𝑜𝑛𝑣2, ⋯ , 𝐶𝑜𝑛𝑣 𝐿}, normalization layer {𝐵𝑁1, 𝐵𝑁2, ⋯ , 𝐵𝑁 𝐿} and activation function {𝑅𝑒𝐿𝑈1, 𝑅𝑒𝐿𝑈2, ⋯ , 𝑅𝑒𝐿𝑈 𝐿}, the highest dimension time domain feature 𝑒𝑛𝑐 v L After adding and fusing the L-th extraction result in the speech feature set, it is input from the channel fusion layer of the first third convolution module, and passes through the one-dimensional convolution layer, normalization layer and activation function layer in sequence. The output result is then added and fused with the L-1-th extraction result in the speech feature set and input into the second third convolution module, and so on, until the tail third convolution module CONV is obtained. cL The output of the third convolution module CONV cL The output of is input to the output module to obtain denoised and enhanced speech.

[0082] In the above implementation, the channel fusion layer, upsampling layer, one-dimensional convolution layer, normalization layer and activation function layer in the third convolution module are connected in sequence, so that feature extraction and dimensionality transformation can be carried out normally. The number of channels of the one-dimensional convolution layer in each third convolution module decreases from the input end to the output end of the decoder, which helps to gradually reduce the feature dimension and adapt to the needs of the output module. Finally, the output module is used to output the audio data with the same dimension as the input.

[0083] Reference Figure 5 As an implementation of step S101, constructing a speech enhancement network includes training a preset neural network model to obtain a training step of the speech enhancement network; the training step includes:

[0084] Step S1011: Acquire a training data set; the training data set includes a noisy original speech signal, an ideal noise-free speech signal and an ideal noise;

[0085] Step S1012: inputting the noisy original speech signal into an encoder, and extracting L time domain features of different dimensions from the audio data through the encoder;

[0086] Step S1013: extracting L time domain features respectively through a sound-noise separator, and mapping the extraction results to a preset feature set; wherein the preset feature set also includes a noise feature set;

[0087] Step S1014: fusing the time domain feature with the highest dimension and each extraction result in the speech feature set through the first decoder to obtain actual denoised speech;

[0088] Step S1015: fusing each extraction result in the time domain feature with the highest dimension and the noise feature set through a second decoder to obtain actual noise;

[0089] It should be noted that step S1014 and step S1015 can be executed in parallel, and the number of the first decoder and the second decoder can be set according to actual conditions.

[0090] Step S1016: According to the comparison results between the actual denoised speech and the ideal noise-free speech signal and the comparison results between the actual noise and the ideal noise, the model parameters of the preset neural network model are iteratively updated to obtain the speech enhancement network.

[0091] In the above implementation, a noisy original speech signal, an ideal noise-free speech signal and an ideal noise are obtained from a training data set, and the noisy original speech signal is input into an encoder. After being processed by the encoder, L time domain features of different dimensions are successfully extracted from the audio data. The audio-noise separator further processes these features and maps them to a preset feature set. The first decoder is then used to fuse the highest-dimensional time domain features with the extraction results in the speech feature set to generate actual denoised speech. At the same time, the second decoder is used to generate actual noise. The actual denoised speech and the actual noise are compared with the corresponding ideal signals to update the parameters of the neural network model. After multiple iterative updates, a speech enhancement network with better performance can be obtained.

[0092] The present application embodiment discloses a speech enhancement system, a speech enhancement system comprising:

[0093] A network construction unit, used to construct a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator and a decoder connected in sequence;

[0094] The speech enhancement unit is used to input the audio data to be processed into the speech enhancement network, extract L time domain features of different dimensions from the audio data through the encoder; extract the L time domain features respectively through the sound-noise separator, and map the extraction results to the preset feature set; and fuse the highest-dimensional time domain feature and each extraction result in the corresponding preset feature set through the decoder to obtain denoised and enhanced speech and / or background sound.

[0095] The speech enhancement system provided in the present application can implement the above-mentioned speech enhancement method, and the specific working process of the speech enhancement system can refer to the corresponding process in the above-mentioned method embodiment.

[0096] It should be noted that in the above embodiments, the description of each embodiment has different emphases, and for parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0097] Based on the same technical concept, the present invention also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to any of the above methods.

[0098] The present invention also discloses a computer-readable storage medium, which includes a computer program that can be loaded by a processor and executed as in any of the above methods.

[0099] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0100] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0101] The above are all preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any feature disclosed in this specification (including the abstract and drawings), unless otherwise stated, can be replaced by other equivalent or alternative features with similar purposes. That is, unless otherwise stated, each feature is only an example of a series of equivalent or similar features.

Claims

1. A speech enhancement method, characterized in that: include: Building a speech enhancement network; The speech enhancement network includes an encoder, a sound-noise separator and a decoder connected in sequence; Obtain the audio data to be processed, and input the audio data into the speech enhancement network, extract L time domain features of different dimensions from the audio data through the encoder; extract the L time domain features respectively through the sound-noise separator, and map the extraction results to the preset feature set; The decoder performs a fusion process on the highest-dimensional time domain feature and each extraction result in the corresponding preset feature set to obtain denoised enhanced speech and / or background sound; the preset feature set includes a speech feature set and / or a noise feature set; The encoder comprises L first convolution modules connected in sequence; an output end of each of the first convolution modules is connected to a sound-noise separator to output L time domain features to the sound-noise separator; The decoder includes a speech decoder and / or a noise decoder, and the speech decoder and the noise decoder each include L third convolution modules connected in sequence and an output module connected to the third convolution module at the end; each of the third convolution modules is connected to an attention module, and the first third convolution module is also connected to an encoder to obtain the highest-dimensional time domain features; The first third convolution module fuses and adds the time domain feature with the highest dimension and the extraction result in the corresponding preset feature set to obtain a fused feature, and outputs the fused feature to the next third convolution module connected thereto; Each of the remaining third convolution modules fuses and adds the fused features output by the previous third convolution module and the extraction results in the corresponding preset feature set, and outputs the fused features output by the tail third convolution module to the output module, so that the output module outputs denoised and enhanced speech and / or background sound.

2. The method according to claim 1, characterized in that: Each first convolution module includes a one-dimensional convolution layer, a normalization layer, an activation function layer, and a downsampling pooling layer; The first first convolution module is used to input audio data, and the output of each first convolution module is used as the input of the next first convolution module connected to it. Each first convolution module outputs a time domain feature, and the number of channels of the one-dimensional convolution layer in each first convolution module increases successively from the input end of the encoder to the output end of the encoder, thereby obtaining time domain features with increasing dimensions.

3. The method according to claim 2, characterized in that: The sound-noise separator includes L separation modules, each of which includes a plurality of second convolution modules connected in sequence and at least one attention module connected to the output end of the last second convolution module; wherein each of the attention modules corresponds to a preset feature set; the input end of the first second convolution module in each separation module corresponds to the output end of the first convolution module to obtain time domain features, and the second convolution module is used to extract the time domain features to obtain extraction results; Each of the attention modules maps the extraction results to a corresponding preset feature set; Each of the second convolution modules includes a one-dimensional convolution layer, a normalization layer and an activation function.

4. The method according to claim 3, characterized in that The attention module includes a self-attention layer or a multi-head self-attention layer.

5. The method according to claim 1, characterized in that The third convolution module includes a channel fusion layer, an upsampling layer, a one-dimensional convolution layer, a normalization layer and an activation function layer connected in sequence; the number of channels of the one-dimensional convolution layer in each of the third convolution modules decreases from the input end to the output end of the decoder; the output module includes a one-dimensional convolution layer, a normalization layer and an activation function layer, and the number of channels of the one-dimensional convolution layer in the output module is 1.

6. The method according to claim 4, characterized in that The constructing of the speech enhancement network includes a training step of training a preset neural network model to obtain the speech enhancement network; The training steps include: Acquire a training data set; the training data set includes a noisy original speech signal, an ideal noise-free speech signal and an ideal noise; The noisy original speech signal is input into the encoder, and L time domain features of different dimensions are extracted from the audio data through the encoder; The L time domain features are extracted respectively by a sound-noise separator, and the extraction results are mapped to a preset feature set; wherein the preset feature set also includes a noise feature set; The first decoder performs a fusion process on each extraction result in the time domain feature with the highest dimension and the speech feature set to obtain actual denoised speech; The second decoder performs a fusion process on each extraction result in the time domain feature with the highest dimension and the noise feature set to obtain actual noise; According to the comparison results of the actual denoised speech and the ideal noise-free speech signal and the comparison results of the actual noise and the ideal noise, the model parameters of the preset neural network model are iteratively updated to obtain the speech enhancement network.

7. A speech enhancement system, characterized in that: A method for performing a speech enhancement method according to any one of claims 1 to 6, comprising: A network construction unit, used to construct a speech enhancement network; the speech enhancement network includes an encoder, a sound-noise separator and a decoder connected in sequence; The speech enhancement unit is used to input the audio data to be processed into the speech enhancement network, extract L time domain features of different dimensions from the audio data through the encoder; extract the L time domain features respectively through the sound-noise separator, and map the extraction results to the preset feature set; and fuse the highest-dimensional time domain feature and each extraction result in the corresponding preset feature set through the decoder to obtain denoised and enhanced speech and / or background sound.

8. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program according to any one of the methods of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The method comprises storing a computer program which can be loaded by a processor and execute the method according to any one of claims 1 to 6.

Images