Voice signal noise reduction method and device based on U-net structure

Abstract

The invention provides a voice signal noise reduction method and device based on a U-net structure, and the method comprises the steps: carrying out the short-time Fourier transform of a to-be-processed voice signal, and obtaining the amplitude spectrums corresponding to a plurality of short-time frames; for each short-time frame, carrying out encoding processing on non-direct-current information in the amplitude spectrum corresponding to the short-time frame through a multi-layer encoder to obtain an encoding result; processing the coding result through each layer of time domain convolutional network to obtain a feature extraction result; the feature extraction result is decoded through a multi-layer decoder, and a denoised target voice signal is obtained; according to the method, a U-net structure is adopted, processing is carried out based on the amplitude spectrum, and through the multi-layer encoder, the time domain convolutional network and the multi-layer decoder which are suitable for amplitude spectrum operation, the calculation amount can be reduced, the calculation efficiency can be improved, meanwhile, the occupation of a memory is reduced, and the noise reduction effect on the to-be-processed voice signal is improved.

Description

Technical Field

[0001] The present invention relates to the field of speech processing technology, and in particular to a speech signal denoising method and device based on a U-net structure. Background Art

[0002] In the collected voice signal, affected by the user's environment, there are usually various background noises in the voice signal. In order to make the heard voice clearer, the collected voice signal needs to be denoised. In the related technology, the voice signal is usually denoised based on GRU (Gated Recurrent Unit). However, as a recursive structure, GRU usually requires more parameters to model the information in the signal, which increases the amount of calculation, reduces the calculation efficiency, and requires more memory, which has a certain impact on the noise reduction effect. Summary of the invention

[0003] The purpose of the present invention is to provide a speech signal denoising method and device based on a U-net structure, so as to reduce the amount of calculation, improve the calculation efficiency, reduce the memory usage, and improve the denoising effect on the processed speech signal.

[0004] The present invention provides a speech signal denoising method based on a U-net structure, the method comprising: obtaining a speech signal to be processed; performing short-time Fourier transform on the speech signal to be processed to obtain a plurality of short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each short-time frame; for each short-time frame, encoding non-DC information in the amplitude spectrum corresponding to the short-time frame by a multi-layer encoder to obtain an encoding result; sending the encoding result to a multi-layer time domain convolutional network to process the encoding result by each layer of the time domain convolutional network to obtain a feature extraction result; decoding the feature extraction result by a multi-layer decoder to obtain a target speech signal after the speech signal to be processed is denoised; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

[0005] Furthermore, each layer of the time domain convolutional network includes: a first convolutional layer, a second convolutional layer, an information output module and a gated signal generation module that are parallel to each other; the information output module includes: a first activation function layer, a first normalization layer and a dilated convolutional layer that are connected in sequence; the gated signal generation module includes a first activation function layer, a first normalization layer, a dilated convolutional layer and a second activation function layer that are connected in sequence; the first activation function layer in the information output module and the first activation function layer in the gated signal generation module are also respectively connected to the first convolutional layer;

[0006] For each layer of the time domain convolutional network, the first convolutional layer is used to perform convolution processing on the input signal received by the time domain convolutional network of this layer to obtain a first convolution result, and send the first convolution result to each first activation function layer respectively; each first activation function layer is used to perform nonlinear operation on the first convolution result respectively to obtain a first operation result, and send the first operation result to each correspondingly connected first normalization layer respectively; each first normalization layer is used to perform normalization processing on the first operation result respectively to obtain a first normalization result, and send the first normalization result to each correspondingly connected expansion convolutional layer respectively; each expansion convolutional layer is used to expand the receptive field of the first normalization result respectively to obtain a first output result; the second activation function layer is also used to receive the first output result sent by the connected expansion convolutional layer, map the first output result, and output a gating signal; the second convolutional layer is used to receive the processing result after multiplying the first output result output by the information output module and the gating signal, and convolve the processing result to obtain the output result of the time domain convolutional network of this layer.

[0007] Furthermore, in different layers of the time domain convolutional network, the expansion rates of the expansion convolutional layers are different; the last layer of the time domain convolutional network also includes a fully connected layer; the fully connected layer is used to perform fully connected processing on the output result of the second convolutional layer of the last layer of the time domain convolutional network to obtain the final output result of the last layer of the time domain convolutional network.

[0008] Furthermore, each layer of the encoder includes: a deep convolution layer, a point convolution layer, a second normalization layer and a first non-linear activation layer connected in sequence; for each layer of the encoder, the deep convolution layer is used to perform a convolution operation on each input channel of the encoder of this layer respectively to obtain a deep convolution result; the point convolution layer is used to perform a point convolution operation on each deep convolution result to obtain a point convolution result; the second normalization layer is used to normalize the point convolution result to obtain a second normalized result; the non-linear activation layer is used to perform a nonlinear operation on the second normalized result to obtain an output result of the encoder of this layer.

[0009] Furthermore, each layer of the decoder includes: a second nonlinear activation layer, a third normalization layer, a point deconvolution layer and a depth deconvolution layer connected in sequence; for each layer of the decoder, the second nonlinear activation layer is used to perform an inverse linear operation on the input information of the decoder of this layer to obtain a second operation result; the third normalization layer is used to perform an inverse normalization process on the second operation result to obtain an inverse normalized result; the point deconvolution layer is used to perform an inverse dot product operation on the inverse normalization result to obtain a point deconvolution result; the depth deconvolution layer is used to perform a deconvolution operation on the point deconvolution result to obtain an output result of the decoder of this layer.

[0010] Furthermore, the step of decoding the feature extraction result by a multi-layer decoder to obtain the target speech signal after noise reduction for the speech signal to be processed includes: decoding the feature extraction result by a multi-layer decoder to obtain an amplitude mask; masking the amplitude spectrum based on the amplitude mask to obtain a masking result; and performing an inverse short-time Fourier transform on the masking result to obtain the target speech signal after noise reduction corresponding to the speech signal to be processed.

[0011] Furthermore, the first normalization layer is used to perform normalization using the following formula:

[0012]

[0013] Among them, ft represents the k-th frame signal in the first operation result received by the first normalization layer; ft represents the t-th frame signal in the first operation result received by the first normalization layer; E[ft≤k] represents taking the mean of the first frame signal to the k-th frame signal; var[ft≤k] represents taking the variance of the first frame signal to the k-th frame signal; ∈ is a preset value; γ and β are both trainable parameters that can be applied to each frame signal in the first operation result received by the first normalization layer.

[0014] Furthermore, the step of obtaining the voice signal to be processed includes: obtaining an original voice signal collected by a voice collection device; and preprocessing the original voice signal to obtain the voice signal to be processed.

[0015] Furthermore, the steps of performing short-time Fourier transform on the processed speech signal to obtain multiple short-time frames corresponding to the processed speech signal and the amplitude spectrum corresponding to each short-time frame include: performing short-time Fourier transform on the processed speech signal to segment the processed speech signal to obtain multiple short-time frames corresponding to the processed speech signal; performing signal decomposition on each short-time frame to obtain the amplitude spectrum corresponding to each short-time frame.

[0016] The present invention provides a speech signal denoising device based on a U-net structure, the device comprising: an acquisition module, used for acquiring a speech signal to be processed; a Fourier transform module, used for performing short-time Fourier transform on the speech signal to be processed, to obtain a plurality of short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each short-time frame; an encoding module, used for encoding non-DC information in the amplitude spectrum corresponding to each short-time frame through a multi-layer encoder to obtain an encoding result; a processing module, used for sending the encoding result to a multi-layer time domain convolutional network, so as to process the encoding result through each layer of the time domain convolutional network to obtain a feature extraction result; a decoding module, used for decoding the feature extraction result through a multi-layer decoder to obtain a target speech signal after the speech signal to be processed is denoised; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

[0017] An electronic device provided by the present invention includes a processor and a memory, wherein the memory stores machine executable instructions that can be executed by the processor, and the processor executes the machine executable instructions to implement any of the above-mentioned speech signal denoising methods based on the U-net structure.

[0018] The present invention provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement any of the above-mentioned speech signal denoising methods based on the U-net structure.

[0019] The speech signal denoising method and device based on the U-net structure provided by the present invention obtain a speech signal to be processed; perform short-time Fourier transform on the speech signal to be processed to obtain multiple short-time frames corresponding to the speech signal to be processed, and the amplitude spectrum corresponding to each short-time frame; for each short-time frame, encode the non-DC information in the amplitude spectrum corresponding to the short-time frame through a multi-layer encoder to obtain an encoding result; send the encoding result to a multi-layer time domain convolutional network to process the encoding result through each layer of the time domain convolutional network to obtain a feature extraction result; decode the feature extraction result through a multi-layer decoder to obtain a target speech signal after the denoising of the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder. This method adopts the U-net structure and performs processing based on the amplitude spectrum. Through the multi-layer encoder, time domain convolutional network and multi-layer decoder suitable for amplitude spectrum calculation, the amount of calculation can be reduced, the calculation efficiency can be improved, and the memory usage can be reduced, thereby improving the denoising effect of the speech signal to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 A flowchart of a method for reducing noise of a speech signal based on a U-net structure provided by an embodiment of the present invention;

[0022] Figure 2 A schematic diagram of the structure of a time domain convolutional network provided by an embodiment of the present invention;

[0023] Figure 3 A schematic diagram of a speech signal denoising method based on a U-net structure provided in an embodiment of the present invention;

[0024] Figure 4 A schematic diagram of a speech signal denoising system based on a U-net structure provided by an embodiment of the present invention;

[0025] Figure 5 A schematic diagram of the structure of a speech signal denoising device based on a U-net structure provided by an embodiment of the present invention;

[0026] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The speech noise reduction technology and system in the related art are designed to perform noise suppression on the captured speech signal through the speech transmission and receiving equipment, and usually implement noise elimination on the captured speech signal based on GRU. GRU is a variant of recurrent neural network used to process time series data. However, GRU has shortcomings in speech noise reduction: first, as a recursive structure, GRU usually requires more parameters to model the information in the signal, which may lead to large computational complexity and memory overhead in some real-time system applications, thereby reducing computational efficiency and having a certain impact on the noise reduction effect; second, GRU usually needs to iterate by time step, which may lead to higher computational complexity, thereby reducing real-time performance; finally, the long short-term memory property of GRU makes it more suitable for capturing long-term dependencies, which may not be the best choice in some speech scenarios. Based on this, an embodiment of the present invention provides a speech signal noise reduction method and device based on a U-net structure, which can be applied to applications that require noise reduction processing of speech signals.

[0029] To facilitate understanding of this embodiment, firstly, a speech signal denoising method based on a U-net structure disclosed in an embodiment of the present invention is introduced. The U-net structure consists of an encoder and a decoder, and can effectively use the skip connection between the encoder and the decoder to retain relatively low-level and relatively high-level features, thereby improving the generalization ability of the model and having a lower computational complexity. Figure 1 As shown, the method comprises the following steps:

[0030] Step S102, obtaining a speech signal to be processed;

[0031] The above-mentioned speech signal to be processed is usually a speech signal obtained by preprocessing the original speech signal collected from a microphone, etc. The speech signal to be processed usually contains various background noises, such as outdoor car sounds, bird sounds, etc. In actual implementation, when it is necessary to perform noise reduction processing on the speech signal, it is necessary to first obtain the speech signal to be processed that needs noise reduction.

[0032] Step S104, performing short-time Fourier transform on the speech signal to be processed to obtain a plurality of short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each short-time frame;

[0033] The short-time Fourier transform STFT (Short-Time Fourier Transform) can be used to determine the frequency and phase of the local sinusoidal wave of the time-varying signal; the above-mentioned short-time frame can be understood as multiple short-time signals obtained after segmenting the speech signal to be processed, and the length of each short-time frame is usually the same; the above-mentioned amplitude spectrum can be used to characterize the distribution of the amplitude of the signal with frequency; in actual implementation, after obtaining the speech signal to be processed, the speech signal to be processed can be short-time Fourier transformed to obtain multiple short-time frames and the amplitude spectrum corresponding to each short-time frame.

[0034] Step S106, for each short-time frame, encoding the non-DC information in the amplitude spectrum corresponding to the short-time frame through a multi-layer encoder to obtain an encoding result;

[0035] The above-mentioned encoder can be represented by Encoder, and the number of layers of the encoder can be set according to actual needs. For example, a 4-layer encoder can be used; in actual implementation, after obtaining the amplitude spectrum corresponding to each short-time frame, for each short-time frame, the non-DC information in the amplitude spectrum corresponding to the short-time frame can be extracted. The size of the non-DC information is usually FFT_n / 2 (FFT, Fast Fourier Transform), where FFT_n represents the number of sampling points of the short-time Fourier transform. In this embodiment, FFT_n can preferably use 512, then the number of sampling points of the non-DC information is 256, and the non-DC information can be used as input and encoded through a multi-layer encoder to obtain an encoding result.

[0036] Step S108, sending the encoding result to the multi-layer time domain convolutional network, so that the encoding result is processed by each layer of the time domain convolutional network to obtain a feature extraction result;

[0037] The above-mentioned time domain convolutional network TCN (Temporal Convolutional Network) is an algorithm that can be used to solve time series prediction; the number of layers of the time domain convolutional network TCN can be set according to actual needs, for example, a 6-layer time domain convolutional network TCN can be used; in actual implementation, the encoding results output by the above-mentioned multi-layer encoder can be input into the multi-layer time domain convolutional network to process the encoding results, capture the temporal relationship in the encoding results, and obtain feature extraction results.

[0038] Step S110, decoding the feature extraction result through a multi-layer decoder to obtain a target speech signal after noise reduction of the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

[0039] The above decoder can be represented by Decoder, and the number of layers of the decoder is usually the same as the number of layers of the encoder; in actual implementation, after obtaining the above feature extraction results, the feature extraction results can be input into a multi-layer decoder, and after decoding processing by the multi-layer decoder, a target speech signal after denoising the processed speech signal can be obtained, that is, the target speech signal is a speech signal obtained after reducing the noise and interference in the processed speech signal, and the clarity of the target speech signal is higher.

[0040] The above-mentioned speech signal denoising method based on the U-net structure obtains a speech signal to be processed; performs short-time Fourier transform on the speech signal to be processed to obtain multiple short-time frames corresponding to the speech signal to be processed, and the amplitude spectrum corresponding to each short-time frame; for each short-time frame, the non-DC information in the amplitude spectrum corresponding to the short-time frame is encoded by a multi-layer encoder to obtain an encoding result; the encoding result is sent to a multi-layer time domain convolutional network to process the encoding result by each layer of the time domain convolutional network to obtain a feature extraction result; the feature extraction result is decoded by a multi-layer decoder to obtain a target speech signal after the denoising of the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder. This method adopts the U-net structure and performs processing based on the amplitude spectrum. Through the multi-layer encoder, time domain convolutional network and multi-layer decoder suitable for amplitude spectrum calculation, the amount of calculation can be reduced, the calculation efficiency can be improved, and the memory usage can be reduced, thereby improving the denoising effect of the speech signal to be processed.

[0041] The embodiment of the present invention further provides another method for reducing noise of speech signals based on a U-net structure. The method is implemented on the basis of the method of the above embodiment, and the method comprises the following steps:

[0042] Step 1: obtaining an original voice signal collected by a voice collection device;

[0043] The above-mentioned voice collection device can be a device that can collect voice signals, such as a microphone and a recording device; the original voice signal can be understood as the actually collected voice signal; in actual implementation, the original voice signal can be collected by the voice collection device.

[0044] Step 2: preprocess the original speech signal to obtain a speech signal to be processed.

[0045] Usually, the above-mentioned original voice signal is usually more complex. Direct subsequent processing based on the original voice signal will lead to high computational complexity and poor performance. Therefore, the collected original voice signal is generally preprocessed. The preprocessing method may include at least one of the following: DC offset, sampling rate adjustment, audio normalization, etc., so that the preprocessed voice signal to be processed is within an appropriate range. Subsequent processing based on the voice signal to be processed can greatly reduce the computational complexity and improve performance.

[0046] Step 3, performing short-time Fourier transform on the speech signal to be processed to segment the speech signal to be processed and obtain multiple short-time frames corresponding to the speech signal to be processed;

[0047] Step 4: Decompose the signal of each short-time frame to obtain the amplitude spectrum corresponding to each short-time frame;

[0048] In actual implementation, the speech signal to be processed can be divided into multiple short-time frames through short-time Fourier transform. Since the amount of information in each short-time frame is small, it is easier to carry out subsequent processing; each short-time frame is decomposed into the amplitude information of different frequency components, and the amplitude spectrum corresponding to each short-time frame can be obtained.

[0049] Step 5: for each short-time frame, a multi-layer encoder is used to encode the non-DC information in the amplitude spectrum corresponding to the short-time frame to obtain an encoding result;

[0050] Each layer of the above-mentioned encoder includes: a depth convolution layer, a point convolution layer, a second normalization layer and a first non-linear activation layer connected in sequence; for each layer of the encoder, the depth convolution layer is used to perform a convolution operation on each input channel of the encoder of this layer respectively to obtain a depth convolution result; the point convolution layer is used to perform a point convolution operation on each depth convolution result to obtain a point convolution result; the second normalization layer is used to normalize the point convolution result to obtain a second normalized result; the non-linear activation layer is used to perform a nonlinear operation on the second normalized result to obtain the output result of the encoder of this layer.

[0051] Each layer of the amplitude encoder is composed of a deep convolution layer, a point convolution layer, a second normalization layer and a nonlinear activation layer. By separating the deep convolution layer and the point convolution layer, the overhead and the amount of calculation can be reduced; the second normalization layer can also be called a BN (Batch Normalization) layer, which is in the order of first passing through the deep convolution layer and then passing through the point convolution layer, wherein the number of input channels, the number of output channels and the number of convolution groups of the deep convolution layer are consistent with the number of input channels of the current layer, and the number of output channels of the point convolution layer is consistent with the number of output channels of the current layer, and the number of convolution groups is 1. In the multi-layer encoder, the input of the first layer encoder is the non-DC information in the amplitude spectrum corresponding to each short-time frame, and the input of each layer encoder except the first layer encoder is the output of the previous layer encoder, and the output of the last layer encoder is the encoding result after the above-mentioned multi-layer encoder encoding process.

[0052] Step 6, sending the encoding result to the multi-layer time domain convolutional network, so that the encoding result is processed by each layer of the time domain convolutional network to obtain a feature extraction result;

[0053] Each layer of the time domain convolution network includes: a first convolution layer, a second convolution layer, an information output module and a gated signal generation module which are parallel to each other; the information output module includes: a first activation function layer, a first normalization layer and a dilated convolution layer which are connected in sequence; the gated signal generation module includes a first activation function layer, a first normalization layer, a dilated convolution layer and a second activation function layer which are connected in sequence; the first activation function layer in the information output module and the first activation function layer in the gated signal generation module are also respectively connected to the first convolution layer;

[0054] For each layer of the time domain convolutional network, the first convolutional layer is used to perform convolution processing on the input signal received by the time domain convolutional network layer to obtain a first convolution result, and send the first convolution result to each first activation function layer respectively; each first activation function layer is used to perform nonlinear operation on the first convolution result respectively to obtain a first operation result, and send the first operation result to each first normalization layer connected to the corresponding layer respectively;

[0055] Each first normalization layer is used to normalize the first operation result to obtain a first normalized result, and send the first normalized result to each correspondingly connected expansion convolution layer;

[0056] Considering that the commonly used gLN (global normalization) only normalizes the channels, in the causal time domain convolutional network TCN, theoretically, the normalization effect of cLN, which normalizes layer by layer, will be better. Specifically, the first normalization layer is used for normalization processing using the following formula:

[0057]

[0058] Among them, f t represents the kth frame signal in the first operation result received by the first normalization layer; f t represents the t-th frame signal in the first operation result received by the first normalization layer; E[f t ≤k] means taking the average value from the first frame signal to the kth frame signal; var[f t ≤k] represents taking the variance from the first frame signal to the kth frame signal; ∈ is a preset value; γ and β are both trainable parameters that can be applied to each frame signal in the first operation result received by the first normalization layer.

[0059] Each dilated convolution layer is used to expand the receptive field of the first normalized result to obtain a first output result; the second activation function layer is also used to receive the first output result sent by the connected dilated convolution layer, perform mapping processing on the first output result, and output a gating signal; the second convolution layer is used to receive the processing result after multiplying the first output result output by the information output module and the gating signal, perform convolution processing on the processing result, and obtain the output result of the time domain convolution network of this layer.

[0060] For easier understanding, see Figure 2 The schematic diagram of the structure of a time domain convolutional network is shown in Figure 1. Each layer of the time domain convolutional network adopts Figure 2 In the structural form, the input of the previous layer of time domain convolution network is used as the output of the next layer of time domain convolution network; in each layer of time domain convolution network TCN, a gated convolution operation is added. The main difference between gated convolution and ordinary convolution is the introduction of a gating mechanism. Gated convolution usually consists of two parallel convolution kernels, one for generating the first output result and the other for generating the output gating signal. This gating signal controls the flow of information to a certain extent and plays a role in screening and regulation. Figure 2 As shown in the figure, for each layer of the time domain convolutional network, the input of the time domain convolutional network is sent to 1*1-Conv (corresponding to the first convolutional layer mentioned above), and the input can be channel-reduced through 1*1-Conv to output the first convolution result; the first convolution result is divided into two paths and enters two PReLUs (corresponding to the first activation function layer mentioned above) to increase nonlinear features. Figure 2 The PReLU on the left connects cLN (corresponding to the first normalization layer) and D-Conv (corresponding to the dilated convolution layer) in sequence to form the above information output module. Figure 2The PReLU on the right connects cLN (corresponding to the first normalization layer), D-Conv (corresponding to the dilated convolution layer) and Sigmoid (corresponding to the second activation function layer) in sequence to form the gated signal generation module. It can be seen that the gated signal generation module adds a Sigmoid on the basis of the information output module, so that the network has stronger expression ability when processing complex data. The calculation process of gated convolution is mainly divided into three steps. First, the input feature outputs the first output result through the information output module; secondly, the input feature outputs the gated signal through another gated signal generation module; the first output result output by the information output module is multiplied by the gated signal output by the gated signal generation module to obtain the processing result, which can realize the screening of information. Finally, the processing result is restored to the same dimension as the input input, that is, the same number of channels, through 1*1-Conv (corresponding to the second convolution layer).

[0061] Furthermore, in different layers of the temporal convolutional network, the dilation rates of the dilated convolutional layers are different;

[0062] For the time domain convolutional network TCN, the dilated convolution layer is a more important part in the structure. The dilated convolution layer can increase the receptive field while keeping the size of the feature map unchanged, thereby replacing the downsampling and upsampling operations. The change in the receptive field depends on the dilation rate of the dilated convolution layer. The dilation rate formula in this embodiment is:

[0063]

[0064] Among them, i is the number of layers of the time domain convolutional network TCN, and p is the expansion rate.

[0065] The last layer of the time domain convolutional network also includes a fully connected layer; the fully connected layer is used to perform fully connected processing on the output result of the second convolutional layer of the last layer of the time domain convolutional network to obtain the final output result of the last layer of the time domain convolutional network.

[0066] In this embodiment, the last layer of time domain convolutional network TCN includes a fully connected layer, wherein the number of output nodes of the fully connected layer is consistent with the number of input nodes of the first layer of time domain convolutional network TCN. The output result of the second convolutional layer of the last layer of time domain convolutional network is fully connected through the fully connected layer, and the final output result is the feature extraction result obtained after processing the encoding result through each layer of time domain convolutional network.

[0067] Step 7, decoding the feature extraction result through a multi-layer decoder to obtain an amplitude mask;

[0068] Each layer of the above-mentioned decoder includes: a second nonlinear activation layer, a third normalization layer, a point deconvolution layer and a depth deconvolution layer connected in sequence; for each layer of the decoder, the second nonlinear activation layer is used to perform an inverse linear operation on the input information of the decoder of this layer to obtain a second operation result; the third normalization layer is used to perform an inverse normalization process on the second operation result to obtain an inverse normalization result; the point deconvolution layer is used to perform an inverse dot product operation on the inverse normalization result to obtain a point deconvolution result; the depth deconvolution layer is used to perform a deconvolution operation on the point deconvolution result to obtain an output result of the decoder of this layer.

[0069] Each layer of amplitude form decoder is composed of the second nonlinear activation layer, the third normalization layer, the point deconvolution layer and the depth deconvolution layer. The third normalization layer can also be called the BN (Batch Normalization) layer. It adopts the order of first passing through the point deconvolution layer and then passing through the depth deconvolution layer. Among them, the number of input channels of the point deconvolution layer is consistent with the number of input channels of the current layer, the number of output channels is consistent with the number of output channels of the current layer, and the number of deconvolution groups is 1. The number of input channels, the number of output channels and the number of deconvolution groups of the depth deconvolution layer are all consistent with the number of output channels of the current layer; in the multi-layer decoder, the input of the first layer decoder is the feature extraction result obtained by processing the encoding result through each layer of the time domain convolution network, and the input of each layer of decoders except the first layer decoder is the output of the previous layer decoder. The output of the last layer decoder is the amplitude mask obtained after decoding the feature extraction result through the multi-layer decoder.

[0070] Step eight, performing mask processing on the amplitude spectrum based on the amplitude mask to obtain a mask processing result;

[0071] In actual implementation, after the amplitude mask is obtained, the amplitude mask can be applied to the amplitude spectrum to suppress noise and enhance the speech signal. Specifically, the amplitude mask can be multiplied by the amplitude spectrum to obtain a mask processing result.

[0072] Step nine, performing an inverse short-time Fourier transform on the mask processing result to obtain a denoised target speech signal corresponding to the speech signal to be processed.

[0073] The above-mentioned inverse short-time Fourier transform ISTFT (Inverse Short-Time Fourier Transform) can be used to convert the signal represented in the frequency domain back to the time domain. In actual implementation, the mask processing result can be converted back to the denoised target speech signal in the time domain form through the inverse short-time Fourier transform.

[0074] For easier understanding, see Figure 3A schematic diagram of a speech signal denoising method based on a U-net structure is shown, wherein it includes an encoder of N layers of amplitude spectrum (corresponding to the above-mentioned multi-layer encoder), a TCN of M layers of amplitude spectrum (corresponding to the above-mentioned multi-layer time domain convolution network) and a decoder of N layers of amplitude spectrum (corresponding to the above-mentioned multi-layer decoder). In this embodiment, N=4, M=6 are preferably used, and the number of output channels of encoder 1 to encoder 4 is preferably 1, 32, 32 and 32, and the number of output channels of decoder 1 to decoder 4 is preferably 1, 32, 32 and 32 respectively. The number of input channels and the number of output channels of each encoder are the same, and the number of input channels and the number of output channels of each decoder are the same. The number of input nodes of the 6-layer time domain convolution network TCN in amplitude form is the output size of encoder 4. The output size of encoder 4 is first converted to 64 through a layer of one-dimensional point convolution, and then the local and global features are captured through 6 layers of the same TCN. The 6-layer time domain convolution network TCN is the same except for the different expansion rate of the expansion convolution. After the noisy frequency spectrum (corresponding to the above original speech signal) is processed by the short-time Fourier transform STFT, the amplitude spectrum corresponding to each short-time frame is obtained. The non-DC information in the amplitude spectrum corresponding to each short-time frame is used as input. After being processed by 4 layers of encoders, 6 layers of TCN and 4 layers of decoders, an amplitude mask with the same size as the input can be obtained. Through the inverse short-time Fourier transform ISTFT, the denoised frequency spectrum (corresponding to the above target speech signal) can be obtained. In addition, each layer of encoder is connected to the decoder of the corresponding layer to perform residual calculation. For example, the input of decoder1 is the sum of the output of decoder2 and the output of encoder1.

[0075] The following is a speech signal denoising system based on the U-net structure. Figure 4As shown, the system includes: an acquisition unit, a computing unit, a data transmission unit and a terminal unit; wherein the acquisition unit includes a microphone and an ADC (Analog to Digital Converter) hardware chip, which is responsible for converting the voice analog signal in the environment into a digital signal; the computing unit is composed of a single-chip microcomputer or a computing chip with an operating system, which is used to perform the calculation of the noise reduction model; the processing process of the above-mentioned voice signal noise reduction method in the U-net structure is mainly implemented by the computing unit; the data transmission unit includes a network system that can transmit data, which is used to transmit the calculated data; the terminal unit is composed of any real-time conference communication device with network access, which is used to play the processed target voice signal. These units together constitute a system, starting from the acquisition of voice signals, through calculation and data transmission, and finally transmitting the target voice signal after noise reduction processing to the terminal device for playback.

[0076] The above-mentioned speech signal denoising method based on the U-net structure uses the Unet structure based on amplitude operation, whose input is the amplitude spectrum and the output is the amplitude mask. In the Encoder part, it is implemented by combining the deep convolution and point convolution. In the Decoder part, point deconvolution and deep deconvolution are adopted. On the basis of the U-net structure, the amplitude spectrum is used as input, and the multi-layer encoder Encoder, multi-layer time domain convolution network TCN and multi-layer decoder Decoder are improved by amplitude operation, and modified into a multi-layer encoder Encoder, a multi-layer time domain convolution network TCN and a multi-layer decoder Decoder in the form of amplitude.

[0077] This method is an amplitude spectrum denoising method based on CNN (Convolutional Neural Networks), which can overcome the problems existing in the prior art. CNN has the advantage of parallel computing and can process multiple positions of the input sequence at the same time. In contrast, GRU usually needs to iterate by time step, which may lead to low computational efficiency. Since CNN can improve efficiency through parallel computing, it may have more advantages in real-time performance, which makes CNN more suitable for real-time application scenarios that require fast response. In addition, CNN has translation invariance, which means that it is sensitive to position changes of the input signal. In speech signals, burst noise is one of the most common scenarios in noise reduction, so models with translation invariance are more suitable for handling this situation. However, CNN is not as good as GRU in handling time-series scenarios, so this solution replaces TCN with ordinary CNN. TCN combines the local feature extraction ability of the convolutional layer and the global capture ability of the dilated convolution, so that it can effectively capture different features in time series data. Therefore, the advantages of TCN in the audio field are mainly reflected in the processing of long dependencies, the efficiency of parallel computing and parameter sharing. In addition, the dilated convolution in TCN can control the length of time dependency and has high flexibility. In general, the system based on TCN amplitude spectrum denoising has higher adaptability and performance, can provide better speech denoising effect in various complex noise environments, and can be applied to application fields such as voice communication, audio processing and speech recognition.

[0078] The above-mentioned speech signal denoising method based on the U-net structure, firstly, by adopting the U-net structure of amplitude operation mode, the denoising effect of the model is significantly improved, especially in low signal-to-noise ratio, low volume and burst noise scenes, and the restoration ability of human voice is greatly improved. Secondly, by adopting a special Encoder and Decoder implementation method, the amount of model calculation is effectively reduced, so that the model can run in real time on a platform with lower performance; furthermore, the TCN model has the advantages of CNN network feature extraction ability, fast inference speed and parallel operation, and its unique expansion convolution enhances the model's time series data processing ability, and can be aligned with or even better than GRU in the case of lower performance consumption than GRU. In addition, by adding gated convolution operation, the network can learn more complex feature representations, and the gating mechanism helps the network learn long-distance dependencies, making the network more effective in processing tasks such as time series data. Finally, by introducing the amplitude mask, the complexity of model learning is reduced, which is conducive to model training. The entire system can be flexibly deployed in real-time conference communication equipment to achieve real-time noise reduction and suppression of speech.

[0079] The embodiment of the present invention provides a speech signal denoising device based on a U-net structure, such as Figure 5As shown, the device includes: an acquisition module 50, used to acquire a speech signal to be processed; a Fourier transform module 51, used to perform short-time Fourier transform on the speech signal to be processed, and obtain multiple short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each short-time frame; an encoding module 52, used to encode the non-DC information in the amplitude spectrum corresponding to each short-time frame through a multi-layer encoder to obtain an encoding result; a processing module 53, used to send the encoding result to a multi-layer time domain convolutional network, so as to process the encoding result through each layer of the time domain convolutional network to obtain a feature extraction result; a decoding module 54, used to decode the feature extraction result through a multi-layer decoder to obtain a target speech signal after denoising the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

[0080] The above-mentioned speech signal denoising device based on U-net structure adopts U-net structure and performs processing based on amplitude spectrum. Through multi-layer encoders, time domain convolutional networks and multi-layer decoders suitable for amplitude spectrum operations, it can reduce the amount of calculation and improve the calculation efficiency, while reducing the memory usage and improving the denoising effect on the processed speech signal.

[0081] Furthermore, each layer of the time domain convolutional network includes: a first convolutional layer, a second convolutional layer, an information output module and a gated signal generation module that are parallel to each other; the information output module includes: a first activation function layer, a first normalization layer and a dilated convolutional layer that are connected in sequence; the gated signal generation module includes a first activation function layer, a first normalization layer, a dilated convolutional layer and a second activation function layer that are connected in sequence; the first activation function layer in the information output module and the first activation function layer in the gated signal generation module are also respectively connected to the first convolutional layer;

[0082] For each layer of the time domain convolutional network, the first convolutional layer is used to perform convolution processing on the input signal received by the time domain convolutional network of this layer to obtain a first convolution result, and send the first convolution result to each first activation function layer respectively; each first activation function layer is used to perform nonlinear operation on the first convolution result respectively to obtain a first operation result, and send the first operation result to each correspondingly connected first normalization layer respectively; each first normalization layer is used to perform normalization processing on the first operation result respectively to obtain a first normalization result, and send the first normalization result to each correspondingly connected expansion convolutional layer respectively; each expansion convolutional layer is used to expand the receptive field of the first normalization result respectively to obtain a first output result; the second activation function layer is also used to receive the first output result sent by the connected expansion convolutional layer, map the first output result, and output a gating signal; the second convolutional layer is used to receive the processing result after multiplying the first output result output by the information output module and the gating signal, and convolve the processing result to obtain the output result of the time domain convolutional network of this layer.

[0083] Furthermore, in different layers of the time domain convolutional network, the expansion rates of the expansion convolutional layers are different; the last layer of the time domain convolutional network also includes a fully connected layer; the fully connected layer is used to perform fully connected processing on the output result of the second convolutional layer of the last layer of the time domain convolutional network to obtain the final output result of the last layer of the time domain convolutional network.

[0084] Furthermore, each layer of the encoder includes: a deep convolution layer, a point convolution layer, a second normalization layer and a first non-linear activation layer connected in sequence; for each layer of the encoder, the deep convolution layer is used to perform a convolution operation on each input channel of the encoder of this layer respectively to obtain a deep convolution result; the point convolution layer is used to perform a point convolution operation on each deep convolution result to obtain a point convolution result; the second normalization layer is used to normalize the point convolution result to obtain a second normalized result; the non-linear activation layer is used to perform a nonlinear operation on the second normalized result to obtain an output result of the encoder of this layer.

[0085] Furthermore, each layer of the decoder includes: a second nonlinear activation layer, a third normalization layer, a point deconvolution layer and a depth deconvolution layer connected in sequence; for each layer of the decoder, the second nonlinear activation layer is used to perform an inverse linear operation on the input information of the decoder of this layer to obtain a second operation result; the third normalization layer is used to perform an inverse normalization process on the second operation result to obtain an inverse normalized result; the point deconvolution layer is used to perform an inverse dot product operation on the inverse normalization result to obtain a point deconvolution result; the depth deconvolution layer is used to perform a deconvolution operation on the point deconvolution result to obtain an output result of the decoder of this layer.

[0086] Furthermore, the decoding module 54 is also used to: decode the feature extraction result through a multi-layer decoder to obtain an amplitude mask; mask the amplitude spectrum based on the amplitude mask to obtain a mask processing result; perform an inverse short-time Fourier transform on the mask processing result to obtain a denoised target speech signal corresponding to the speech signal to be processed.

[0087] Furthermore, the first normalization layer is used to perform normalization using the following formula:

[0088]

[0089] Among them, ft represents the k-th frame signal in the first operation result received by the first normalization layer; ft represents the t-th frame signal in the first operation result received by the first normalization layer; E[ft≤k] represents taking the mean of the first frame signal to the k-th frame signal; var[ft≤k] represents taking the variance of the first frame signal to the k-th frame signal; ∈ is a preset value; γ and β are both trainable parameters that can be applied to each frame signal in the first operation result received by the first normalization layer.

[0090] Furthermore, the acquisition module 50 is also used to: acquire an original voice signal collected by a voice collection device; and pre-process the original voice signal to obtain a voice signal to be processed.

[0091] Furthermore, the Fourier transform module 51 is also used to: perform short-time Fourier transform on the speech signal to be processed, so as to segment the speech signal to be processed and obtain multiple short-time frames corresponding to the speech signal to be processed; perform signal decomposition on each short-time frame and obtain the amplitude spectrum corresponding to each short-time frame.

[0092] The speech signal denoising device based on the U-net structure provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned speech signal denoising method based on the U-net structure. For the sake of brief description, for matters not mentioned in the embodiment of the speech signal denoising device based on the U-net structure, reference may be made to the corresponding contents in the aforementioned speech signal denoising method based on the U-net structure.

[0093] The embodiment of the present invention further provides an electronic device, see Figure 6 As shown, the electronic device includes a processor 130 and a memory 131, wherein the memory 131 stores machine executable instructions that can be executed by the processor 130, and the processor 130 executes the machine executable instructions to implement the above-mentioned speech signal denoising method based on the U-net structure.

[0094] Further, Figure 6 The electronic device shown further includes a bus 132 and a communication interface 133 , and the processor 130 , the communication interface 133 and the memory 131 are connected via the bus 132 .

[0095] The memory 131 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 133 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 132 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0096] The processor 130 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 130. The above processor 130 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 131, and the processor 130 reads the information in the memory 131 and completes the steps of the method of the above embodiment in combination with its hardware.

[0097] An embodiment of the present invention also provides a machine-readable storage medium, which stores machine-executable instructions. When the machine-executable instructions are called and executed by a processor, the machine-executable instructions prompt the processor to implement the above-mentioned speech signal denoising method based on the U-net structure. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0098] The computer program product of the speech signal denoising method and device based on the U-net structure provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the method described in the previous method embodiment. The specific implementation can be referred to the method embodiment, which will not be repeated here.

[0099] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech signal denoising method based on U-net structure, characterized in that: The method comprises: Obtaining a voice signal to be processed; Performing short-time Fourier transform on the speech signal to be processed to obtain a plurality of short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each of the short-time frames; For each of the short-time frames, encoding the non-DC information in the amplitude spectrum corresponding to the short-time frame is performed by a multi-layer encoder to obtain an encoding result; Sending the encoding result to a multi-layer time-domain convolutional network, so that the encoding result is processed by each layer of the time-domain convolutional network to obtain a feature extraction result; The feature extraction result is decoded by a multi-layer decoder to obtain a target speech signal after noise reduction of the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

2. The method according to claim 1, characterized in that Each layer of the time domain convolution network includes: a first convolution layer, a second convolution layer, an information output module and a gate signal generation module which are parallel to each other; the information output module includes: a first activation function layer, a first normalization layer and a dilated convolution layer which are connected in sequence; the gate signal generation module includes the first activation function layer, the first normalization layer, the dilated convolution layer and the second activation function layer which are connected in sequence; the first activation function layer in the information output module and the first activation function layer in the gate signal generation module are also respectively connected to the first convolution layer; For each layer of the time domain convolutional network, the first convolutional layer is used to perform convolution processing on the input signal received by the time domain convolutional network layer to obtain a first convolution result, and send the first convolution result to each of the first activation function layers respectively; Each of the first activation function layers is used to perform nonlinear operations on the first convolution results to obtain first operation results, and send the first operation results to each of the first normalization layers connected to the first convolution results; Each of the first normalization layers is used to perform normalization processing on the first operation results respectively to obtain first normalized results, and send the first normalized results to each of the correspondingly connected dilated convolutional layers respectively; Each of the dilated convolutional layers is used to expand the receptive field of the first normalized result to obtain a first output result; The second activation function layer is further used to receive the first output result sent by the connected dilated convolutional layer, perform mapping processing on the first output result, and output a gating signal; The second convolution layer is used to receive a processing result after multiplying the first output result output by the information output module and the gating signal, and perform convolution processing on the processing result to obtain an output result of the time domain convolution network of this layer.

3. The method according to claim 2, characterized in that In the time domain convolutional networks of different layers, the dilation rates of the dilated convolutional layers are different; the last layer of the time domain convolutional network also includes a fully connected layer; The fully connected layer is used to perform fully connected processing on the output result of the second convolutional layer of the last layer of the time domain convolutional network to obtain the final output result of the last layer of the time domain convolutional network.

4. The method according to claim 1, characterized in that Each layer of the encoder includes: a depth convolution layer, a point convolution layer, a second normalization layer and a first nonlinear activation layer connected in sequence; For each layer of the encoder, the depth convolution layer is used to perform a convolution operation on each input channel of the encoder of the layer to obtain a depth convolution result; The point convolution layer is used to perform a point product operation on each of the depth convolution results to obtain a point convolution result; The second normalization layer is used to normalize the point convolution result to obtain a second normalized result; The nonlinear activation layer is used to perform a nonlinear operation on the second normalization result to obtain an output result of the encoder of this layer.

5. The method according to claim 1, characterized in that Each layer of the decoder comprises: a second nonlinear activation layer, a third normalization layer, a point deconvolution layer and a depth deconvolution layer connected in sequence; For each layer of the decoder, the second nonlinear activation layer is used to perform an inverse linear operation on the input information of the decoder of the layer to obtain a second operation result; The third normalization layer is used to perform a denormalization process on the second operation result to obtain a denormalized result; The point deconvolution layer is used to perform a deconvolution operation on the denormalized result to obtain a point deconvolution result; The depth deconvolution layer is used to perform a deconvolution operation on the point deconvolution result to obtain the output result of the decoder of this layer.

6. The method according to claim 1, characterized in that The step of decoding the feature extraction result by a multi-layer decoder to obtain a target speech signal after the noise of the speech signal to be processed comprises: Decoding the feature extraction result through a multi-layer decoder to obtain an amplitude mask; performing mask processing on the amplitude spectrum based on the amplitude mask to obtain a mask processing result; An inverse short-time Fourier transform is performed on the mask processing result to obtain a noise-reduced target speech signal corresponding to the speech signal to be processed.

7. The method according to claim 2, characterized in that The first normalization layer is used to perform normalization processing using the following formula: Among them, ft represents the kth frame signal in the first operation result received by the first normalization layer; ft represents the tth frame signal in the first operation result received by the first normalization layer; E[ft≤k] represents taking the average of the first frame signal to the kth frame signal; var[ft≤k] represents taking the variance of the first frame signal to the kth frame signal; ∈ is a preset value; γ and β are both trainable parameters that can be applied to each frame signal in the first operation result received by the first normalization layer.

8. The method according to claim 1, characterized in that The steps of obtaining the speech signal to be processed include: Acquire the original voice signal collected by the voice collection device; The original speech signal is preprocessed to obtain a speech signal to be processed.

9. The method according to claim 1, characterized in that: The step of performing short-time Fourier transform on the speech signal to be processed to obtain a plurality of short-time frames corresponding to the speech signal to be processed and an amplitude spectrum corresponding to each of the short-time frames comprises: Performing short-time Fourier transform on the speech signal to be processed to segment the speech signal to be processed and obtain a plurality of short-time frames corresponding to the speech signal to be processed; Signal decomposition is performed on each of the short-time frames to obtain an amplitude spectrum corresponding to each of the short-time frames.

10. A speech signal denoising device based on a U-net structure, characterized in that: The device comprises: An acquisition module, used for acquiring a speech signal to be processed; A Fourier transform module, used for performing a short-time Fourier transform on the speech signal to be processed to obtain a plurality of short-time frames corresponding to the speech signal to be processed, and an amplitude spectrum corresponding to each of the short-time frames; An encoding module, used for encoding non-DC information in an amplitude spectrum corresponding to each short-time frame through a multi-layer encoder to obtain an encoding result; A processing module, used for sending the encoding result to a multi-layer time-domain convolutional network, so as to process the encoding result through each layer of the time-domain convolutional network to obtain a feature extraction result; A decoding module is used to decode the feature extraction result through a multi-layer decoder to obtain a target speech signal after denoising the speech signal to be processed; wherein the number of layers of the encoder is the same as the number of layers of the decoder.

Images