A network audio enhancement method based on deep learning

Abstract

The invention discloses a network audio enhancement method based on deep learning, comprising the following steps: step one, audio input; step two, speech feature extraction; step three, feature refinement; step four, jump connection; step five, speech signal recovery; step six, audio output; in the step two, h is the number of output channels, which is 48; in the step three, a SEQUENCE network is responsible for extracting finer-grained features of an output sequence of an ENCODER network based on an encoder part in a transformer structure. The invention proposes an improved version of U‑net suitable for speech enhancement, which consists of a convolution-based and SEQUENCE network, and generates and optimizes a model through hierarchical generation, so that the clean version of the speech signal is directly output, while minimizing the regression loss function. The PESQ of the model on the data set Valentinibenchmark reaches 3.02, and the STOI (%) is 95. When it is actually put into production, the cost of operation can be shortened through objective and subjective measurements, and the performance meets industrial requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of speech noise reduction, and in particular to a network audio enhancement method based on deep learning. Background Art

[0002] On voice devices, the recorded conversational voice signals contain some form of noise that hinders the intelligibility of the voice, such as street noise, dog barking, keyboard typing, etc. The task of speech enhancement is to maximize the perceived quality of the voice signal, especially by removing background noise. Therefore, speech enhancement itself is a very important task for conference audio equipment. On the one hand, it needs to run in real time on the communication equipment with as little delay as possible, and on the other hand, it needs to improve the clarity of the sound after noise reduction. The traditional speech enhancement method is to estimate the noise model and use it to restore the noise-subtracted speech, either by generating an enhanced version of the amplitude spectrum or generating an estimate of the ideal binary mask, which is then used to enhance the amplitude spectrum. Although it has good cross-domain generalization, it still has difficulties in dealing with common noises, such as the non-stationary noise or babbling noise encountered when a group of people talk at the same time. The existence of this type of noise greatly reduces the hearing clarity of human language. Summary of the invention

[0003] The purpose of the present invention is to provide a network audio enhancement method based on deep learning to solve the problems raised in the above background technology.

[0004] To achieve the above object, the present invention provides the following technical solution: a network audio enhancement method based on deep learning, comprising the following steps: step one, audio input; step two, speech feature extraction; step three, feature refinement; step four, jump connection; step five, speech signal recovery; step six, audio output;

[0005] In the above step 1, a time domain audio signal x∈R mixed with noise is input n , where n is the length of the audio time domain;

[0006] In the above step 2, the speech feature extraction includes the following steps:

[0007] 1) Use the first downsampling layer of the ENCODER network to extract features from the input audio signal x and obtain the feature encoding vector Where t1 is the number of frames after downsampling, which is equal to n / 4;

[0008] 2) Use the second downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x1 to obtain the feature encoding vector Where t2 is the number of frames after downsampling, which is equal to t1 / 4;

[0009] 3) Use the third downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x2 to obtain the feature encoding vector Where t3 is the number of frames after downsampling, which is equal to t2 / 4;

[0010] 4) Use the fourth downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x3 to obtain the feature encoding vector Where t4 is the number of frames after downsampling, which is equal to t3 / 4;

[0011] 5) Use the fifth downsampling layer of the ENCODER network to extract the feature encoding vector enc_x4 and obtain the feature encoding vector Where t5 is the number of frames after downsampling, which is equal to t4 / 4;

[0012] In the above step 3, the SEQUENCE network is used to extract the sequence feature enc_x5, and the

[0013] In the above step 4, a connection between the ENCODER network and the DECODER network is established based on a jump connection;

[0014] In the above step 5, voice signal recovery includes the following steps:

[0015] 1) Use the DECODER network to combine att_x and enc_x5 to get dec_x = att_x + enc_x5, and use the first upsampling layer of the DECODER network to upsample dec_x to get

[0016] 2) Use the DECODER network to combine dec_x1 and enc_x4 to obtain dec_x = dec_x1 + enc_x4, and use the second upsampling layer of the DECODER network to upsample dec_x to obtain

[0017] 3) Use the DECODER network to combine dec_x2 and enc_x3 to get dec_x = dec_x2 + enc_x3, and use the third upsampling layer of the DECODER network to upsample dec_x to get

[0018] 4) Use the DECODER network to combine dec_x3 and enc_x2 to obtain dec_x = dec_x3 + enc_x2, and use the fourth upsampling layer of the DECODER network to upsample dec_x to obtain

[0019] 5) Use the DECODER network to combine dec_x4 and enc_x1 to obtain dec_x=dec_x4+enc_x1, and use the fifth upsampling layer of the DECODER network to upsample dec_x to obtain dec_x5∈RN;

[0020] In the above step 6, the audio after signal recovery can be output, and high-quality voice audio can be obtained at one end.

[0021] Preferably, in step 2, h represents the number of output channels, and the specific number is 48.

[0022] Preferably, in the step 2, each sampling layer of the ENCODER network includes two one-dimensional convolutions, the input channel number channel_in of the first convolution is 1, the output channel number channel_out is 48, the kernel size kernel_size is 8, and the stride is 4; the input channel number channel_in of the second convolution is 48, the output channel number channel_out is 48, the kernel size kernel_size is 1, and the stride is 1.

[0023] Preferably, in step three, the SEQUENCE network is based on the encoder part in the transformer structure and is responsible for performing more fine-grained feature extraction on the output sequence of the ENCODER network.

[0024] Preferably, in step 4, the skip connection structure can transfer the features extracted from the downsampling layer of the ENCODER network to the upsampling layer of the ECODER network.

[0025] Preferably, in the step five, each sampling layer of the DECODER network includes a one-dimensional convolution and a deconvolution operation, the number of input channels (channel_in) of the one-dimensional convolution is 48, the number of output channels (channel_out) is 48, the kernel size (kernel_size) is 1, and the stride (stride) is 1; the number of input channels (channel_in) of the deconvolution is 48, the number of output channels (channel_out) is 48, the kernel size (kernel_size) is 8, and the stride (stride) is 4.

[0026] Compared with the prior art, the beneficial effects of the present invention are as follows: the present invention proposes an improved version of U-net suitable for speech enhancement, which is composed of a convolution-based and SEQUENCE network. By generating and optimizing the model in layers, the model can directly output a clean version of the speech signal while minimizing the regression loss function. The PESQ of the model on the Valentini benchmark dataset reaches 3.02, and the STOI (%) is 95. When it is actually put into production, the operating cost can be shortened through objective and subjective measurements, and the performance meets industrial requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 is a flow chart of the method of the present invention;

[0028] Figure 2 This is a diagram of an operation example of the present invention. DETAILED DESCRIPTION

[0029] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] See also Figure 1-2 , an embodiment provided by the present invention: a network audio enhancement method based on deep learning, comprising the following steps: step one, audio input; step two, speech feature extraction; step three, feature refinement; step four, skip connection; step five, speech signal recovery; step six, audio output;

[0031] In the above step 1, a time domain audio signal x∈R mixed with noise is input n , where n is the length of the audio time domain;

[0032] In the above step 2, the speech feature extraction includes the following steps:

[0033] 1) Use the first downsampling layer of the ENCODER network to extract features from the input audio signal x and obtain the feature encoding vector Where t1 is the number of frames after downsampling, which is equal to n / 4;

[0034] 2) Use the second downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x1 to obtain the feature encoding vector Where t2 is the number of frames after downsampling, which is equal to t1 / 4;

[0035] 3) Use the third downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x2 to obtain the feature encoding vector Where t3 is the number of frames after downsampling, which is equal to t2 / 4;

[0036] 4) Use the fourth downsampling layer of the ENCODER network to extract features from the feature encoding vector enc_x3 to obtain the feature encoding vector Where t4 is the number of frames after downsampling, which is equal to t3 / 4;

[0037] 5) Use the fifth downsampling layer of the ENCODER network to extract the feature encoding vector enc_x4 and obtain the feature encoding vector Where t5 is the number of frames after downsampling, which is equal to t4 / 4;

[0038] Among them, h represents the number of output channels, the specific number is 48. Each sampling layer of the ENCODER network includes two one-dimensional convolutions. The input channel number channel_in of the first convolution is 1, the output channel number channel_out is 48, the kernel size kernel_size is 8, and the stride is 4. The input channel number channel_in of the second convolution is 48, the output channel number channel_out is 48, the kernel size kernel_size is 1, and the stride is 1.

[0039] In the above step 3, the SEQUENCE network is used to extract the sequence feature enc_x5, and the The SEQUENCE network is based on the encoder part of the transformer structure and is responsible for extracting finer-grained features from the output sequence of the ENCODER network.

[0040] In the above step 4, a connection is established between the ENCODER network and the DECODER network based on a skip connection. The skip connection structure can transfer the features extracted from the downsampling layer of the ENCODER network to the upsampling layer of the ECODER network.

[0041] In the above step 5, voice signal recovery includes the following steps:

[0042] 1) Use the DECODER network to combine att_x and enc_x5 to get dec_x = att_x + enc_x5, and use the first upsampling layer of the DECODER network to upsample dec_x to get

[0043] 2) Use the DECODER network to combine dec_x1 and enc_x4 to obtain dec_x = dec_x1 + enc_x4, and use the second upsampling layer of the DECODER network to upsample dec_x to obtain

[0044] 3) Use the DECODER network to combine dec_x2 and enc_x3 to get dec_x = dec_x2 + enc_x3, and use the third upsampling layer of the DECODER network to upsample dec_x to get

[0045] 4) Use the DECODER network to combine dec_x3 and enc_x2 to obtain dec_x = dec_x3 + enc_x2, and use the fourth upsampling layer of the DECODER network to upsample dec_x to obtain

[0046] 5) Use the DECODER network to combine dec_x4 and enc_x1 to obtain dec_x=dec_x4+enc_x1, and use the fifth upsampling layer of the DECODER network to upsample dec_x to obtain dec_x5∈RN;

[0047] Among them, each sampling layer of the DECODER network includes a one-dimensional convolution and a deconvolution operation. The number of input channels (channel_in) of the one-dimensional convolution is 48, the number of output channels (channel_out) is 48, the kernel size (kernel_size) is 1, and the stride (stride) is 1. The number of input channels (channel_in) of the deconvolution is 48, the number of output channels (channel_out) is 48, the kernel size (kernel_size) is 8, and the stride (stride) is 4;

[0048] In the above step 6, the audio after signal recovery can be output, and high-quality voice audio can be obtained at one end.

[0049] The above operation requires the use of loss functions and data enhancement. The loss function uses the L1 loss on the waveform together with the multi-resolution STFT loss on the spectrum amplitude. Formally, given y and ^y as the clean signal and the enhanced signal respectively, the STFT loss is defined as the sum of the spectral convergence loss (sc) and the amplitude loss. The specific formula is as follows:

[0050] L stft (y, ^y) = L sc (y, ^y) + L mag (y, ^y)

[0051] Lstft (y,^y)=(|||STFT(y)-STFT(^y)|||F)÷|||STFT(y)|||F

[0052] L stft (y,^y)=1 / T||log|STFT(y)|-log|STFT(^y)|||1

[0053] where ||.||F and ||.||1 are the Frobenius of the L1 norm, respectively. The multi-resolution STFT loss is defined as the sum of all STFT loss functions using different STFT parameters. The goal is to reduce the following:

[0054] where M is the number of STFT losses, each The STFT loss is applied with different resolutions and numbers of FFT units ∈ {512, 1024, 2048}, hop sizes ∈ {50, 120, 240} and finally window lengths ∈ {240, 600, 1200}.

[0055] Signal enhancement includes shift enhancement and feature enhancement. The shift enhancement operation is as follows: a random shift is applied between 0 and S seconds. Given an initial gain λ, early delay τ and RT60, it adds a series of N attenuated clean speech and noise echoes to the noise signal. The delay of the nth echo is nτ+jitter, and the gain is ρnλ. The selection of n and ρ makes it possible to obtain ρN≤le-3 when the total delay reaches RT60. λ, τ and RT60 are uniformly sampled in [0,0.3]s, [10,30]ms and [0.3,1.3]s respectively.

[0056] The feature enhancement operation is as follows: the noise is shuffled within a batch to form a new noise mixture, a band-stop digital filter is used with a stopband between f0 and f1, and sampling is performed to uniformly remove 20% of the frequencies in the mel scale, similar to SpecAug enhancement for ASR training.

[0057] Based on the above, the advantage of the present invention is that it provides an end-to-end speech enhancement method based on a deep learning model, which is significantly better than traditional methods in processing non-stationary noise and babble noise, and produces higher quality speech. The core part of this method is to propose an improved version of U-net suitable for speech enhancement, which consists of a convolutional and SEQUENCE network with a frame size of 64 milliseconds. The model is generated from waveform to waveform through layered generation (using U-Net like skip connections), and by optimizing the model, it directly outputs a "clean" version of the speech signal.

[0058] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A network audio enhancement method based on deep learning, comprising the following steps: Step 1, audio input; Step 2, speech feature extraction; Step 3, feature refinement; Step 4, jump connection; Step 5, speech signal recovery; Step 6, audio output; It is characterized by: In the above step 1, a time domain audio signal mixed with noise is input , where n is the length of the audio time domain; In the above step 2, the speech feature extraction includes the following steps: 1) Use the first downsampling layer of the ENCODER network to extract features from the input audio signal x and obtain the feature encoding vector ,in is the number of frames after downsampling, equal to ; 2) Use the second downsampling layer of the ENCODER network to encode the feature vector Perform feature extraction to obtain feature encoding vector ,in is the number of frames after downsampling, equal to ; 3) Use the third downsampling layer of the ENCODER network to encode the feature vector Perform feature extraction to obtain feature encoding vector ,in is the number of frames after downsampling, equal to ; 4) Use the fourth downsampling layer of the ENCODER network to encode the feature vector Perform feature extraction to obtain feature encoding vector ,in is the number of frames after downsampling, equal to ; 5) Use the fifth downsampling layer of the ENCODER network to encode the feature vector Perform feature extraction to obtain feature encoding vector ,in is the number of frames after downsampling, equal to ; In step 2, h is the number of output channels, which is 48; In the above step 3, the SEQUENCE network is used to analyze the sequence features. Perform feature extraction and obtain ; In the above step 4, a connection between the ENCODER network and the DECODER network is established based on a jump connection; In the above step 5, voice signal recovery includes the following steps: 1) Utilize DECODER network to join and ,get , using the first upsampling layer of the DECODER network Upsampling, we get ; 2) Utilize DECODER network to join and ,get , using the second upsampling layer of the DECODER network Upsampling, we get ; 3) Utilize DECODER network to join and ,get , using the third upsampling layer of the DECODER network Upsampling, we get ; 4) Utilize DECODER network to join and ,get , using the fourth upsampling layer of the DECODER network Upsampling, we get ; 5) Utilize DECODER network to join and ,get , using the fifth upsampling layer of the DECODER network Upsampling, we get ; In the above step 6, the audio after signal recovery can be output, and high-quality voice audio can be obtained at one end.

2. According to the deep learning-based network audio enhancement method of claim 1, it is characterized by: In the step 2, each sampling layer of the ENCODER network includes two one-dimensional convolutions. The input channel number channel_in of the first convolution is 1, the output channel number channel_out is 48, the kernel size kernel_size is 8, and the stride is 4. The input channel number channel_in of the second convolution is 48, the output channel number channel_out is 48, the kernel size kernel_size is 1, and the stride is 1.

3. According to the deep learning-based network audio enhancement method of claim 1, it is characterized in that: In the step 3, the SEQUENCE network is based on the encoder part in the transformer structure and is responsible for extracting finer-grained features from the output sequence of the ENCODER network.

4. The method for enhancing network audio based on deep learning according to claim 1, characterized in that: In the step 4, the skip connection structure can transfer the features extracted from the downsampling layer of the ENCODER network to the upsampling layer of the ECODER network.

5. The method for enhancing network audio based on deep learning according to claim 1, characterized in that: In the step 5, each sampling layer of the DECODER network includes a one-dimensional convolution and a deconvolution operation. The number of input channels of the one-dimensional convolution is 48, the number of output channels is 48, the kernel size is 1, and the stride is 1. The number of input channels of the deconvolution is 48, the number of output channels is 48, the kernel size is 8, and the stride is 4.

Images