Automatic restoration method of missing audio based on U-Net model

Abstract

The application discloses a kind of based on U-Net model's missing audio automatic repair method, including the following contents: audio signal is handled to the sample of same length and constructs multiple audio data sets;All audio samples are handled into fixed length three parts, from left to right respectively left context part, gap part and right context part;Using short-time Fourier transform, audio context part and gap part are all converted into spectrum diagram;The encoder in deep learning model carries out feature learning to the spectrum diagram of context part, and the decoder realizes the automatic repair of gap part missing audio, and the two corresponding layers are connected by jump connection to expand network receiving range, improve network repair performance;Finally, the generated spectrum diagram is converted into audio signal using local weighted summation strategy.The application uses U-Net deep learning model and local weighted summation strategy to repair missing audio, and the repair effect is good, and the calculation cost is lower, suitable for the repair of audio missing situation in short audio.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of audio processing, and particularly relates to an automatic missing audio repairing method based on a U-Net model. BACKGROUND

[0002] In the process of audio transmission and processing, some side effects such as impulse noise, clipping and audio gap often occur, all of which are collectively referred to as audio data distortion. Since such distortion often occurs in audio transmission, many targeted reconstruction methods have been developed. Traditional audio recovery schemes are generally based on sparse representation (SR), which has been proven to be able to successfully model audio or waveform signals and solve the audio repairing framework. However, this method is relatively complex to implement, and the effect has already been unable to improve significantly. In recent years, with the development of technology, the field of deep neural networks has developed rapidly, and this technology has shown great advantages in many fields such as image synthesis, audio synthesis and image repair. In the field of image repair, many different neural networks have been used and ideal repair effects have been achieved. However, the effect of these repair methods still needs to be improved, especially the inaccuracy of repair details and the poor repair of high-frequency parts. SUMMARY

[0003] The application aims to provide an automatic missing audio repairing method based on a U-Net model.

[0004] The technical solution for achieving the application is as follows: an automatic missing audio repairing method based on a U-Net model, comprising the following steps:

[0005] Step 1, constructing an audio data set: constructing an audio data set containing three kinds of audio: human voice, musical instrument sound and natural sound, and processing all these audios into samples of the same length;

[0006] Step 2, audio segmentation preprocessing: processing all audio samples into three parts of fixed length, from left to right, which are left context part, gap part and right context part; wherein the left and right sides are collectively referred to as context parts, and the lengths are the same;

[0007] Step 3, generating a spectrum graph: using short-time Fourier transform to perform form conversion on the preprocessed audio samples, and converting all the samples into spectrum graphs;

[0008] Step 4, model training: taking the generated spectrum graph as input, and using the U-Net model to learn it; the encoder in the model learns the context part, and converts all the input contexts into latent representations of feature descriptions; and the decoder combines a complete spectrum graph according to the feature information of the obtained spectrum graph; the skip connection transmits the gradient in the graph by connecting the encoder and the decoder;

[0009] Step 5: Receive the spectrogram, integrate and transform the spectrogram in the time domain using a local weighted summation strategy to generate the final repaired audio signal.

[0010] Compared with the existing technology, the present invention has the following significant advantages: 1) It realizes automatic restoration of missing audio through the deep learning method of U-Net based on jump connection, which solves the problems of inaccurate repair of details and poor repair of high-frequency parts when using traditional methods; 2) It uses a local weighted summation strategy instead of the common Griffin-Lim method and inverse short-time Fourier transform, which has faster iteration speed and better listening experience of the converted audio. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 This is a flowchart of the missing audio automatic repair method based on the U-Net model of the present invention.

[0012] Figure 2 This is a graph showing the reconstruction comparison results of a random instrument audio signal obtained by the missing audio automatic repair method based on the U-Net model described in an embodiment of the present invention.

[0013] Figure 3 This is a graph showing the comparison results of the reconstruction of pure tone audio signals obtained using the missing audio automatic repair method based on the U-Net model described in an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention is further described in detail below with reference to the accompanying drawings and examples.

[0015] Combine Figure 1 The present invention provides a method for automatically repairing missing audio based on a U-Net model, comprising the following steps:

[0016] Step 1. Build an audio dataset: Build an audio dataset containing three types of audio: human voice, instrumental sound, and natural sound, and process all of these audios into samples of the same length;

[0017] Step 2: Audio segmentation preprocessing: All audio samples are processed into three parts of fixed length, from left to right: the left context part, the gap part, and the right context part. The left and right sides are collectively referred to as the context part, and they have the same length.

[0018] Step 3: Generate spectrogram: Use short-time Fourier transform to transform the pre-processed audio samples into spectrograms.

[0019] Step 4: Model training: The generated spectrogram is used as input and learned using the U-Net model. The encoder in the model learns the context and converts all input context into a potential representation of feature descriptions. The decoder combines the complete spectrogram based on the acquired feature information of the spectrogram. Skip connections propagate gradients in the graph by connecting the encoder and decoder.

[0020] Step 5: Receive the spectrogram, integrate and transform the spectrogram in the time domain using a local weighted summation strategy to generate the final repaired audio signal.

[0021] Furthermore, in one embodiment, in step 2, the audio is processed into three parts of fixed length: a middle gap part and left and right context parts, specifically:

[0022] The samples in the NSynth dataset are processed into short monophonic 1-second audio signals and resampled at 16kHz to ensure the uniform format of the samples. The NSynth dataset is a mixed audio dataset that includes 305,979 short musical notes from 1,006 instruments, each with its own unique pitch, timbre, and envelope.

[0023] The audio signal of one second is divided into 5120 samples, and the part less than one second is automatically discarded. In a 320ms sample, the length of the context part is 128ms and the length of the interval part is 64ms.

[0024] Furthermore, in one embodiment, step 3 uses short-time Fourier transform to convert the pre-processed audio samples into spectrograms, specifically:

[0025] Set a short-time Fourier transform with a hop count of 128 samples, a Hanning window function, a window size of 1024, and a sample length of 512.

[0026] Furthermore, in one embodiment, in step 4, the U-Net model is used to learn the spectrogram to automatically repair the missing audio, specifically:

[0027] Encoder: The encoder extracts spectrogram features from the input data through six convolutional layers. Each convolutional layer is followed by a batch normalization layer and a Leaky ReLU layer. Two spectrograms of the same shape are treated as two channels, and the input context is downsampled through all convolutional layers. The encoder module learns how to analyze and blend the two channel features, transforming all input context into a latent representation describing the features. This encoder module design improves model training efficiency.

[0028] Decoder: The purpose of the decoder is to combine the complete spectrogram based on the feature information of the acquired spectrogram. The decoder module accepts the output of the encoder module and consists of six deconvolution layers. After each deconvolution layer, there is a batch normalization layer and a Leaky ReLU layer to improve the performance of the deconvolution layer.

[0029] Skip connections: Skip connections typically connect the corresponding layers of the encoder module and the corresponding layers of the decoder module to help the decoder better recover more useful detailed information of the spectrogram, that is, the spectrogram. Skip connections connect the corresponding layers of the convolutional layer and the deconvolution layer in the encoder module and the decoder module. All of these connections can effectively expand the depth of the deep neural network model. In addition, skip connections can expand the network's receptive range, thereby enhancing the performance of the network model. In addition, skip connections can also be mathematically explained to accelerate network convergence:

[0030] x (i+1) =ω(G(x (i) )+x (N+1-i)

[0031] Where x (i+1) is the feature map of convolution layer i+1, x (i) is the feature map of convolution layer i, ω is the nonlinear activation function Leaky ReLUs, G is the function of convolution layer i, and N is the number of convolution layers.

[0032] Furthermore, in one embodiment, the loss function defined in the U-Net model in step 4 is specifically:

[0033] The goal of the network training process is to continuously reduce the total loss of the model. During training, we rely on a loss function to optimize the network, which can be optimized together using the Adam optimizer. We briefly describe the loss function as follows:

[0034] 1) Reconstruction loss: Normalized mean square error (NMSE):

[0035]

[0036] in,

[0037]

[0038]

[0039] Where N is the number of samples, X is the training sample set, S g is the initial audio gap, S g ' is the repaired audio gap part.

[0040] NMSE usually finds the most obvious differences between data. If the NMSE of the original data and the restored data in the model is very low, it means that it performs well in both space and time.

[0041] 2) Total loss: the sum of the reconstruction loss and the regularization term that controls the weights during training:

[0042]

[0043] Where, L total is the total loss of the model, L nmse is the reconstruction loss, λ is the regularization parameter of the model, ω i is the weight of the model.

[0044] Furthermore, in one embodiment, in step 5, the reconstructed spectrogram is converted into an audio signal. The local weighted summation strategy adopted by the present invention is specifically as follows:

[0045] The local weighted summation strategy directly iterates each bin in the time-frequency domain. This strategy is proven to be about six times faster than the 60 iterations of the Griffin-Lim method on a CPU, and is also less computationally intensive than the common inverse short-time Fourier transform. After transforming the spectrogram using the weighted summation strategy, the reconstructed audio signal in the gap is L' g , the reconstructed audio signals of the left and right context parts are L'1 and L'2 respectively, and all these parts are combined into the complete audio signal L'.

[0046] The spectrum corresponding to the complete audio after repair is as follows Figure 2 and Figure 3 As shown in the figure, it can be seen that in the high-frequency part, the method of the present invention has a better effect on the details of the missing audio.

[0047] In summary, the U-Net model of the present invention automatically repairs the missing parts in the audio, has good repair effect, and has low computational cost, and is suitable for repairing audio missing parts in short audio.

Claims

1. A method for automatically repairing missing audio based on a U-Net model, characterized in that: The following steps are involved: Step 1: Construct an audio dataset: Construct an audio dataset containing three types of audio: human voice, instrument sound, and natural sound, and process these audios into samples of the same length; Step 2: Audio segmentation preprocessing: All audio samples are processed into three parts of fixed length, from left to right: the left context part, the gap part, and the right context part. The left and right sides are collectively referred to as the context part, and they have the same length. Step 3: Generate spectrogram: Use short-time Fourier transform to transform the pre-processed audio samples into spectrograms. Step 4: Model training: Take the generated spectrogram as input and use the U-Net model to learn it; The encoder in the model learns the context and converts all input context into a potential representation of the feature description; the decoder combines the complete spectrogram based on the feature information of the acquired spectrogram; the skip connection propagates the gradient in the graph by connecting the encoder and decoder; Step 5: Receive the spectrogram, integrate and transform the spectrogram in the time domain using a local weighted summation strategy to generate the final repaired audio signal.

2. The method for automatically repairing missing audio based on the U-Net model according to claim 1, characterized in that As described in step 2, all audio samples are processed into three parts of fixed length, which are the left context part, the gap part, and the right context part from left to right. Specifically: Step 2-1: Cut the audio in the data set containing the three types of audio into audio of the same length, and record it as the original audio signal L; Step 2-2: Process the audio signal L and divide it into three parts: the gap part Lg in the middle, the left context part denoted as L1, and the right context part denoted as L2.

3. The method for automatically repairing missing audio based on the U-Net model according to claim 2, characterized in that: Step 3 uses short-time Fourier transform to transform the pre-processed audio samples into spectrograms, specifically: Step 3-1. Set the short-time Fourier transform parameters: the number of hops is 128 samples, the window function is the Hanning window, the window size is 1024, the sample length is 512, and the short-time Fourier transform formula is as follows: Where x(t) is the input signal function, ω(t-τ) is the window function, τ is the time axis, ω is the frequency axis, and t is the sampling time point; Step 3-2: Apply short-time Fourier transform to the audio contexts L1 and L2 and the gap portion Lg to convert the audio signal into a spectrogram; the spectrogram representations of the left and right contexts after conversion are S1 and S2 respectively, and the gap portion is Sg.

4. The method for automatically repairing missing audio based on the U-Net model according to claim 3, characterized in that: In step 4, the generated spectrogram is used as input and learned using the U-Net model, specifically: The spectrum graph generated by the process of steps 3-1 and 3-2; the U-Net model consists of an encoder, a decoder, and a skip connection part, specifically: The encoder module first extracts features from the input spectrogram through six convolutional layers. After each convolutional layer, a batch normalization layer and a nonlinear activation function Leaky ReLU layer are sequentially executed. Inputs S1 and S2 with the same characteristics are treated as two channels, and the input context is partially downsampled through all convolutional layers. The encoder module mixes the features of the two channels and converts all input context into a potential representation of the feature description. The decoder module accepts the output of the encoder module and consists of six deconvolution layers; after each deconvolution layer, there is a batch normalization layer and a nonlinear activation function Leaky ReLUs layer; The decoder module generates a spectrogram of the gap part based on the characteristic information of the acquired spectrogram, and combines them to generate the spectrogram of the final audio; The skip connection connects the corresponding layer of the encoder module and the corresponding layer of the decoder module. The formula is as follows: x(i+1)=ω(G(x(i))+x(N+1-i) where x(i+1) is the feature map of convolutional layer i+1, x(i) is the feature map of convolutional layer i, ω is the nonlinear activation function Leaky ReLUs, G is the function of convolutional layer i, and N is the total number of convolutional layers.

Images