Audio restoration method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides an audio restoration method and device, equipment, a storage medium and a program product, and the method comprises the steps: obtaining lossless audio data and damaged audio data, carrying out the damage simulation processing of the lossless audio data, and obtaining simulated audio data, training a teacher network model based on the lossless audio data and the simulated audio data; inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training the student network model based on the damaged audio data and the target audio data; and obtaining a to-be-processed audio signal, performing frequency domain transformation processing on the to-be-processed audio signal to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal. According to the scheme, the student network model deployed at the mobile terminal is obtained based on teacher network model training, and the real-time audio repairing requirement is met.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of audio technology, and in particular to an audio repair method, apparatus, device, storage medium, and program product. Background Art

[0002] With the continuous advancement of Internet technology, network bandwidth has been significantly improved, providing a stable and high-speed data transmission channel for real-time calls. This allows audio data to be transmitted more smoothly in the network, reducing delays and freezes. Among them, in real-time audio call scenarios, affected by the sound reception conditions and software and hardware equipment, the sound signal received by the user at the data receiving end will be damaged, such as high-frequency loss, word loss, low-frequency harmonic damage, etc., which will affect the clarity of the sound and the integrity of the information.

[0003] The deep learning-based audio repair algorithm used in related technologies, on the one hand, relies on data-driven model training. In the training process of the deep learning model, data pairs before and after the damage are required. However, real data pairs are often not available in actual scenarios, and simulation data cannot fully represent the state of the application scenario, and cannot improve the audio repair ability of the deep learning model, resulting in poor audio repair effects. On the other hand, it is necessary to increase the model complexity to improve the modeling and generalization capabilities of the deep learning model, so as to improve the real-time processing of the model trained with simulation data in the target scenario with better results. However, the model has high complexity and high computing power requirements for computing devices, and cannot be adapted to applications in real-time scenarios on mobile terminals. Summary of the invention

[0004] The embodiments of the present application provide an audio repair method, apparatus, device, storage medium and program product, which solve the problems in the related art that the training of deep learning models cannot obtain real data pairs, resulting in poor audio repair effect, and the deep learning models with higher complexity cannot adapt to the application in real-time scenarios on mobile terminals. The teacher network model that has been trained is implemented to provide real and effective data pairs for training student network models with lower computational complexity, which is beneficial to improving the audio repair ability of the student network model and ensuring the audio repair effect. At the same time, it can be flexibly deployed in real-time scenarios on mobile terminals with lower computing power requirements to meet the needs of real-time audio repair.

[0005] In a first aspect, an embodiment of the present application provides an audio repair method, the method comprising:

[0006] Acquire lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data;

[0007] Inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training the pre-constructed student network model based on the damaged audio data and the target audio data;

[0008] Acquire an audio signal to be processed, perform frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and perform time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

[0009] In a second aspect, an embodiment of the present application further provides an audio repair device, the device comprising:

[0010] A teacher model training module is configured to obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data;

[0011] A student model training module is configured to input the damaged audio data into the trained teacher network model to obtain target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data;

[0012] The audio signal repair module is configured to obtain an audio signal to be processed, perform frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and perform time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

[0013] In a third aspect, an embodiment of the present application further provides an audio repair device, the device comprising:

[0014] one or more processors;

[0015] a storage device configured to store one or more programs,

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the audio repair method described in the embodiment of the present application.

[0017] In a fourth aspect, an embodiment of the present application further provides a non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to execute the audio repair method described in the embodiment of the present application when executed by a computer processor.

[0018] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor of the device reads and executes the computer program from the computer-readable storage medium, so that the device performs the audio repair method described in the embodiment of the present application.

[0019] In an embodiment of the present application, lossless audio data and damaged audio data are obtained, damage simulation processing is performed on the lossless audio data to obtain simulated audio data, and a pre-constructed teacher network model is trained based on the lossless audio data and the simulated audio data; the damaged audio data is input into the trained teacher network model to obtain target audio data, and a pre-constructed student network model is trained based on the damaged audio data and the target audio data; an audio signal to be processed is obtained, a frequency domain transform processing is performed on the audio signal to be processed to obtain a frequency domain signal, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal, and the frequency domain repair signal is transformed in the time domain to obtain an audio repair signal. In the above scheme, since the teacher network model does not need to be deployed in the actual scene, it is not restricted by the model calculation complexity and delay during design. By training the pre-built teacher network model based on lossless audio data and simulated audio data, an audio repair model with high calculation complexity and large parameter scale can be obtained. In addition, by inputting the damaged audio data into the trained teacher network model to obtain the target audio data, audio repair data matching the damaged audio data can be obtained, providing real and effective data pairs for training the student network model with lower calculation complexity, which is beneficial to improving the audio repair ability of the student network model and ensuring the audio repair effect. In addition, the student network model is used to repair the audio signal to be processed, which can be flexibly deployed in real-time scenarios on mobile terminals with lower computing power requirements to meet the needs of real-time audio repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 A flowchart of an audio repair method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a simulation link for processing and obtaining simulated audio data provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of the structure of a teacher network model provided in an embodiment of the present application;

[0023] Figure 4 A schematic diagram of the structure of a first-stage model in a teacher network model provided in an embodiment of the present application;

[0024] Figure 5A schematic diagram of the structure of a feature extraction layer in a time series modeling layer provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of the structure of a channel attention layer in a temporal modeling layer provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of an energy correction module in a one-stage model provided in an embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of a two-stage model provided in an embodiment of the present application;

[0028] Fig. 9 A schematic diagram of the structure of a fusion module in a teacher network model provided in an embodiment of the present application;

[0029] Fig.10 A schematic diagram of the structure of a student network model provided in an embodiment of the present application;

[0030] Fig.11 A flowchart of an audio restoration method including a training process of a student network model provided in an embodiment of the present application;

[0031] Fig.12 A schematic diagram of a process of training a student network model based on regulated knowledge distillation provided in an embodiment of the present application;

[0032] Fig.13 A flowchart of an audio restoration method including a process of fusing and determining a target frequency domain restoration signal provided in an embodiment of the present application;

[0033] Fig.14 A flowchart of an audio restoration method including a process of performing low-frequency energy detection on an audio signal to be processed provided in an embodiment of the present application;

[0034] Fig.15 A flowchart of an audio restoration method including a process of determining a cutoff frequency of frequency domain information provided in an embodiment of the present application;

[0035] Fig.16 A schematic diagram of a process of performing audio restoration processing on an audio signal to be processed provided by an embodiment of the present application;

[0036] Fig.17 A structural block diagram of an audio repair device provided in an embodiment of the present application;

[0037] Fig.18 A schematic diagram of the structure of an audio repair device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0038] The embodiments of the present application are further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the embodiments of the present application, rather than to limit the embodiments of the present application. It should also be noted that, for ease of description, only parts related to the embodiments of the present application are shown in the accompanying drawings, rather than all structures.

[0039] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0040] The audio repair method provided in the embodiment of the present application can use lossless audio data and simulated audio data to train a teacher network model with high computational complexity and large parameter scale, and use the teacher network model to process the damaged audio data of the real scene to obtain the target audio data, and form a data pair before and after the repair with the damaged audio data, and use the data pair to train the student network model with low computational complexity. The student network model is suitable for mobile terminal deployment and can effectively perform frequency domain signal repair. Relevant application scenarios may include: audio calls, audio live broadcasts, etc. The several application scenarios listed above are only exemplary and explanatory. In actual applications, the audio processing method can also be used in audio repair in other scenarios, and the embodiment of the present application does not limit this. The audio processing method, device, equipment, storage medium and program product provided in the embodiment of the present application are intended to solve the problem that the training of deep learning models in related technologies cannot obtain real data pairs, resulting in poor audio repair effects, and the deep learning models with higher complexity cannot adapt to applications in real-time scenarios on mobile terminals.

[0041] In the audio repair method provided in the embodiment of the present application, the executor of each step can be a computer device, which refers to any electronic device with data calculation, processing and storage capabilities, such as mobile phones, PCs (Personal Computers), tablet computers and other terminal devices, and can also be servers and other devices, which are not limited in the embodiment of the present application.

[0042] Figure 1A flowchart of an audio repair method provided in an embodiment of the present application. In an audio transmission scenario, the data transmitting end collects the audio signal and performs near-end algorithm processing, then encodes it through an audio encoder, and transmits the encoding result to the data receiving end through a physical channel. After the data receiving end receives the encoding result, it decodes the encoding result through an audio decoder to obtain an audio signal, and applies the audio repair method provided in an embodiment of the present application to repair the audio signal to obtain a target audio signal, and then plays the target audio signal. Figure 1 As shown, the audio repair method includes the following steps:

[0043] Step S101, obtaining lossless audio data and damaged audio data, performing damage simulation processing on the lossless audio data to obtain simulated audio data, and training a pre-constructed teacher network model based on the lossless audio data and the simulated audio data.

[0044] Among them, the lossless audio data can be high-quality audio from a public data set, without problems such as high-frequency loss, word loss, and low-frequency harmonic damage. The damaged audio data can be audio data received by a data receiving end in a real call scenario. Due to the influence of the sound reception conditions, software and hardware equipment, there are problems such as high-frequency loss, word loss, and low-frequency harmonic damage. Since it is difficult to obtain data pairs before and after damage in real-time call scenarios, in order to train the teacher network model, the lossless audio data can be subjected to damage simulation processing to obtain simulated audio data, and obtain audio data pairs for training the teacher network model. The simulated audio data can be used as training data, and the lossless audio data can be used as label data. In one embodiment, a simulation link that matches the actual application scenario can be pre-built, please refer to Figure 2 , Figure 2A schematic diagram of a simulation link for processing simulated audio data provided in an embodiment of the present application, the simulation link may include an environment simulation module 101, an acquisition simulation module 102, a near-end processing module 103, and a codec transmission simulation module 104. The environment simulation module 101 uses an environment simulation algorithm to perform noise mixing, echo addition, and environment attenuation processing on the received lossless audio data. For example, the signal-to-noise ratio interval used for noise mixing processing may be [-20, 0], in dB, and the signal-to-echo ratio interval used for echo addition processing may be [-25, 40], in dB. When performing environment attenuation processing, simulated air and room attenuation may be used to weaken the high-frequency part of the audio. The acquisition simulation module 102 uses an acquisition algorithm to process the output result of the environment simulation module 101, which is used to simulate the quantization noise that may be introduced in the analog-to-digital conversion. The near-end processing module 103 uses a near-end processing algorithm to process the output result of the acquisition simulation module 102, which is used to perform echo cancellation, noise reduction, and automatic volume gain, etc., which is consistent with the specific algorithm used in the actual application scenario. The codec transmission simulation module 104 uses an audio codec algorithm and a simulated channel transmission algorithm to process the output result of the near-end processing module to obtain simulated audio data, thereby effectively performing loss simulation processing on lossless audio data to obtain audio data that simulates damage in actual application scenarios.

[0045] In one embodiment, Figure 3 A structural diagram of a teacher network model provided in an embodiment of the present application is shown in FIG. Figure 3 As shown, the teacher network model includes a pre-processing module 201, a one-stage model 202, a two-stage model 203, a fusion module 204 and a post-processing module 205 which are connected in sequence. The pre-processing module 201 and the one-stage module are respectively connected to the fusion module 204 in a jump step. The pre-processing module 201 is used to receive the damaged audio data and perform frequency domain conversion processing on the damaged audio data to obtain the first frequency domain data. The one-stage model 202 is a self-encoding structure for repairing the amplitude spectrum of the first frequency domain data. The two-stage model 203 is a vocoder structure for further processing the amplitude spectrum of the processing result output by the one-stage model to obtain natural phase information. The fusion module 204 is used to fuse the first frequency domain data, the first processing result output by the one-stage model 202 and the second processing result output by the two-stage model 203 to obtain the second frequency domain data, aiming to reduce the degradation of the two-stage model to the low-frequency original signal and retain the amplitude and phase of the original audio signal when the original audio signal has a large energy. The post-processing module 205 is used to perform time domain conversion processing on the second frequency domain data to obtain target audio data, and output the target audio data.

[0046] Optional, Figure 4 A schematic diagram of the structure of a stage model in a teacher network model provided in an embodiment of the present application, such as Figure 4 As shown, the one-stage model uses a two-dimensional non-causal autoencoder structure. The one-stage model includes an encoder module, a temporal modeling layer, a decoder module and an energy correction module connected in sequence. The encoder module includes multiple encoders, and the decoder module includes multiple decoders. The encoder module is connected to the decoder module in a jump step, so that the encoder and decoder at the same level can directly interact with each other. The encoder includes a first convolution layer, several first feature optimization layers, a first channel frequency normalization layer and a first activation layer connected in sequence. The decoder includes several second feature optimization layers, a transposed convolution layer, a second channel frequency normalization layer and a second activation layer connected in sequence. The first feature optimization layer includes a first point convolution layer, a hole convolution layer and a second point convolution layer connected in sequence, and the structure of the second feature optimization layer is the same as that of the first feature optimization layer. The temporal modeling layer includes several feature extraction layers set in parallel and channel attention layers connected to the several feature extraction layers respectively. The first optimization layer in the encoder and the second feature optimization layer in the encoder are only set at the layers closest to the input and output, so that the position layer close to the input and output has a higher complexity, which is conducive to modeling and restoring the original features. The time series modeling layer can be used to capture time series information and purify features. The energy correction module can further perform energy correction on the repair residual information output by the decoder module to prevent the problem of three-frequency imbalance in the repair residual information after processing.

[0047] In one embodiment, Figure 5 A structural diagram of a feature extraction layer in a time series modeling layer provided in an embodiment of the present application, such as Figure 5 As shown, the feature extraction layer includes a third point convolution layer, a third activation layer, a first channel normalization layer, a first depth convolution layer, a first matrix multiplication layer, a fourth activation layer, a second channel normalization layer, a fourth point convolution layer and a first matrix addition layer connected in sequence, and the first matrix addition layer is used to receive feature information input to the third point convolution layer and feature information output by the fourth point convolution layer for processing. In addition, the feature extraction layer also includes a fifth activation layer, a third channel normalization layer, a second depth convolution layer and a sixth activation layer connected in sequence, and the fifth activation layer is connected to the third point convolution layer.

[0048] Optional, Figure 6 A schematic diagram of the structure of a channel attention layer in a temporal modeling layer provided in an embodiment of the present application, such as Figure 6As shown, the channel attention layer includes an intermediate processing module, a seventh activation layer, a second matrix multiplication layer and a linear weighting layer connected in sequence, and the linear weighting layer is used to receive feature information input to the intermediate processing module and feature information output by the second matrix multiplication layer for processing. The intermediate processing module includes a plurality of intermediate processing layers, which include a first pooling layer, a second matrix addition layer, an eighth activation layer and a first matrix splicing layer connected in sequence, and also include a second matrix splicing layer and a second pooling layer arranged in parallel with the first pooling layer, the second matrix splicing layer is connected to the second matrix multiplication layer, and the second pooling layer is connected to the second matrix addition layer.

[0049] In one embodiment, Figure 7 A structural schematic diagram of an energy correction module in a one-stage model provided in an embodiment of the present application, the energy correction module includes a third matrix addition layer, a second convolutional layer and a third convolutional layer connected to the third matrix addition layer, a third matrix multiplication layer connected to the second convolutional layer and the third convolutional layer, a fourth convolutional layer connected to the third matrix multiplication layer, and a fourth matrix multiplication layer connected to the fourth convolutional layer, the third matrix addition layer is used to receive input feature information output by the preprocessing module and predicted feature information output by the encoder module, the second matrix multiplication layer is used to receive output information of the fourth convolutional layer and predicted feature information output by the encoder module, and output audio repair information.

[0050] Optional, Figure 8 A schematic diagram of a two-stage model provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the two-stage model includes a first convolution module, a fourth matrix addition layer, a second convolution module, a fifth matrix addition layer, and a short-time Fourier inverse transform module connected in sequence. The fourth matrix addition layer is used to process the feature information input to the first convolution module and the spectral feature information output by the first convolution module, and the fifth matrix addition layer is used to process the spectral feature information input to the second convolution module and the spectral feature information output by the second convolution module. The short-time Fourier inverse transform module is used to output a time domain signal. In addition, the first convolution module includes a plurality of convolution blocks connected in sequence, and the convolution block includes a third depth convolution layer, a fifth point convolution layer, and a sixth point convolution layer connected in sequence, and the second convolution module has the same structure as the first convolution module.

[0051] In one embodiment, Fig. 9 A structural diagram of a fusion module in a teacher network model provided in an embodiment of the present application, such as Fig. 9As shown, the fusion module includes a logarithmic calculation module, a first scale transformation module, a multi-scale smoothing module, a dry-wet ratio mixed calculation module and an index evaluation and screening module connected in sequence. Among them, the logarithmic calculation module is used to receive the first spectrum information output by the preprocessing module, the second spectrum information obtained by converting the first time domain signal output by the two-stage model, and the third spectrum information obtained by converting the second time domain signal output by the one-stage model, and the second spectrum information and the third spectrum information are respectively calculated with the first spectrum information to obtain the first logarithmic difference information and the second logarithmic difference information. For example, after the first spectrum information, the second spectrum information and the third spectrum information are respectively converted into the first logarithmic information, the second logarithmic information and the third logarithmic information with the base of 10, the second logarithmic information and the third logarithmic information are respectively subtracted from the first logarithmic information to obtain the first logarithmic difference information and the second logarithmic difference information. The first scale transformation module is used to control the transition band width and starting point of the fusion, and the multi-scale smoothing module is used to smoothly fuse the first logarithmic difference information and the second logarithmic difference information to obtain the target difference information to avoid repeated jumps in the fusion area. The dry-wet ratio hybrid calculation module is used to fuse the third spectrum information and the target difference information according to multiple available dry-wet ratios to obtain multiple fusion results. Finally, the multiple fusion results are evaluated by the objective evaluation index through the index evaluation screening module, and the one with the best comprehensive index is selected as the final fusion result output. Optionally, the objective evaluation index can adopt SISNR (Signal-to-Interference and Noise Ratio), DNSMOS (Deep Noise Suppression and MOS-based Objective Metric, deep noise suppression and speech quality objective evaluation index), etc., which are not limited in this application.

[0052] In one embodiment, the lossless audio data and the simulated audio data can be used to train the pre-built teacher network model under the generative adversarial framework until the teacher network model reaches the preset loss convergence condition and stops training. The generative adversarial framework may include a generator and a discriminator, the generator may be a convolutional neural network, and the discriminator may be composed of a multi-scale frequency domain discriminator and a multi-cycle time domain discriminator. The multi-scale frequency domain discriminator uses different short-time Fourier transform points to transform the input signal into the frequency domain, obtains the frequency domain amplitude spectrum at different resolutions, and then divides the obtained frequency domain amplitude spectrum into bands by a preset number of bands, and extracts features from the banded frequency spectrum amplitude spectrum through several two-dimensional convolutional layers, and finally outputs the feature map of each layer and the final score. The multi-cycle time domain discriminator analyzes the signal at different time scales to capture the long-term and short-term features in the audio to identify and score the audio. The multi-cycle time domain discriminator slices the input signal according to different time scales, and then extracts features from the sliced ​​time domain signal through several two-dimensional convolutional layers, and outputs the feature map of each layer and the final score. In the generative adversarial training process of the teacher network model, the complexity of the discriminator can be adjusted according to different training stages. For example, the channels of each layer of the multi-scale frequency domain discriminator are set to 32, the number of layers is 5, the number of bands is set to 5, the number of short-time Fourier transform points is set to [2048, 1024, 512], and the channels of each layer of the multi-period time domain discriminator are set to [32, 128, 512, 1024], and the time scale is set to [2, 3, 5, 7, 11]. In addition, the generative adversarial loss function, the time domain loss function and the multi-scale spectral loss function can be used in the generative adversarial training process. The formulas of each loss function are as follows:

[0053] Generator loss function: L G =E[Relu(1-D(G(x)))],

[0054] Discriminator loss function: L D =E[Relu(1+D(G(x)))]+E[Relu(1-D(y))],

[0055] Discriminator feature loss function: L Df =E[|D feat (y)-D feat )G(x)0|1],

[0056] Time domain loss function: L T =|yG(x)|1,

[0057] Multi-scale amplitude spectrum loss function:

[0058]

[0059] Compatibility loss function:

[0060] L c =|STFT(G(x))-STFT(ISTFT(G(x)))|1,

[0061] The total loss function formula is:

[0062]

[0063] In the above formula, x represents the signal before processing, y represents the training reference signal, G(x) represents the signal after processing, E(*) represents the averaging function, Relu(*) is the activation function, D(*) represents the discriminator score output, and D feat (*) indicates the discriminator feature output, Mag(*) indicates the amplitude spectrum, STFT(*) indicates the short-time Fourier transform, ISTFT(*) indicates the inverse short-time Fourier transform, subscript n indicates the number of short-time Fourier transform points, which is 320, 512, or 2048 according to empirical values, |*|1 indicates the 1-norm, and |*|2 indicates the 2-norm. a, b, c, d

[0064] According to empirical values, they are 1.0, 100, 0.5, and 0.1 respectively.

[0065] Step S102: input the damaged audio data into the trained teacher network model to obtain the target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data.

[0066] Among them, by inputting the damaged audio data into the trained teacher network model, the target audio data can be obtained. The target audio data is repaired by using the teacher network model to repair the damaged audio data in the real call scenario, and the audio data pair used to train the student network model is obtained. The damaged audio data can be used as training data, and the target audio data can be used as label data.

[0067] In one embodiment, Fig.10 A schematic diagram of a student network model provided in an embodiment of the present application is shown in FIG. Fig.10As shown, the student network model includes a scale transformation module 301, a first encoder module 302, a first decoder module 303, a time series modeling layer 304, a second encoder module 305, a second decoder module 306 and an energy correction module 307 connected in sequence, the first encoder module 302 jumps to the second decoder module 306, the first decoder module 303 jumps to the second encoder module 305, the scale transformation module 301 is used to receive the frequency domain signal and perform feature compression processing on the frequency domain signal, and the energy correction module is used to fuse the frequency domain signal and the output result of the second decoder module 306 to obtain a frequency domain repair signal, and output the frequency domain repair signal. Among them, the first encoder module 302 and the second decoder module 306 respectively include a plurality of first encoders and second decoders, and the first encoder module 302 jumps to the second decoder module 306 for direct information interaction between the codecs of the same level. The first decoder module 303 and the second encoder module 305 respectively include a plurality of first decoders and second encoders, and the first decoder module 303 jumps to the second encoder module 305 for direct information interaction between the codecs of the same level. Optionally, the timing modeling layer can be a gated cyclic unit layer. The energy correction module can refer to the specific settings of the energy correction model of the first-stage model in the aforementioned embodiment, which will not be described in detail in this application. It should be noted that in order to save the amount of calculation, after the frequency domain signal is input into the student network model, it is first subjected to feature compression by the scale transformation module 301, for example, the Mel scale can be used. Optionally, each codec can be composed of a layer of one-dimensional convolution and a layer of activation function, and the time axis is not compressed during the serial processing process, and the frequency axis is subjected to the encoding and decoding processing of first expansion and then compression through channel changes. At the same time, the timing modeling layer 304 can adopt the GRU layer, which can complete the modeling requirements of historical information while introducing less overhead. Finally, similar to the teacher network model, the student network model is also provided with an energy correction module to perform energy correction on the repair residual information output by the second decoder module 306.

[0068] In one embodiment, the damaged audio data and the target audio data can be used to train the pre-constructed student network model under the generative adversarial framework until the student network model reaches the preset loss convergence condition and stops training. The specific settings of the discriminator and the loss function in the generative adversarial framework can refer to the configuration of the teacher network model in the aforementioned embodiment, and this application will not go into details here. In the generative adversarial training process of the student network model, the complexity of the discriminator can be adjusted according to different training stages. For example, the channels of each layer of the multi-scale frequency domain discriminator are set to 32, the number of layers is 3, the number of sub-bands is set to 4, the number of short-time Fourier transform points is set to [2048,512,320], the channels of each layer of the multi-period time domain discriminator are set to [32,128], and the time scale is set to [2,3,5,7,11].

[0069] Step S103, obtaining an audio signal to be processed, performing frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

[0070] Among them, the frequency domain transformation processing is used to convert the audio signal to be processed from the original time domain signal to the frequency domain signal, which can be specifically done by using methods such as short-time Fourier transform, discrete cosine transform or wavelet transform, which are not limited in this application. In addition, the time domain transformation processing is used to convert the frequency domain repair signal from the original frequency domain signal to the time domain signal to obtain the audio repair signal, which can be specifically done by using methods such as short-time Fourier inverse transform, discrete cosine quasi-transform or wavelet inverse transform, which are not limited in this application. The student network model is a model with low computational complexity and can be deployed at the data receiving end of a real-time call scenario for audio repair of the audio signal to be processed.

[0071] For example, the frequency domain transform process may adopt a short-time Fourier transform method.

[0072] Among them, the formula of short-time Fourier transform is as follows:

[0073]

[0074] Wherein, x[k] represents the time series of the audio signal to be processed, w[*] represents the window function, for example, the Hanning window is used in this embodiment, X[f,n] ​​represents the frequency domain signal, f and n are the indexes of the frequency and time frame respectively. N represents the number of Fourier transform points, in order to obtain a better phase estimation effect, a long analysis window is used in this embodiment, for example, N=320.

[0075] For example, the frequency domain transform process may adopt a short-time inverse Fourier transform method.

[0076] Among them, the formula for short-time inverse Fourier transform is as follows:

[0077]

[0078] Wherein, x[k] represents the time series of the audio repair signal, w[*] represents the window function, for example, the Hanning window is used in this embodiment, X[f,n] ​​represents the frequency domain repair signal, f and n are the indexes of the frequency and time frame respectively. N represents the number of Fourier transform points, in order to obtain a better phase estimation effect, a long analysis window is used in this embodiment, for example, N=320.

[0079] In the above, by acquiring lossless audio data and damaged audio data, performing damage simulation processing on the lossless audio data to obtain simulated audio data, and training a pre-constructed teacher network model based on the lossless audio data and the simulated audio data; inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training a pre-constructed student network model based on the damaged audio data and the target audio data; acquiring an audio signal to be processed, performing frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal. In the above scheme, since the teacher network model does not need to be deployed in the actual scene, it is not restricted by the model calculation complexity and delay during design. By training the pre-built teacher network model based on lossless audio data and simulated audio data, an audio repair model with high calculation complexity and large parameter scale can be obtained. In addition, by inputting the damaged audio data into the trained teacher network model to obtain the target audio data, audio repair data matching the damaged audio data can be obtained, providing real and effective data pairs for training the student network model with lower calculation complexity, which is beneficial to improving the audio repair ability of the student network model and ensuring the audio repair effect. In addition, the student network model is used to repair the audio signal to be processed, which can be flexibly deployed in real-time scenarios on mobile terminals with lower computing power requirements to meet the needs of real-time audio repair.

[0080] Fig.11 A flowchart of an audio restoration method including a training process of a student network model provided in an embodiment of the present application. Fig.11 As shown, the following steps are included:

[0081] Step S201, obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data.

[0082] Step S202: input the damaged audio data into the trained teacher network model to obtain the target audio data, fuse the target audio data with the damaged audio data according to the set dry-wet ratio factor to obtain the reference audio data, the dry-wet ratio factor is determined based on the number of trained cycles, the total number of training cycles and the minimum value of the preset factor, and train the pre-constructed student network model based on the damaged audio data and the reference audio data.

[0083] Among them, please refer to Fig.12 , Fig.12 A schematic diagram of a process of training a student network model based on regulated knowledge distillation provided in an embodiment of the present application is shown in FIG. Fig.12 As shown, the mediation-type knowledge distillation is specifically divided into a first stage and a second stage. In the first stage, the simulation processing module 401 performs damage simulation processing after receiving the lossless audio data and outputs the simulated audio data to the teacher model training module 402. The teacher model training module 402 trains the pre-constructed teacher network model based on the simulated audio data and the lossless audio data. In the second stage, the teacher model training module processes the damaged audio data through the trained teacher network model, and outputs the target audio data to the dry-wet adjustment module 403. After receiving the damaged audio data and the target audio data, the dry-wet adjustment module 403 fuses the target audio data with the damaged audio data according to the set dry-wet ratio factor to obtain the reference audio data, and outputs the reference audio data to the student model training module 404. The student model training module 404 trains the pre-constructed student network model based on the damaged audio data and the reference audio data.

[0084] Among them, the dry-wet ratio factor can be used to control the degree of influence of the target audio data output by the teacher network model on the student network model. The larger the dry-wet ratio factor, the larger the fusion weight corresponding to the target audio data, and the greater the degree of influence of the target audio data output by the teacher network model on the student network model. The dry-wet ratio factor can be a set fixed value, or it can be set based on the set minimum dry-wet ratio, the total number of training cycles, and the number of trained cycles, which is not limited in this application. In one embodiment, a dry-wet ratio factor that varies with the training cycle can be set, and the calculation formula of the dry-wet ratio factor is as follows:

[0085]

[0086] in, is the dry-wet ratio factor used in the current training cycle, is the preset factor minimum value, Epoch cur Epoch is the number of training cycles. maxOf course, there may be other ways to set the dry-wet ratio factor, for example, based on the preset minimum factor value during the training process, the dry-wet ratio factor is gradually increased by a fixed amplitude, for example, the first and second training cycles are 40%, the third and fourth training cycles are 45%, and so on, which is not limited in this application.

[0087] In one embodiment, the relevant formula for fusing the target audio data with the damaged audio data according to the set dry-wet ratio factor to obtain the reference audio data is as follows:

[0088]

[0089] in, is the dry-wet ratio factor, x is the damaged audio data, Teacher(x) is the target audio data output by the teacher network model, For reference audio data.

[0090] Step S203, obtaining an audio signal to be processed, performing frequency domain transform processing on the audio signal to be processed to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transform processing on the frequency domain repair signal to obtain an audio repair signal.

[0091] As mentioned above, by setting a gradual dry-wet ratio factor, the weight of the target audio data output by the teacher network model is gradually increased during the training process, so that the student network model can gradually learn the audio restoration ability of the teacher network model, and the training is more stable and reliable.

[0092] Fig.13 The present invention provides a flowchart of an audio repair method including a process of integrating and determining a target frequency domain repair signal. Fig.13 As shown, the following steps are included:

[0093] Step S301, obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data.

[0094] Step S302: input the damaged audio data into the trained teacher network model to obtain the target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data.

[0095] Step S303, obtain the audio signal to be processed, perform frequency domain transform processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and determine the target frequency domain repair signal based on the amplitude information of the frequency domain repair signal and the phase information of the frequency domain signal.

[0096] Among them, since the student network model is deployed at the data receiving end, the phase information obtained in its audio encoding and decoding stage is relatively natural and does not need to be repaired. Therefore, the phase information of the original frequency domain signal can be retained, and only the amplitude information of the frequency domain repair signal is used to fuse the amplitude information of the frequency domain repair signal and the phase information of the frequency domain signal to determine the final target frequency domain repair signal.

[0097] Step S304: Perform time domain transformation processing on the target frequency domain repair signal to obtain an audio repair signal.

[0098] As described above, by fusing the amplitude information of the frequency domain repair signal and the phase information of the frequency domain signal to determine the target frequency domain repair signal, the more natural phase information of the signal itself after audio encoding and decoding can be effectively utilized, and the audio damage problem caused by encoding and decoding and transmission can also be alleviated.

[0099] Fig.14 The present invention provides a flowchart of an audio repair method including a process of performing low-frequency energy detection on an audio signal to be processed. Fig.14 As shown, the following steps are included:

[0100] Step S401, obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data.

[0101] Step S402: input the damaged audio data into the trained teacher network model to obtain the target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data.

[0102] Step S403: obtaining an audio signal to be processed, and performing frequency domain transformation on the audio signal to be processed to obtain a frequency domain signal.

[0103] Step S404: divide the frequency domain signal into a preset number of frequency bands, determine the energy values ​​of the multiple frequency bands according to the amplitude information corresponding to the frequency domain signal, subtract the energy values ​​of adjacent frequency bands in the multiple frequency bands to obtain the first energy difference values ​​corresponding to each group of adjacent frequency bands, and subtract the first energy difference value corresponding to each group of adjacent frequency bands from the set preset energy threshold to obtain the second energy difference value.

[0104] Among them, this embodiment can pre-detect the low-frequency energy state of the frequency domain signal to determine its low-frequency damage state. For example, for a 16kHz audio signal, its effective frequency range is 0-8kHz, and the preset number is set to 4, so it can be divided into four frequency bands: 0-2kHz, 2kHz-4kHz, 4kHz-6kHz, and 6kHz-8kHz. Of course, the preset number can be adaptively set by the developer according to the effective frequency range and detection accuracy requirements of the actual application scenario, and this application does not limit it. According to the standard energy distribution of normal audio, the energy difference between the four frequency bands presents a fixed smooth change law. Accordingly, a preset energy threshold corresponding to the energy difference between adjacent frequency bands can be pre-set. Normally, if the energy difference between adjacent frequency bands and the preset energy threshold are less than the set energy tolerance threshold, it can be regarded as the low-frequency band energy being damaged. Therefore, the energy values ​​of the four frequency bands of 0-2kHz, 2kHz-4kHz, 4kHz-6kHz and 6kHz-8kHz can be determined according to the amplitude information corresponding to the frequency domain signal, and the energy values ​​of adjacent frequency bands can be subtracted to obtain three first energy difference values, and each first energy difference value is set with a corresponding preset energy threshold. For example, the preset energy threshold can be 40dB, 100dB and 150dB, and the first energy difference value is subtracted from the preset energy threshold to obtain the second energy difference value, and the second energy difference value is compared with the corresponding energy tolerance threshold. The energy tolerance threshold can be set by reducing the preset energy threshold by a preset amplitude, which is not limited in this application. For example, the preset amplitude is 5dB, etc. Of course, the specific preset energy threshold and energy tolerance threshold can be adaptively set according to the energy distribution law and detection accuracy requirements of the actual audio signal, which is not limited in this application. If the corresponding second energy difference values ​​in each group of adjacent frequency bands are greater than or equal to the set energy tolerance threshold, it can be considered that the low-frequency frequency band energy is not damaged. If the corresponding second energy difference values ​​in each group of adjacent frequency bands are less than the set energy tolerance threshold, it can be considered that the low-frequency frequency band energy is damaged.

[0105] Step S405: When there is a corresponding second energy difference value in each group of adjacent frequency bands that is less than the set energy tolerance threshold, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal, and the frequency domain repair signal is transformed in the time domain to obtain an audio repair signal.

[0106] Among them, if the corresponding second energy difference value in each group of adjacent frequency bands is less than the set energy tolerance threshold, it can be regarded as low-frequency damage, and the subsequent steps are continued to perform high-frequency repair. If the corresponding second energy difference value in each group of adjacent frequency bands is greater than or equal to the set energy tolerance threshold, the attenuation detection of the frequency band energy can be continued to determine the cutoff frequency to determine whether there is a high-frequency loss. If there is a high-frequency loss, the subsequent steps are continued to perform audio repair. If there is no high-frequency loss, there is no need to perform audio repair.

[0107] As described above, by performing low-frequency energy state detection on frequency domain information, it is possible to effectively determine whether the audio signal to be processed is in a low-frequency damage state, and determine whether there is a corresponding second energy difference value in each group of adjacent frequency bands that is less than the set energy tolerance threshold, so as to determine whether to directly execute the subsequent audio repair steps, increase the damage judgment link, and eliminate the need to repair the audio signal to be processed with better quality, thereby improving the audio processing efficiency.

[0108] Fig.15 A flowchart of an audio restoration method including a process of determining a cutoff frequency of frequency domain information provided by an embodiment of the present application. Fig.15 As shown, the following steps are included:

[0109] Step S501, obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data.

[0110] Step S502: input the damaged audio data into the trained teacher network model to obtain the target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data.

[0111] Step S503: obtaining an audio signal to be processed, performing frequency domain transformation on the audio signal to be processed to obtain a frequency domain signal, and performing attenuation detection of frequency band energy on the frequency domain signal to determine a cutoff frequency.

[0112] Among them, since the energy of the frequency domain signal will decay rapidly at the cutoff frequency, correspondingly, the slope of the frequency curve at the frequency point corresponding to the cutoff frequency is less than zero, and the absolute value reaches the maximum, and the energy will be significantly attenuated in a frequency band after the cutoff frequency. Therefore, the cutoff frequency can be determined by performing attenuation detection of the frequency band energy on the frequency domain information. The specific implementation process is as follows: First, determine the retrieval starting point and preset bandwidth length on the frequency curve corresponding to the frequency domain information. For example, for a 16kHz audio signal, its effective frequency range is 0-8kHz, and the corresponding retrieval starting point can be set within 0-4kHz according to the actual accuracy requirements. The preset bandwidth length can be set to 200Hz, 500Hz, etc., so as to obtain a first reference frequency band for calculating the slope. Based on the two endpoints of the first reference frequency band, the first energy attenuation value can be calculated, and the first energy attenuation value can be used to characterize the energy attenuation degree of the first reference frequency band. The first energy attenuation value is compared with the set first energy threshold. If the first energy attenuation value is less than or equal to the first energy threshold, it can be regarded that the energy attenuation degree of the first reference frequency band is small, and the possibility of it being the cutoff frequency is low. The search starting point can be right-shifted according to the preset step value, and the second reference frequency band is obtained in combination with the preset bandwidth length. The second energy attenuation value is calculated based on the two endpoints of the second reference frequency band, and the second energy attenuation value is compared with the first energy threshold, and the subsequent judgment process is re-executed. If the first energy attenuation value is greater than the first energy threshold, in order to prevent the two endpoints of the first reference frequency band from being extreme points, resulting in misjudgment of the cutoff frequency, the two endpoints of the first reference frequency band can be used as midpoints to determine the corresponding two narrow bands, and the difference in the average energy of the two narrow bands can be calculated respectively, and the difference is compared with the first energy threshold. If the difference is less than or equal to the first energy threshold, the search starting point is right-shifted according to the preset step value, and the subsequent judgment process is re-executed. If the difference is greater than the first energy threshold, the right endpoint of the first reference frequency band is shifted right according to the preset step value, and the third energy attenuation value is calculated based on the two endpoints of the first reference frequency band. The third energy attenuation value is compared with the second energy threshold. If the third energy attenuation value is greater than or equal to the second energy threshold, the right endpoint is continuously shifted right according to the step value, and the new energy attenuation value is re-determined to be compared with the second energy threshold until the step limit is reached. If the third energy attenuation value is less than the second energy threshold or the right shift of the right endpoint has reached the step limit, the in-band energy roll-off slope of the first reference frequency band is calculated. The in-band energy roll-off slope is compared with the recorded existing maximum slope. If the in-band energy roll-off slope exceeds the existing maximum slope, the in-band energy roll-off slope is updated to the new maximum slope. After recording the left endpoint corresponding to the first reference frequency band, the left endpoint is shifted right according to the preset step value, and a new reference frequency band is obtained in combination with the preset bandwidth length, and the subsequent judgment process is continued until the band edge of the frequency domain information is reached.If the energy roll-off slope in the band does not exceed the existing maximum slope, the left endpoint is directly shifted right according to the preset step value, and a new reference frequency band is obtained in combination with the preset bandwidth length, and the subsequent judgment process is continued until the frequency band edge of the frequency domain information is reached. Thus, the frequency of the left endpoint corresponding to the maximum slope finally obtained is the cutoff frequency.

[0113] Step S504: when the cutoff frequency is less than a preset frequency threshold, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal, and the frequency domain repair signal is transformed in the time domain to obtain an audio repair signal.

[0114] Among them, if the cutoff frequency is greater than or equal to the preset frequency threshold, it can be regarded that the high-frequency information of the audio signal to be processed is relatively complete, and the low-frequency energy state detection can be continued to determine whether the audio signal to be processed is in a low-frequency damage state. If there is low-frequency damage, continue to execute the subsequent steps to repair the audio. If there is no low-frequency damage, there is no need to repair the audio. Of course, the specific preset frequency threshold can be adaptively set according to the effective frequency range and detection accuracy requirements of the actual audio signal, and this application does not limit it. If the cutoff frequency is less than the preset frequency threshold, it can be regarded as the lack of high-frequency information of the audio signal to be processed, and continue to execute the subsequent steps to repair the audio.

[0115] As described above, by performing attenuation detection of the frequency band energy on the frequency domain information to determine the cutoff frequency, it is possible to effectively determine whether the audio signal to be processed is in a high-frequency missing state, and determine whether to directly execute the subsequent high-frequency repair step based on the comparison result of the cutoff frequency and the preset frequency threshold, thereby adding a high-frequency missing judgment link, eliminating the need to repair the audio signal to be processed with better quality, and improving the audio processing efficiency.

[0116] In one embodiment, Fig.16 A schematic diagram of a process of performing audio repair processing on an audio signal to be processed provided in an embodiment of the present application, such as Fig.16As shown, the silent frame detection module 501 performs silent detection on the input audio signal to be processed. If it is detected that the audio signal to be processed belongs to a non-silent frame, the audio signal to be processed is passed to the pre-processing module 502. The pre-processing module 502 performs frequency domain transformation processing on the audio signal to be processed to obtain frequency domain information, and passes the frequency domain information to the damage detection module 503 and the trained student network model 504. The damage detection module 503 performs low-frequency energy state detection and frequency band energy attenuation detection on the frequency domain information respectively. The specific detection process can refer to the aforementioned embodiment, and this application will not repeat it here. If it is detected that the audio signal to be processed is in a low-frequency loss state or a high-frequency missing state, the frequency domain information is passed to the student network model 504. The student network model 504 processes the frequency domain information to obtain a frequency domain repair signal, and passes the frequency domain repair signal to the post-processing module 505. The post-processing module 505 performs time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal, and outputs the audio repair signal.

[0117] As described above, by utilizing the trained student network model to perform real-time processing on audio signals detected as damaged in a call scenario, audio restoration can be achieved, damage and quantization noise problems caused by audio encoding and decoding can be solved, and the clarity and intelligibility of the audio signal can be improved.

[0118] Fig.17 This is a structural block diagram of an audio repair device provided in an embodiment of the present application. The device is configured to execute the audio repair method provided in the above embodiment and has the corresponding functional modules and beneficial effects of the execution method. Fig.17 As shown, the device comprises:

[0119] The teacher model training module 601 is configured to obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-built teacher network model based on the lossless audio data and the simulated audio data;

[0120] The student model training module 602 is configured to input the damaged audio data into the trained teacher network model to obtain the target audio data, and train the pre-built student network model based on the damaged audio data and the target audio data;

[0121] The audio signal repair module 603 is configured to obtain an audio signal to be processed, perform frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and perform time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

[0122] In the above, by acquiring lossless audio data and damaged audio data, performing damage simulation processing on the lossless audio data to obtain simulated audio data, and training a pre-constructed teacher network model based on the lossless audio data and the simulated audio data; inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training a pre-constructed student network model based on the damaged audio data and the target audio data; acquiring an audio signal to be processed, performing frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal. In the above scheme, since the teacher network model does not need to be deployed in the actual scene, it is not restricted by the model calculation complexity and delay during design. By training the pre-built teacher network model based on lossless audio data and simulated audio data, an audio repair model with high calculation complexity and large parameter scale can be obtained. In addition, by inputting the damaged audio data into the trained teacher network model to obtain the target audio data, audio repair data matching the damaged audio data can be obtained, providing real and effective data pairs for training the student network model with lower calculation complexity, which is beneficial to improving the audio repair ability of the student network model and ensuring the audio repair effect. In addition, the student network model is used to repair the audio signal to be processed, which can be flexibly deployed in real-time scenarios on mobile terminals with lower computing power requirements to meet the needs of real-time audio repair.

[0123] In a possible embodiment, the student model training module 602 is further configured to:

[0124] The target audio data and the damaged audio data are fused according to the set dry-wet ratio factor to obtain the reference audio data, where the dry-wet ratio factor is determined based on the number of trained cycles, the total number of training cycles and the minimum value of the preset factor;

[0125] The pre-built student network model is trained on the corrupted audio data and the reference audio data.

[0126] In one possible embodiment, the teacher network model includes a preprocessing module, a one-stage model, a two-stage model, a fusion module and a post-processing module connected in sequence. The preprocessing module and the one-stage module are respectively connected to the fusion module in a jump manner. The preprocessing module is used to receive the damaged audio data and perform frequency domain conversion processing on the damaged audio data to obtain first frequency domain data. The one-stage model is an autoencoder structure, and the two-stage model is a voice coder structure. The fusion module is used to fuse the first frequency domain data, the first processing result output by the one-stage model, and the second processing result output by the two-stage model to obtain second frequency domain data. The post-processing module is used to perform time domain conversion processing on the second frequency domain data to obtain target audio data, and output the target audio data.

[0127] In one possible embodiment, the student network model includes a scale transformation module, a first encoder module, a first decoder module, a timing modeling layer, a second encoder module, a second decoder module and an energy correction module connected in sequence, the first encoder module is jump-connected to the second decoder module, the first decoder module is jump-connected to the second encoder module, the scale transformation module is used to receive the frequency domain signal and perform feature compression processing on the frequency domain signal, the energy correction module is used to fuse the frequency domain signal and the output result of the second decoder module to obtain a frequency domain repair signal, and output the frequency domain repair signal.

[0128] In a possible embodiment, a signal fusion module is further included, configured as follows:

[0129] The target frequency domain repair signal is determined by fusing the amplitude information of the frequency domain repair signal and the phase information of the frequency domain signal;

[0130] Correspondingly, the audio signal repair module 603 is further configured as follows:

[0131] The target frequency domain repair signal is processed in the time domain to obtain the audio repair signal.

[0132] In a possible embodiment, an energy detection module is further included, configured as follows:

[0133] Divide the frequency domain signal into a preset number of frequency bands, and determine energy values ​​of the multiple frequency bands according to amplitude information corresponding to the frequency domain signal;

[0134] Subtracting energy values ​​of adjacent frequency bands in the multiple frequency bands to obtain first energy difference values ​​corresponding to each group of adjacent frequency bands;

[0135] Subtracting the first energy difference value corresponding to each group of adjacent frequency bands from the preset energy threshold value to obtain a second energy difference value;

[0136] Correspondingly, the audio signal repair module 603 is further configured as follows:

[0137] In the case that there is a corresponding second energy difference value in each group of adjacent frequency bands that is less than the set energy tolerance threshold, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal.

[0138] In a possible embodiment, a cutoff frequency determination module is further included, configured as follows:

[0139] Performing attenuation detection of frequency band energy on the frequency domain signal to determine the cutoff frequency;

[0140] Correspondingly, the audio signal repair module 603 is further configured as follows:

[0141] When the cutoff frequency is less than the preset frequency threshold, the frequency domain signal is input into the trained student network model to obtain the frequency domain repair signal.

[0142] Fig.18 A schematic diagram of the structure of an audio repair device provided in an embodiment of the present application is shown in FIG. Fig.18 As shown, the device includes a processor 701, a memory 702, an input device 703 and an output device 704; the number of processors 701 in the device can be one or more. Fig.18 A processor 701 is taken as an example; the processor 701, the memory 702, the input device 703 and the output device 704 in the device can be connected by a bus or other means. Fig.18 The example of connection via a bus is taken. The memory 702, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the audio repair method in the embodiment of the present application. The processor 701 executes various functional applications and data processing of the device by running the software programs, instructions and modules stored in the memory 702, that is, implements the above-mentioned audio repair method. The input device 703 can be configured to receive input digital or character information, and generate key signal input related to user settings and function controls of the device. The output device 704 may include display devices such as display screens.

[0143] An embodiment of the present application also provides a non-volatile storage medium containing computer executable instructions, which, when executed by a computer processor, is configured to execute an audio repair method described in the above embodiment, which includes: obtaining lossless audio data and damaged audio data, performing damage simulation processing on the lossless audio data to obtain simulated audio data, and training a pre-constructed teacher network model based on the lossless audio data and the simulated audio data; inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training a pre-constructed student network model based on the damaged audio data and the target audio data; obtaining an audio signal to be processed, performing frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, inputting the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and performing time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

[0144] It is worth noting that in the embodiment of the above-mentioned audio repair device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not configured to limit the protection scope of the embodiments of the present application.

[0145] In some possible implementations, various aspects of the method provided in the present application may also be implemented in the form of a program product, which includes a program code. When the program product is run on a computer device, the program code is configured to enable the computer device to execute the steps of the method according to various exemplary embodiments of the present application described above in this specification. For example, the computer device may execute the audio repair method described in the embodiment of the present application. The program product may be implemented in any combination of one or more readable media.

Claims

1. An audio repair method, characterized in that: include: Acquire lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data; Inputting the damaged audio data into the trained teacher network model to obtain target audio data, and training the pre-constructed student network model based on the damaged audio data and the target audio data; Acquire an audio signal to be processed, perform frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and perform time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

2. The audio repair method according to claim 1, characterized in that: The training of the pre-constructed student network model based on the damaged audio data and the target audio data comprises: The target audio data and the damaged audio data are fused according to a set dry-wet ratio factor to obtain reference audio data, wherein the dry-wet ratio factor is determined based on the number of trained cycles, the total number of training cycles and a preset factor minimum value; A pre-built student network model is trained based on the damaged audio data and the reference audio data.

3. The audio repair method according to claim 1, characterized in that: The teacher network model includes a preprocessing module, a one-stage model, a two-stage model, a fusion module and a post-processing module connected in sequence. The preprocessing module and the one-stage module are respectively connected to the fusion module in a jump manner. The preprocessing module is used to receive the damaged audio data and perform frequency domain conversion processing on the damaged audio data to obtain first frequency domain data. The one-stage model is an autoencoder structure, and the two-stage model is a vocoder structure. The fusion module is used to fuse the first frequency domain data, the first processing result output by the one-stage model, and the second processing result output by the two-stage model to obtain second frequency domain data. The post-processing module is used to perform time domain conversion processing on the second frequency domain data to obtain the target audio data, and output the target audio data.

4. The audio repair method according to claim 1, characterized in that: The student network model includes a scale transformation module, a first encoder module, a first decoder module, a timing modeling layer, a second encoder module, a second decoder module and an energy correction module which are connected in sequence. The first encoder module is connected to the second decoder module in a jump step, and the first decoder module is connected to the second encoder module in a jump step. The scale transformation module is used to receive the frequency domain signal and perform feature compression processing on the frequency domain signal. The energy correction module is used to fuse the frequency domain signal and the output result of the second decoder module to obtain the frequency domain repair signal, and output the frequency domain repair signal.

5. The audio repair method according to claim 1, characterized in that: Before performing time domain transform processing on the frequency domain repair signal to obtain the audio repair signal, the method further includes: Determine a target frequency domain repair signal by fusing the amplitude information of the frequency domain repair signal and the phase information of the frequency domain signal; Correspondingly, performing time domain transform processing on the frequency domain repair signal to obtain the audio repair signal includes: The target frequency domain repair signal is subjected to time domain transformation processing to obtain an audio repair signal.

6. The audio repair method according to claim 1, characterized in that: Before inputting the frequency domain signal into the trained student network model to obtain the frequency domain repair signal, the method further includes: Divide the frequency domain signal into a preset number of frequency bands, and determine energy values ​​of the multiple frequency bands according to amplitude information corresponding to the frequency domain signal; Subtracting energy values ​​of adjacent frequency bands in the multiple frequency bands to obtain first energy difference values ​​corresponding to each group of adjacent frequency bands; Subtracting the first energy difference value corresponding to each group of adjacent frequency bands from the preset energy threshold value to obtain a second energy difference value; Accordingly, the step of inputting the frequency domain signal into the trained student network model includes: In the case that there is a corresponding second energy difference value in each group of adjacent frequency bands that is less than a set energy tolerance threshold, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal.

7. The audio repair method according to claim 1, characterized in that: Before inputting the frequency domain signal into the trained student network model to obtain the frequency domain repair signal, the method further includes: Performing attenuation detection of frequency band energy on the frequency domain signal to determine a cutoff frequency; Accordingly, the step of inputting the frequency domain signal into the trained student network model includes: When the cut-off frequency is less than a preset frequency threshold, the frequency domain signal is input into the trained student network model to obtain a frequency domain repair signal.

8. An audio repair device, characterized in that: include: A teacher model training module is configured to obtain lossless audio data and damaged audio data, perform damage simulation processing on the lossless audio data to obtain simulated audio data, and train a pre-constructed teacher network model based on the lossless audio data and the simulated audio data; A student model training module is configured to input the damaged audio data into the trained teacher network model to obtain target audio data, and train the pre-constructed student network model based on the damaged audio data and the target audio data; The audio signal repair module is configured to obtain an audio signal to be processed, perform frequency domain transformation processing on the audio signal to be processed to obtain a frequency domain signal, input the frequency domain signal into the trained student network model to obtain a frequency domain repair signal, and perform time domain transformation processing on the frequency domain repair signal to obtain an audio repair signal.

9. An audio repair device, the device comprising: one or more processors; The storage device is configured to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the audio repair method according to any one of claims 1 to 7.

10. A non-volatile storage medium storing computer executable instructions, wherein the computer executable instructions are configured to perform the audio repair method according to any one of claims 1 to 7 when executed by a computer processor.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the audio repair method according to any one of claims 1 to 7 is implemented.

Images