Audio restoration method and apparatus, and storage medium and electronic device

Abstract

An audio restoration method and apparatus, and a storage medium and an electronic device. The method comprises: extracting frequency-domain features from audio to be restored; using a frequency-domain restoration generator to perform restoration processing on the frequency-domain features, so as to obtain restored frequency-domain features; and using a signal conversion generator to perform time-domain conversion processing on the restored frequency-domain features, so as to obtain restored audio.

Description

Audio repair method, device, storage medium and electronic device

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on October 17, 2023, with application number 202311345973.4 and application name “Audio repair method, device, storage medium and electronic device”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of audio processing technology, and in particular to an audio repair method, device, storage medium, and electronic device. Background Art

[0003] Audio restoration involves repairing audio for super-resolution, reverberation, and noise to achieve high-quality audio. Currently, common audio restoration solutions, such as the VoiceFixer model, typically operate offline, making them unsuitable for scenarios like televisions where real-time restoration is required. Furthermore, while they typically work well for speech, they perform poorly for music and other audio where high-frequency components have a low correlation with low-frequency components. Technical issues

[0004] Current audio repair solutions have the problem of poor performance in online real-time audio repair. Technical Solutions

[0005] The embodiments of the present application provide a solution that can effectively improve the effect of online real-time audio repair.

[0006] The embodiments of this application provide the following technical solutions:

[0007] According to one embodiment of the present application, an audio repair method includes: obtaining audio to be repaired; extracting frequency domain features of the audio to be repaired to obtain frequency domain features of the audio to be repaired; using a frequency domain repair generator to repair the frequency domain features to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by performing adversarial training on a real-time repair generator and a repair discriminator; using a signal conversion generator to perform time domain conversion processing on the repair frequency domain features to obtain repaired audio, wherein the signal conversion generator is a trained real-time conversion generator obtained by performing adversarial training on a real-time conversion generator and a conversion discriminator.

[0008] In some embodiments of the present application, the frequency domain repair generator is used to repair the frequency domain features to obtain repaired frequency domain features, including: generating input frequency domain features based on the frequency domain features; using the frequency domain repair generator to encode and decode the input frequency domain features to obtain codec features; and averaging the codec features to obtain average features; multiplying the average features with the intermediate frames of the input frequency domain features to obtain the repaired frequency domain features.

[0009] In some embodiments of the present application, the encoding and decoding processing of the input frequency domain features to obtain encoding and decoding features includes: performing a first convolution processing on the input frequency domain features to obtain a first convolution result; performing a first pooling processing on the first convolution result to obtain a first pooling result; performing a second convolution processing on the first pooling result to obtain a second convolution result; performing a second pooling processing on the second convolution result to obtain a second pooling result; performing a third convolution processing on the second pooling result to obtain a third convolution result; performing a recurrent neural network encoding processing on the third convolution result to obtain an encoding result; splicing the encoding result and the third convolution result and then performing a first deconvolution processing to obtain a first deconvolution result; splicing the first deconvolution result and the second convolution result and then performing a second deconvolution processing to obtain a second deconvolution result; splicing the second deconvolution result and the first convolution result and then performing a third deconvolution processing to obtain a third deconvolution result; performing a fourth convolution processing on the third deconvolution result to obtain the encoding and decoding features.

[0010] In some embodiments of the present application, the frequency domain repair generator is trained in the following manner: obtaining a repair training set, the repair training set including different sample frequency domain features and corresponding label frequency domain features, the label frequency domain features being extracted from the sample audio, and the sample frequency domain features being extracted from the damaged audio corresponding to the sample audio; performing multiple rounds of iterative adversarial training on the first generative adversarial network based on the sample frequency domain features and the corresponding label frequency domain features in the repair training set to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes: using the sample frequency domain features and the label frequency domain features as the input data and label data of the real-time repair generator respectively, adversarially training the real-time repair generator based on the repair discriminator to obtain the real-time repair generator after training of the round; and, adversarially training the repair discriminator based on the real-time repair generator after training of the round.

[0011] In some embodiments of the present application, the real-time repair generator is adversarially trained based on the repair discriminator to obtain the real-time repair generator after training for the round, including: inputting the sample frequency domain features into the real-time repair generator to obtain output frequency domain features; caching the output frequency domain features corresponding to a predetermined number of the sample frequency domain features to obtain an output frequency domain feature set; calculating a first loss based on the output frequency domain feature set and a label frequency domain feature set, wherein the label frequency domain feature set includes label frequency domain features corresponding to a predetermined number of the sample frequency domain features; inputting the output frequency domain feature set as pseudo data into the repair discriminator to obtain a first probability value; calculating a second loss based on the first probability value and the true label; and updating the parameters of the real-time repair generator by combining the first loss and the second loss to obtain the real-time repair generator after training for the round.

[0012] In some embodiments of the present application, the real-time repair generator after training based on the round performs adversarial training on the repair discriminator, including: inputting the label frequency domain feature set as true data into the repair discriminator to obtain a second probability value; calculating a third loss based on the second probability value and the false label; inputting the sample frequency domain features into the real-time repair generator after training of the round to obtain predicted frequency domain features; caching the predicted frequency domain features corresponding to the sample frequency domain features for a predetermined number of frames to obtain a predicted frequency domain feature set; inputting the predicted frequency domain feature set as pseudo data into the repair discriminator to obtain a third probability value; calculating a fourth loss based on the third probability value and the true label; and updating the parameters of the repair discriminator by combining the third loss and the fourth loss.

[0013] In some embodiments of the present application, the signal conversion generator is trained in the following manner: obtaining a conversion training set, the conversion training set including different preset frequency domain features and corresponding label time domain data; performing multiple rounds of iterative adversarial training on the second generative adversarial network based on the preset frequency domain features and the corresponding label time domain data in the conversion training set to obtain a trained signal conversion generator; wherein, the iterative adversarial training of the second generative adversarial network in each round includes: using the preset frequency domain features and the label time domain data as the input data and label data of the real-time conversion generator respectively, and adversarially training the real-time conversion generator based on the conversion discriminator to obtain the real-time conversion generator after training of the round; and, performing adversarial training on the conversion discriminator based on the real-time conversion generator after training of the round.

[0014] In some embodiments of the present application, the real-time conversion generator obtained by adversarial training of the conversion discriminator to obtain the real-time conversion generator after training of the round includes: inputting the preset frequency domain features into the real-time conversion generator to obtain output time domain data; caching the output time domain data corresponding to the preset frequency domain features of the target number of frames to obtain an output time domain data set; calculating a fifth loss based on the output time domain data set and the label time domain data set, the label time domain data set including the label time domain data corresponding to the preset frequency domain features of the target number of frames; performing short-time Fourier transform on the output frequency domain feature set and the label time domain data set respectively and calculating a sixth loss; performing short-time Fourier transform on the output frequency domain feature set The transformed data after the time Fourier transform is input into the frequency domain discriminator as pseudo data to obtain a fourth probability value; the seventh loss is calculated based on the fourth probability value and the true label; the output frequency domain feature set is input into the time domain discriminator as pseudo data to obtain a first output feature and a fifth probability value; the eighth loss is calculated based on the fifth probability value and the true label; the labeled time domain data corresponding to the preset frequency domain feature is input into the time domain discriminator as true data to obtain a second output feature; the ninth loss is calculated based on the second output feature and the first output feature; the fifth loss, the sixth loss, the seventh loss, the eighth loss and the ninth loss are combined to update the real-time conversion generator to obtain the real-time conversion generator after training of the round.

[0015] In some embodiments of the present application, the real-time conversion generator after training based on the round performs adversarial training on the time domain discriminator, including: inputting the labeled time domain data set z as true data into the time domain discriminator to obtain a sixth probability value, and calculating the tenth loss based on the sixth probability value and the true label; inputting the transformed data after the short-time Fourier transform of the labeled time domain data set into the time domain discriminator as true data to obtain a seventh probability value; calculating the eleventh loss based on the seventh probability value and the true label; inputting the preset frequency domain feature into the real-time conversion generator after training of the round to obtain the predicted time domain data; cache the predicted time domain data corresponding to the preset frequency domain features of the target number of frames to obtain a predicted time domain data set; input the predicted time domain data set as pseudo data into the time domain discriminator to obtain an eighth probability value; calculate the twelfth loss based on the eighth probability value and the false label; input the transformed data after short-time Fourier transform of the predicted time domain data set into the time domain discriminator as pseudo data to obtain a ninth probability value; calculate the thirteenth loss based on the ninth probability value and the false label; and update the parameters of the time domain discriminator by combining the tenth loss, the eleventh loss, the twelfth loss and the thirteenth loss.

[0016] According to one embodiment of the present application, an audio repair device includes: an acquisition module for acquiring audio to be repaired; an extraction module for performing frequency domain feature extraction on the audio to be repaired to obtain frequency domain features of the audio to be repaired; a repair module for using a frequency domain repair generator to repair the frequency domain features to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by performing adversarial training on a real-time repair generator and a repair discriminator; a conversion module for using a signal conversion generator to perform time domain conversion processing on the repair frequency domain features to obtain repaired audio, wherein the signal conversion generator is a trained real-time conversion generator obtained by performing adversarial training on a real-time conversion generator and a conversion discriminator.

[0017] In some embodiments of the present application, the repair module is used to: generate input frequency domain features based on the frequency domain features; use the frequency domain repair generator to encode and decode the input frequency domain features to obtain codec features; and average the codec features to obtain average features; multiply the average features with the intermediate frames of the input frequency domain features to obtain the repaired frequency domain features.

[0018] In some embodiments of the present application, the repair module is used to: perform a first convolution process on the input frequency domain feature to obtain a first convolution result; perform a first pooling process on the first convolution result to obtain a first pooling result; perform a second convolution process on the first pooling result to obtain a second convolution result; perform a second pooling process on the second convolution result to obtain a second pooling result; perform a third convolution process on the second pooling result to obtain a third convolution result; perform a recurrent neural network encoding process on the third convolution result to obtain an encoding result; perform a first deconvolution process on the encoding result and the third convolution result after splicing the encoding result and the third convolution result to obtain a first deconvolution result; perform a second deconvolution process on the first deconvolution result and the second convolution result after splicing the first deconvolution result to obtain a second deconvolution result; perform a third deconvolution process on the second deconvolution result and the first convolution result after splicing the second deconvolution result to obtain a third deconvolution result; perform a fourth convolution process on the third deconvolution result to obtain the encoding and decoding feature.

[0019] In some embodiments of the present application, the device also includes a first training module, which is used to: obtain a repair training set, wherein the repair training set includes different sample frequency domain features and corresponding label frequency domain features, the label frequency domain features are extracted from the sample audio, and the sample frequency domain features are extracted from the damaged audio corresponding to the sample audio; based on the sample frequency domain features and the corresponding label frequency domain features in the repair training set, perform multiple rounds of iterative adversarial training on the first generative adversarial network to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes: using the sample frequency domain features and the label frequency domain features as the input data and label data of the real-time repair generator respectively, and adversarially training the real-time repair generator based on the repair discriminator to obtain the real-time repair generator after training of the round; and adversarially training the repair discriminator based on the real-time repair generator after training of the round.

[0020] In some embodiments of the present application, the first training module is used to: input the sample frequency domain features into the real-time repair generator to obtain output frequency domain features; cache the output frequency domain features corresponding to a predetermined number of frames of the sample frequency domain features to obtain an output frequency domain feature set; calculate a first loss based on the output frequency domain feature set and a label frequency domain feature set, wherein the label frequency domain feature set includes label frequency domain features corresponding to a predetermined number of frames of the sample frequency domain features; input the output frequency domain feature set as pseudo data into the repair discriminator to obtain a first probability value; calculate a second loss based on the first probability value and the true label; and update the parameters of the real-time repair generator by combining the first loss and the second loss to obtain the real-time repair generator after training for the round.

[0021] In some embodiments of the present application, the first training module is used to: input the label frequency domain feature set as true data into the repair discriminator to obtain a second probability value; calculate a third loss based on the second probability value and the false label; input the sample frequency domain features into the real-time repair generator after training of the round to obtain predicted frequency domain features; cache the predicted frequency domain features corresponding to the sample frequency domain features for a predetermined number of frames to obtain a predicted frequency domain feature set; input the predicted frequency domain feature set as pseudo data into the repair discriminator to obtain a third probability value; calculate a fourth loss based on the third probability value and the true label; and update the parameters of the repair discriminator by combining the third loss and the fourth loss.

[0022] In some embodiments of the present application, the device also includes a second training module, which is used to: obtain a conversion training set, which includes different preset frequency domain features and corresponding label time domain data; perform multiple rounds of iterative adversarial training on the second generative adversarial network based on the preset frequency domain features and the corresponding label time domain data in the conversion training set to obtain a trained signal conversion generator; wherein, the iterative adversarial training of the second generative adversarial network in each round includes: using the preset frequency domain features and the label time domain data as the input data and label data of the real-time conversion generator respectively, and adversarially training the real-time conversion generator based on the conversion discriminator to obtain the real-time conversion generator after training of the round; and, performing adversarial training on the conversion discriminator based on the real-time conversion generator after training of the round.

[0023] In some embodiments of the present application, the second training module is used to: input the preset frequency domain features into the real-time conversion generator to obtain output time domain data; cache the output time domain data corresponding to the preset frequency domain features for a target number of frames to obtain an output time domain data set; calculate the fifth loss based on the output time domain data set and the label time domain data set, wherein the label time domain data set includes the label time domain data corresponding to the preset frequency domain features for a target number of frames; calculate the sixth loss after performing short-time Fourier transform on the output frequency domain feature set and the label time domain data set; use the transformed data after the short-time Fourier transform of the output frequency domain feature set as pseudo data Input the frequency domain discriminator to obtain a fourth probability value; calculate the seventh loss based on the fourth probability value and the true label; input the output frequency domain feature set as pseudo data into the time domain discriminator to obtain a first output feature and a fifth probability value; calculate the eighth loss based on the fifth probability value and the true label; input the labeled time domain data corresponding to the preset frequency domain feature as true data into the time domain discriminator to obtain a second output feature; calculate the ninth loss based on the second output feature and the first output feature; combine the fifth loss, sixth loss, seventh loss, eighth loss and ninth loss to update the real-time conversion generator to obtain the real-time conversion generator after training of the round.

[0024] In some embodiments of the present application, the second training module is configured to: input the labeled time domain dataset z as true data into the time domain discriminator to obtain a sixth probability value, and calculate a tenth loss based on the sixth probability value and the true label; input the transformed data after short-time Fourier transform of the labeled time domain dataset into the time domain discriminator as true data to obtain a seventh probability value; calculate an eleventh loss based on the seventh probability value and the true label; input the preset frequency domain features into the round of post-training real-time conversion generator to obtain predicted time domain data; cache the predicted time domain data corresponding to the preset frequency domain features for a target number of frames to obtain a predicted time domain dataset; input the predicted time domain dataset as pseudo data into the time domain discriminator to obtain an eighth probability value; calculate a twelfth loss based on the eighth probability value and the false label; input the transformed data after short-time Fourier transform of the predicted time domain dataset into the time domain discriminator as pseudo data to obtain a ninth probability value; calculate a thirteenth loss based on the ninth probability value and the false label; and update the parameters of the time domain discriminator by combining the tenth loss, the eleventh loss, the twelfth loss, and the thirteenth loss.

[0025] According to another embodiment of the present application, a storage medium stores a computer program thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method described in the embodiment of the present application.

[0026] According to another embodiment of the present application, an electronic device may include: a memory storing a computer program; and a processor reading the computer program stored in the memory to execute the method described in the embodiment of the present application.

[0027] According to another embodiment of the present application, a computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in various optional implementations described in the embodiments of the present application. Beneficial effects

[0028] In an embodiment of the present application, audio to be repaired is obtained; frequency domain features are extracted from the audio to be repaired to obtain frequency domain features of the audio to be repaired; a frequency domain repair generator is used to repair the frequency domain features to obtain repaired frequency domain features, and the frequency domain repair generator is a trained real-time repair generator obtained by performing adversarial training on a real-time repair generator and a repair discriminator; a signal conversion generator is used to perform time domain conversion processing on the repaired frequency domain features to obtain repaired audio, and the signal conversion generator is a trained real-time conversion generator obtained by performing adversarial training on a real-time conversion generator and a time domain discriminator.

[0029] In this way, the frequency domain repair generator and signal conversion generator obtained through adversarial training are connected in series to form a repair network to repair the audio. On the one hand, through adversarial training, the repair network formed in series can have a good repair effect on music or other audio whose high-frequency components have a lower correlation with their low-frequency components. On the other hand, the frequency domain repair generator and signal conversion generator trained respectively by the real-time repair generator and the real-time conversion generator during adversarial training can achieve real-time audio repair. Furthermore, this application as a whole can effectively improve the effect of online real-time audio repair. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0031] FIG1 shows a flowchart of an audio repair method according to an embodiment of the present application.

[0032] FIG2 shows a flowchart of adversarial training according to an embodiment of the present application.

[0033] FIG3 shows a flowchart of an audio restoration process according to an embodiment of the present application.

[0034] FIG4 is a schematic diagram showing structural elements of the generator in the embodiment of FIG3 .

[0035] FIG5 shows a structural diagram of a discriminator according to an embodiment of the present application.

[0036] FIG. 6 is a schematic diagram showing the structural elements of the discriminator in the embodiment of FIG. 5 .

[0037] FIG7 shows a flowchart of adversarial training according to another embodiment of the present application.

[0038] FIG8 shows a flowchart of adversarial training according to another embodiment of the present application.

[0039] FIG9 shows a block diagram of an audio repair apparatus according to an embodiment of the present application.

[0040] FIG10 shows a block diagram of an electronic device according to an embodiment of the present application.

[0041] Implementation Methods of the Application

[0042] The present disclosure will be further described in detail below in conjunction with the accompanying drawings and examples. It should be understood that the examples provided herein are merely for explaining the present disclosure and are not intended to limit the present disclosure. In addition, the examples provided below are partial examples for implementing the present disclosure, rather than providing all examples for implementing the present disclosure. In the absence of conflict, the technical solutions described in the examples of the present disclosure may be implemented in any combination.

[0043] It should be noted that, in the embodiments of the present disclosure, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a method or apparatus comprising a series of elements includes not only the elements explicitly stated, but also other elements not explicitly listed, or also includes elements inherent to the implementation of the method or apparatus. In the absence of further restrictions, an element defined by the sentence "comprising a ..." does not exclude the presence of other related elements (such as steps in the method or units in the apparatus, for example, a unit may be part of a circuit, part of a processor, part of a program or software, etc.) in the method or apparatus comprising the element.

[0044] For example, the audio repair method provided by the embodiment of the present disclosure includes a series of steps, but the audio repair method provided by the embodiment of the present disclosure is not limited to the recorded steps. Similarly, the audio repair device provided by the embodiment of the present disclosure includes a series of units, but the device provided by the embodiment of the present disclosure is not limited to including the units explicitly recorded, and may also include units that need to be set up to obtain relevant information or perform processing based on the information.

[0045] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present disclosure pertains. The terms used herein are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure.

[0046] Figure 1 schematically illustrates a flow chart of an audio repair method according to one embodiment of the present application. The audio repair method can be executed by any device and / or server with processing capabilities, such as televisions, computers, mobile phones, smart watches, and household appliances, and servers such as cloud servers, physical servers, or server machines.

[0047] As shown in FIG. 1 , the audio repair method may include steps S110 to S140 .

[0048] Step S110, obtaining the audio to be repaired; step S120, performing frequency domain feature extraction on the audio to be repaired to obtain the frequency domain features of the audio to be repaired; step S130, using a frequency domain repair generator to repair the frequency domain features to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by performing adversarial training on a real-time repair generator and a repair discriminator; step S140, using a signal conversion generator to perform time domain conversion processing on the repaired frequency domain features to obtain repaired audio, wherein the signal conversion generator is a trained real-time conversion generator obtained by performing adversarial training on a real-time conversion generator and a conversion discriminator.

[0049] The audio to be repaired is the audio to be repaired. The method of obtaining the audio to be repaired can be set according to actual conditions. For example, the audio to be repaired can be obtained from a predetermined storage location. This application does not make specific limitations on this.

[0050] Frequency domain features are extracted from the audio to be repaired to obtain frequency domain features of the audio to be repaired. Specifically, the one-dimensional audio signal of the audio to be repaired is upsampled to a target sampling rate to obtain upsampled audio; then, the logarithmic Mel spectrum features (Log-Mel spectrum features) of the upsampled audio are extracted, and the logarithmic Mel spectrum features (Log-Mel spectrum features) are used as the frequency domain features of the audio to be repaired. It is understood that in other optional methods, the Mel spectrum of the audio to be repaired can also be extracted as the frequency domain features.

[0051] The real-time repair generator and the repair discriminator are pre-trained in an adversarial manner to obtain the trained real-time repair generator. The trained real-time repair generator is the frequency domain repair generator, which is also a deep neural network used to repair frequency domain signals. The deep neural network that constitutes the frequency domain repair generator can be a neural network such as ResUnet or a convolutional neural network. The repair discriminator is a discriminator in the generative adversarial network, used to distinguish between true and false.

[0052] The frequency domain features are input into the frequency domain repair generator, which can perform repair processing on the frequency domain features to obtain repaired frequency domain features. A predetermined number of frames (which can be pre-specified based on actual conditions, such as 0.25 seconds or 0.144 seconds) of frequency domain features can be cached in buffer H1, and then the cached predetermined number of frames of frequency domain features are input into the frequency domain repair generator to ensure the real-time processing of the frequency domain repair generator.

[0053] The real-time conversion generator and the conversion discriminator are pre-trained in an adversarial manner to obtain the trained real-time conversion generator. The trained real-time conversion generator is also known as the signal conversion generator, which is also a deep neural network used to convert frequency-domain signals into time-domain signals. The deep neural network that constitutes the signal conversion generator can be a recurrent neural network or a convolutional neural network. The conversion discriminator is a discriminator in the generative adversarial network, used to distinguish true from false.

[0054] The repaired frequency domain features are input into the signal conversion generator, which performs time domain conversion on the repaired frequency domain features to obtain repaired audio. The target number of frames of repaired frequency domain features can be cached in buffer H2, and then the cached target number of frames (which can be pre-specified based on actual conditions) of repaired frequency domain features are input into the signal conversion generator to ensure real-time processing by the signal conversion generator.

[0055] In this way, based on steps S110 to S140, the frequency domain repair generator and signal conversion generator obtained through adversarial training are connected in series to form a repair network to repair the audio. On the one hand, through adversarial training, the repair network formed in series can have a good repair effect on music or other audio whose high-frequency components have a lower correlation with their low-frequency components. On the other hand, during adversarial training, the frequency domain repair generator and signal conversion generator trained respectively by the real-time repair generator and the real-time conversion generator can achieve real-time audio repair. Furthermore, this application as a whole can effectively improve the effect of online real-time audio repair.

[0056] The following describes further optional specific embodiments of each step performed when performing audio repair in the embodiment of Figure 1.

[0057] In one embodiment, the frequency domain repair generator may be trained in the following manner:

[0058] Obtaining a restoration training set, wherein the restoration training set includes different sample frequency domain features and corresponding label frequency domain features, wherein the label frequency domain features are extracted from the sample audio, and the sample frequency domain features are extracted from the damaged audio corresponding to the sample audio;

[0059] Based on the frequency domain features of the samples in the repair training set and the corresponding frequency domain features of the labels, the first generative adversarial network is subjected to multiple rounds of iterative adversarial training to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes:

[0060] The sample frequency domain features and the label frequency domain features are respectively used as the input data and label data of the real-time repair generator, and the real-time repair generator is adversarially trained based on the repair discriminator to obtain the real-time repair generator after training for the round; and the repair discriminator is adversarially trained based on the real-time repair generator after training for the round.

[0061] The sample audio can be speech from a television source, or audio from scenes such as movies, music, news, and sports. The sample audio is processed by adding noise, reverberation, downsampling, and low-bitrate compression, followed by decompression and upsampling to obtain the corresponding impaired audio. The frequency domain features of the sample audio are extracted as sample frequency domain features, and the frequency domain features of the impaired audio corresponding to the sample audio are extracted as label frequency domain features.

[0062] A first generative adversarial network can be formed by the real-time repair generator and the repair discriminator. According to the frequency domain features of the samples in the repair training set and the corresponding label frequency domain features, the first generative adversarial network is subjected to multiple rounds of iterative adversarial training until the predetermined training end conditions are met (for example, the loss value converges and the repair effect meets the predetermined requirements). The trained first generative adversarial network is obtained, and the real-time repair generator in the trained first generative adversarial network is the trained real-time repair generator.

[0063] The first generative adversarial network can be formed by the real-time restoration generator and the restoration discriminator, which can enable the trained real-time restoration generator (i.e., the frequency domain restoration generator) to have a good super-resolution effect of predicting high-frequency components through low-frequency components for music or other audio whose high-frequency components have lower correlation with their low-frequency components.

[0064] In one embodiment, the adversarial training of the real-time repair generator based on the repair discriminator to obtain the round of trained real-time repair generator may include:

[0065] Input the sample frequency domain features into the real-time repair generator to obtain output frequency domain features; cache the output frequency domain features corresponding to a predetermined number of the sample frequency domain features to obtain an output frequency domain feature set; calculate a first loss based on the output frequency domain feature set and a label frequency domain feature set, wherein the label frequency domain feature set includes label frequency domain features corresponding to a predetermined number of the sample frequency domain features; input the output frequency domain feature set as pseudo data into the repair discriminator to obtain a first probability value; calculate a second loss based on the first probability value and the true label; and update the parameters of the real-time repair generator by combining the first loss and the second loss to obtain the real-time repair generator after training for the round.

[0066] Specifically, referring to the "Training of the Generator" section in Figure 2, the sample frequency domain features M are input into the real-time repair generator to obtain the output frequency domain features M1. The output frequency domain features corresponding to the sample frequency domain features for a predetermined number of frames (which can be pre-specified according to actual conditions, such as 0.25 seconds or 0.144 seconds) can be cached by buffer 1 to obtain an output frequency domain feature set. At the same time, buffer 2 can cache the label frequency domain features M' corresponding to the sample frequency domain features M for the predetermined number of frames to obtain a label frequency domain feature set. The first loss LOSS1 can be calculated based on the output frequency domain feature set and the label frequency domain feature set.

[0067] The output frequency domain feature set is input as pseudo data into the repair discriminator to obtain the first probability value P1; the second loss LOSS2 is calculated based on the first probability value P1 and the true label 1 (the marked data is true); the first loss LOSS1 and the second loss LOSS2 are combined for back propagation to update the parameters of the real-time repair generator to obtain the real-time repair generator after training in this round.

[0068] For example, the input sample frequency domain feature M is a (1×T×K) structure, its output M1 is a (1×1×K) structure, and its corresponding label frequency domain feature M' is also a (1×1×K) structure. However, the loss function is not calculated directly, but is cached through buffer 1 and buffer 2 to cache T2 frames before the loss function is calculated once. After processing by buffer 1, M1 becomes the output frequency domain feature set of the (1×T2×K) structure, and buffer 2 makes M' also become the label frequency domain feature set of the (1×T2×K) structure. The first loss LOSS1 can be calculated based on the output frequency domain feature set and the label frequency domain feature set. Buffer 1 and buffer 2 can be the same buffer.

[0069] After processing by buffer 1, M1 becomes an output frequency domain feature set with a structure of (1×T2×K). The output frequency domain feature set has a structure of (T2×K) and is sent to the discriminator as pseudo data to obtain the probability value P1. The probability value P1 has a structure of (2×K). It is worth noting that the corresponding label at this time is the true label 1, not the false label 0. This is opposite to the label used in the training of the discriminator, so as to achieve the effect of confrontation between the generator and the discriminator. The goal is to allow the pseudo data generated by the generator to deceive the discriminator.

[0070] In one example, the loss function is calculated using MAE (Mean absolute error), then:

[0071] The combined first loss LOSS1 and second loss LOSS2 can be specifically calculated by the formula LOSS G=LOSS1+λLOSS2 to calculate and get the joint loss LOSS G , according to the joint loss LOSS G Backpropagation can be performed to update the parameters of the real-time repair generator, obtaining the real-time repair generator after training for that round. Here, λ is a preset value. Since the value of LOSS1 is much smaller than that of LOSS2, it is multiplied here to make the two values ​​similar.

[0072] Furthermore, the performing adversarial training on the repair discriminator by the real-time repair generator after the round of training may include:

[0073] The label frequency domain feature set is input into the repair discriminator as true data to obtain a second probability value; a third loss is calculated based on the second probability value and the false label; the sample frequency domain features are input into the real-time repair generator after training of the round to obtain predicted frequency domain features; the predicted frequency domain features corresponding to the sample frequency domain features for a predetermined number of frames are cached to obtain a predicted frequency domain feature set; the predicted frequency domain feature set is input into the repair discriminator as pseudo data to obtain a third probability value; a fourth loss is calculated based on the third probability value and the true label; and the parameters of the repair discriminator are updated by combining the third loss and the fourth loss.

[0074] Specifically, referring to the "Training of the Discriminator" section in Figure 2, the label frequency domain feature set obtained by caching the label frequency domain feature M' in buffer 2 is input into the repair discriminator as true data to obtain the second probability value P2. According to the second probability value P2 and the false label 0, the third loss LOSS3 can be calculated. The sample frequency domain feature M is input into the real-time repair generator after training of the round (that is, the updated repair generator as shown in Figure 2) to obtain the predicted frequency domain feature M2. The predicted frequency domain features corresponding to the sample frequency domain features of a predetermined number of frames are cached in buffer 3 to obtain a predicted frequency domain feature set. The predicted frequency domain feature set is input into the repair discriminator as pseudo data to obtain the third probability value P3. According to the third probability value P3 and the true label 1, the fourth loss LOSS4 can be calculated. The parameters of the repair discriminator can be updated by back propagation by combining the third loss LOSS3 and the fourth loss LOSS4.

[0075] For example, after M' passes through buffer 2, it is converted into a label frequency domain feature set with a structure of (T2×K), which is fed into the repair discriminator as true data. This results in a second probability value P2, which has a structure of (2×K). At this point, the corresponding label is a false label 0. In one example, the loss function is calculated using MAE (mean absolute error), so:

[0076] After the sample frequency domain features M are processed by the real-time repair generator after training in the round, the output M2 of the (1×1×K) structure is converted into a predicted frequency domain feature set of the (T2×K) structure after passing through buffer 3. The predicted frequency domain feature set is sent to the repair discriminator as pseudo data, and then a third probability value P3 is obtained. The third probability value P3 has a (2×K) structure, and the corresponding label at this time is the true label 1. In an example, the loss function is calculated using the MAE (Mean Absolute Error) method, then:

[0077] The third loss LOSS3 and the fourth loss LOSS4 can be specifically combined by the formula LOSS D =LOSS3+LOSS4 to calculate and get the combined loss LOSS D , according to the joint loss LOSS D Backpropagation is performed to update the parameters of the repair discriminator.

[0078] In one embodiment, the frequency domain repair generator is used to repair the frequency domain features to obtain the repaired frequency domain features, which can specifically include: generating input frequency domain features based on the frequency domain features; using the frequency domain repair generator to encode and decode the input frequency domain features to obtain encoding and decoding features; and averaging the encoding and decoding features to obtain average features; multiplying the average features with the intermediate frames of the input frequency domain features to obtain the repaired frequency domain features.

[0079] The number of frequency domain feature subbands can be K. To generate input frequency domain features based on frequency domain features, frequency domain features of T frames can be taken as input frequency domain features at a time. The input frequency domain features have a structure of (1 × T × K). Using the frequency domain restoration generator, the input frequency domain features can be encoded (Encoder) and decoded (Dencoder) to obtain codec features. The codec features also have a structure of (1 × T × K).

[0080] The codec features are averaged to obtain the average feature, which has a structure of (1×1×K). Furthermore, the average feature is multiplied by the intermediate frame of the input frequency domain features (the intermediate frame is the (1×1×K) structure obtained by taking the input frequency domain features of T / 2 frames) to obtain the repaired frequency domain features.

[0081] In the manner of this embodiment, by means of encoding and decoding processing, averaging and multiplication processing, the frequency domain features can be effectively repaired to obtain repaired frequency domain features, which can greatly improve the audio repair effect.

[0082] Furthermore, in one embodiment, encoding and decoding the input frequency domain features to obtain encoding and decoding features includes:

[0083] Perform a first convolution process on the input frequency domain feature to obtain a first convolution result; perform a first pooling process on the first convolution result to obtain a first pooling result; perform a second convolution process on the first pooling result to obtain a second convolution result; perform a second pooling process on the second convolution result to obtain a second pooling result; perform a third convolution process on the second pooling result to obtain a third convolution result; perform a recurrent neural network encoding process on the third convolution result to obtain an encoding result; splice the encoding result and the third convolution result and perform a first deconvolution process to obtain a first deconvolution result; splice the first deconvolution result and the second convolution result and perform a second deconvolution process to obtain a second deconvolution result; splice the second deconvolution result and the first convolution result and perform a third deconvolution process to obtain a third deconvolution result; perform a fourth convolution process on the third deconvolution result to obtain the encoding and decoding feature.

[0084] In this embodiment, the codec features obtained through multiple rounds of convolution processing, pooling processing and deconvolution processing are used to multiply the intermediate frames of the input frequency domain features, so as to obtain repaired frequency domain features with better quality.

[0085] In one embodiment, referring to FIG3 and FIG4 , the network structure of the frequency domain repair generator is specifically shown in FIG3 . The units or elements in the frequency domain repair generator shown in FIG3 are shown in FIG4 , specifically as follows:

[0086] ① Input layer: represents the input frequency domain feature X. In this method, the frequency domain features of T frames are taken as an input frequency domain feature each time. The input frequency domain feature has a structure of (1×T×K).

[0087] ②Output layer: represents the repaired frequency domain feature Y, which can be the Log-Mel spectrum estimated by the frequency domain repair generator. The repaired frequency domain feature Y has a structure of (1×1×K).

[0088] ③Convolution block: The gradient rectangle in Figures 3 and 4. A convolution block can further include one or more serially connected sub-convolution modules. Each sub-convolution module contains a batch normalization operation (cBN), a 2D convolution operation (Convolution 2D, Conv2d) and an activation function (such as the ReLU activation function).

[0089] ④Pool block: The gray rectangle in Figures 3 and 4 contains a maximum pooling layer (Max pooling).

[0090] ⑤RNN block (recurrent neural network module): The white rectangle in Figures 3 and 4 consists of one or more recurrent neural network (RNN) layers connected in series, with the output of the previous layer being the input of the next layer. The RNN can be GRU, LSTM, BLSTM, etc.

[0091] ⑥Deconvolution block: The black rectangle in Figures 3 and 4 is similar to the convolution block in ③. A deconvolution block can further include one or more serially connected sub-deconvolution modules. Each sub-deconvolution module contains a batch normalization operation (Complex Batch Normalization, cBN), a 2D convolution operation (Convolution 2D, Conv2d) and an activation function (such as the Relu activation function). However, as shown in Figure 4, the last layer in the last sub-deconvolution module is a 2D deconvolution operation (Deconvolution 2D, Deconv2d).

[0092] ⑦Mean block: averages the dimensions of the time frame. The input is a (1×T×K) structure, and the output is a (1×1×K) structure.

[0093] ⑧Concatenate: It means concatenation. The dotted arrows in Figures 3 and 4 can be used to concatenate according to the first dimension of the input audio signal (that is, the dimension corresponding to the number of convolution kernels).

[0094] ⑨Multiplication: The multiplication signs in Figures 3 and 4 can specifically multiply elements at the same position. For example, the output of the multiplication module in the structure diagram of Figure 3 is Y.

[0095] Specifically, as shown in FIG3 , in this embodiment, the network structure of the frequency domain restoration generator includes an encoder part and a decoder part.

[0096] The encoding part includes a first convolution module 201, a first pooling module 202, a second convolution module 203, a second pooling module 204, and a third convolution module 205. The decoding part includes a recurrent neural network module 206, a first deconvolution module 207, a second deconvolution module 208, a third deconvolution module 209, and a fourth convolution module 210.

[0097] Specifically, in the first convolution module 201, the input frequency domain feature can be subjected to a first convolution process to obtain a first convolution result; in the first pooling module 202, the first convolution result can be subjected to a first pooling process to obtain a first pooling result; in the second convolution module 203, the first pooling result can be subjected to a second convolution process to obtain a second convolution result; in the second pooling module 204, the second convolution result can be subjected to a second pooling process to obtain a second pooling result; in the third convolution module 205, the second pooling result can be subjected to a third convolution process to obtain a third convolution result; in the recurrent neural network module 206, the third convolution result can be subjected to a third convolution process. The result is subjected to recurrent neural network encoding processing to obtain an encoding result; in the first deconvolution module 207, the encoding result and the third convolution result can be spliced ​​together and then subjected to a first deconvolution processing to obtain a first deconvolution result; in the second deconvolution module 208, the first deconvolution result and the second convolution result can be spliced ​​together and then subjected to a second deconvolution processing to obtain a second deconvolution result; in the third deconvolution module 209, the second deconvolution result and the first convolution result can be spliced ​​together and then subjected to a third deconvolution processing to obtain a third deconvolution result; in the fourth convolution module 210, the third deconvolution result can be subjected to a fourth convolution processing to obtain the encoding and decoding features.

[0098] Furthermore, in the averaging module 211, the encoding and decoding features may be averaged to obtain an average feature; and in the multiplication module 212, the average feature may be multiplied by the intermediate frame of the input frequency domain feature to obtain the repaired frequency domain feature Y.

[0099] Furthermore, in one embodiment, the network structure of the repair discriminator is shown in FIG5 . The repair discriminator may include: a convolution module 301, a basic discriminator module 302, a first re-discrimination module 303, a second re-discriminator module 304, a third re-discriminator module 305, and an absolute value module 306, which are connected in series. Further referring to FIG6 , Convolution Block 1 includes a one-dimensional convolution operation (convolution 1D, Conv1d), a weight normalization (WN), and an activation function (such as the LeakyRelu activation function). Basic Discriminator Block 1 is directly composed of multiple convolution blocks 1 connected in series, but the last one does not have an activation function (it only includes a one-dimensional convolution operation (Conv1d) and a weight normalization (weight Norm)). Each Res Discriminator Block 1 is composed of a module that adds two Basic Discriminator Block 1s, a Conv1d, and a weight normalization (weight Norm). abs (absolute value module): calculates the absolute value. Since the output is a probability value, it must be guaranteed to be non-negative.

[0100] The restoration discriminator processes the input data C in sequence through the first convolution module 301, the basic identification module 302, the first re-identification module 303, the second re-identification module 304, the third re-identification module 305, and the absolute value module 306, to obtain an output probability value P. Using the restoration discriminator structure of this embodiment for adversarial training, the resulting frequency domain restoration generator can further improve audio restoration results.

[0101] In one embodiment, the signal conversion generator is obtained by training in the following manner: obtaining a conversion training set, wherein the conversion training set includes different preset frequency domain features and corresponding labeled time domain data; performing multiple rounds of iterative adversarial training on the second generative adversarial network based on the preset frequency domain features and corresponding labeled time domain data in the conversion training set to obtain a trained signal conversion generator. Wherein, the iterative adversarial training of the second generative adversarial network in each round includes: using the preset frequency domain features and the labeled time domain data as the input data and label data of the real-time conversion generator respectively, and adversarially training the real-time conversion generator based on the conversion discriminator to obtain the real-time conversion generator after training of the round; and adversarially training the conversion discriminator based on the real-time conversion generator after training of the round.

[0102] The labeled time domain data can be speech from a TV source, or audio from scenes such as movies, songs, news, and sports. The frequency domain features extracted from the labeled time domain data can be used as the preset frequency domain features.

[0103] A second generative adversarial network can be formed by the real-time conversion generator and the conversion discriminator. According to the preset frequency domain features and the corresponding label time domain data in the conversion training set, the second generative adversarial network is subjected to multiple rounds of iterative adversarial training until the predetermined training end conditions are met (for example, the loss value converges and the repair effect meets the predetermined requirements). Then, the trained second generative adversarial network is obtained, and the real-time conversion generator in the trained second generative adversarial network is the trained real-time conversion generator.

[0104] A second generative adversarial network can be formed by the real-time conversion generator and the conversion discriminator, which can enable the trained real-time conversion generator (i.e., signal conversion generator) to perform good time domain conversion on frequency domain features.

[0105] In one embodiment, the adversarial training of the real-time transformation generator based on the transformation discriminator to obtain the trained real-time transformation generator of the round includes:

[0106] Input the preset frequency domain features into the real-time conversion generator to obtain output time domain data; cache the output time domain data corresponding to the preset frequency domain features for a target number of frames to obtain an output time domain data set; calculate the fifth loss based on the output time domain data set and the label time domain data set, wherein the label time domain data set includes the label time domain data corresponding to the preset frequency domain features for a target number of frames; perform short-time Fourier transform on the output frequency domain feature set and the label time domain data set respectively and calculate the sixth loss; input the transformed data after short-time Fourier transform on the output frequency domain feature set as pseudo data into the frequency domain discriminator to obtain the fourth loss. probability value; calculating the seventh loss based on the fourth probability value and the true label; inputting the output frequency domain feature set as pseudo data into the time domain discriminator to obtain the first output feature and the fifth probability value; calculating the eighth loss based on the fifth probability value and the true label; inputting the label time domain data corresponding to the preset frequency domain feature as true data into the time domain discriminator to obtain the second output feature; calculating the ninth loss based on the second output feature and the first output feature; combining the fifth loss, the sixth loss, the seventh loss, the eighth loss and the ninth loss to update the real-time conversion generator to obtain the real-time conversion generator after training of the round.

[0107] Specifically, referring to the "Generator Training" section in Figure 7 , preset frequency domain features z (which can be the frequency domain features of the extracted labeled time domain data Z, such as the Log-Mel spectrum) are input into the real-time conversion generator to obtain output time domain data z1. Buffer 4 can cache a target number of frames (which can be pre-specified based on actual conditions) of output time domain data z1 to obtain an output frequency domain feature set.

[0108] At the same time, the buffer 5 can cache the label time domain data Z corresponding to the preset frequency domain features of the target frame to obtain a label frequency domain feature set. A fifth loss LOSS5 can be calculated based on the output frequency domain feature set and the label frequency domain feature set. A sixth loss LOSS6 can be calculated by performing a short-time Fourier transform (SFT) on the output frequency domain feature set and the label time domain data set.

[0109] The transformed data after short-time Fourier transform (stft) of the output frequency domain feature set is input into the frequency domain discriminator as pseudo data to obtain a fourth probability value P4; the seventh loss LOSS7 can be calculated based on the fourth probability value P4 and the true label 1.

[0110] The output frequency domain feature set is input into the time domain discriminator as pseudo data to obtain the first output feature (feat-fake) and the fifth probability value P5; the eighth loss LOSS8 can be calculated based on the fifth probability value P5 and the true label 1.

[0111] The label time domain data Z corresponding to the preset frequency domain feature z is input into the time domain discriminator as the true data to obtain the second output feature (feat-real); the ninth loss LOSS9 is calculated based on the second output feature and the first output feature.

[0112] Combine the fifth loss, sixth loss, seventh loss, eighth loss and ninth loss to perform back propagation, update the parameters of the real-time conversion generator, and obtain the real-time conversion generator after training for the round. Specifically, LOSS can be calculated. G2 =LOSS5+LOSS6+αLOSS7+βLOSS8+γLOSS9, according to LOSS G2 Back propagation is performed to update the parameters of the real-time conversion generator to obtain the real-time conversion generator after training for the round, where α, β, and γ are coefficients that can be set according to actual conditions.

[0113] The input feature z to the real-time transform generator has a structure of (1 × T3 × K), while the output z1 of the real-time transform generator has a structure of (1 × τ × 1), where τ = T3 * hop_length, where hop_length is the length of each frame shift when extracting frequency-domain features (e.g., log-Mel spectrum). Therefore, T3 cannot be too large. When used alone, it can be controlled within a predetermined threshold (e.g., 0.25 seconds or 0.144 seconds) of frames.

[0114] By setting up many buffers to cache during the training process, the real-time generator can be offline during training to achieve better results. The output time domain data z1 of length τ is cached into an output time domain data set of length τ2 (that is, the target number of frames). Every time the cache reaches the length τ2, LOSS is calculated once. G2 , perform backpropagation to update the parameters of the generator.

[0115] Furthermore, in one embodiment, the real-time transformation generator based on the round of training performs adversarial training on the time domain discriminator, including:

[0116] Input the labeled time domain data set as true data into the time domain discriminator to obtain a sixth probability value, and calculate the tenth loss based on the sixth probability value and the true label; input the transformed data after short-time Fourier transform of the labeled time domain data set into the time domain discriminator as true data to obtain a seventh probability value; calculate the eleventh loss based on the seventh probability value and the true label; input the preset frequency domain features into the real-time conversion generator after training of the round to obtain predicted time domain data; cache the predicted time domain data corresponding to the preset frequency domain features for a target number of frames to obtain a predicted time domain data set; input the predicted time domain data set as pseudo data into the time domain discriminator to obtain an eighth probability value; calculate the twelfth loss based on the eighth probability value and the false label; input the transformed data after short-time Fourier transform of the predicted time domain data set into the time domain discriminator as pseudo data to obtain a ninth probability value; calculate the thirteenth loss based on the ninth probability value and the false label; and update the parameters of the time domain discriminator by combining the tenth loss, the eleventh loss, the twelfth loss, and the thirteenth loss.

[0117] Specifically, referring to the "Training of the Discriminator" part of Figure 8, the labeled time domain data set (i.e., the labeled time domain data Z cached by buffer 5) is input as true data into the time domain discriminator to obtain the sixth probability value P6, and the tenth loss LOSS10 is calculated based on the sixth probability value P6 and the true label 1.

[0118] The transformed data after short-time Fourier transform (stft) of the label time domain data set is input into the time domain discriminator as true data to obtain the seventh probability value P7; the eleventh loss LOSS11 is calculated based on the seventh probability value P7 and the true label.

[0119] The preset frequency domain feature z is input into the real-time conversion generator after training of the round (i.e., the updated conversion generator shown in Figure 8) to obtain the predicted time domain data z2; the predicted time domain data z2 corresponding to the preset frequency domain feature z of the target number of frames is cached by the buffer 6 to obtain a predicted time domain data set; the predicted time domain data set is input into the time domain discriminator as pseudo data to obtain the eighth probability value P8; the twelfth loss LOSS12 can be calculated based on the eighth probability value P8 and the false label 0.

[0120] The transformed data after short-time Fourier transform (stft) of the predicted time domain data set is input into the time domain discriminator as pseudo data to obtain the ninth probability value P9; the thirteenth loss LOSS13 can be calculated based on the ninth probability value P9 and the false label 0.

[0121] The parameters of the time domain discriminator can be updated by combining the tenth loss, the eleventh loss, the twelfth loss and the thirteenth loss for back propagation. D2 =LOSS10+LOSS11+LOSS12+LOSS13 to calculate the combined loss LOSS D2 Back propagation is performed to update the parameters of the time domain discriminator.

[0122] The input data to the time domain discriminator is in the form of (1×τ²), and the output is in the form of (1×δ). The input data to the frequency domain discriminator is framed and STFT-ed, with a structure of (T4×N), where T4 = τ² / hop_length, and N is the length of the FFT. The output is in the form of (2×N).

[0123] To facilitate the implementation of the audio repair method provided in the embodiments of this application, the embodiments of this application also provide an audio repair device based on the aforementioned audio repair method. The meanings of the terms herein are the same as those in the aforementioned audio repair method. For specific implementation details, please refer to the description in the method embodiments. Figure 9 shows a block diagram of an audio repair device according to one embodiment of the present application.

[0124] As shown in Figure 9, the audio repair device 300 may include: an acquisition module 310 can be used to obtain the audio to be repaired; an extraction module 320 can be used to extract frequency domain features of the audio to be repaired to obtain the frequency domain features of the audio to be repaired; a repair module 330 can be used to use a frequency domain repair generator to repair the frequency domain features to obtain repaired frequency domain features, and the frequency domain repair generator is a trained real-time repair generator obtained by adversarial training of a real-time repair generator and a repair discriminator; a conversion module 340 can be used to use a signal conversion generator to perform time domain conversion processing on the repair frequency domain features to obtain repaired audio, and the signal conversion generator is a trained real-time conversion generator obtained by adversarial training of a real-time conversion generator and a conversion discriminator.

[0125] In some embodiments of the present application, the repair module is used to: generate input frequency domain features based on the frequency domain features; use the frequency domain repair generator to encode and decode the input frequency domain features to obtain codec features; and average the codec features to obtain average features; multiply the average features with the intermediate frames of the input frequency domain features to obtain the repaired frequency domain features.

[0126] In some embodiments of the present application, the repair module is used to: perform a first convolution process on the input frequency domain feature to obtain a first convolution result; perform a first pooling process on the first convolution result to obtain a first pooling result; perform a second convolution process on the first pooling result to obtain a second convolution result; perform a second pooling process on the second convolution result to obtain a second pooling result; perform a third convolution process on the second pooling result to obtain a third convolution result; perform a recurrent neural network encoding process on the third convolution result to obtain an encoding result; perform a first deconvolution process on the encoding result and the third convolution result after splicing the encoding result and the third convolution result to obtain a first deconvolution result; perform a second deconvolution process on the first deconvolution result and the second convolution result after splicing the first deconvolution result to obtain a second deconvolution result; perform a third deconvolution process on the second deconvolution result and the first convolution result after splicing the second deconvolution result to obtain a third deconvolution result; perform a fourth convolution process on the third deconvolution result to obtain the encoding and decoding feature.

[0127] In some embodiments of the present application, the device also includes a first training module, which is used to: obtain a repair training set, wherein the repair training set includes different sample frequency domain features and corresponding label frequency domain features, the label frequency domain features are extracted from the sample audio, and the sample frequency domain features are extracted from the damaged audio corresponding to the sample audio; based on the sample frequency domain features and the corresponding label frequency domain features in the repair training set, perform multiple rounds of iterative adversarial training on the first generative adversarial network to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes: using the sample frequency domain features and the label frequency domain features as the input data and label data of the real-time repair generator respectively, and adversarially training the real-time repair generator based on the repair discriminator to obtain the real-time repair generator after training of the round; and adversarially training the repair discriminator based on the real-time repair generator after training of the round.

[0128] In some embodiments of the present application, the first training module is used to: input the sample frequency domain features into the real-time repair generator to obtain output frequency domain features; cache the output frequency domain features corresponding to a predetermined number of frames of the sample frequency domain features to obtain an output frequency domain feature set; calculate a first loss based on the output frequency domain feature set and a label frequency domain feature set, wherein the label frequency domain feature set includes label frequency domain features corresponding to a predetermined number of frames of the sample frequency domain features; input the output frequency domain feature set as pseudo data into the repair discriminator to obtain a first probability value; calculate a second loss based on the first probability value and the true label; and update the parameters of the real-time repair generator by combining the first loss and the second loss to obtain the real-time repair generator after training for the round.

[0129] In some embodiments of the present application, the first training module is used to: input the label frequency domain feature set as true data into the repair discriminator to obtain a second probability value; calculate a third loss based on the second probability value and the false label; input the sample frequency domain features into the real-time repair generator after training of the round to obtain predicted frequency domain features; cache the predicted frequency domain features corresponding to the sample frequency domain features for a predetermined number of frames to obtain a predicted frequency domain feature set; input the predicted frequency domain feature set as pseudo data into the repair discriminator to obtain a third probability value; calculate a fourth loss based on the third probability value and the true label; and update the parameters of the repair discriminator by combining the third loss and the fourth loss.

[0130] In some embodiments of the present application, the device also includes a second training module, which is used to: obtain a conversion training set, which includes different preset frequency domain features and corresponding label time domain data; perform multiple rounds of iterative adversarial training on the second generative adversarial network based on the preset frequency domain features and the corresponding label time domain data in the conversion training set to obtain a trained signal conversion generator; wherein, the iterative adversarial training of the second generative adversarial network in each round includes: using the preset frequency domain features and the label time domain data as the input data and label data of the real-time conversion generator respectively, and adversarially training the real-time conversion generator based on the conversion discriminator to obtain the real-time conversion generator after training of the round; and, performing adversarial training on the conversion discriminator based on the real-time conversion generator after training of the round.

[0131] In some embodiments of the present application, the second training module is used to: input the preset frequency domain features into the real-time conversion generator to obtain output time domain data; cache the output time domain data corresponding to the preset frequency domain features for a target number of frames to obtain an output time domain data set; calculate the fifth loss based on the output time domain data set and the label time domain data set, wherein the label time domain data set includes the label time domain data corresponding to the preset frequency domain features for a target number of frames; calculate the sixth loss after performing short-time Fourier transform on the output frequency domain feature set and the label time domain data set; use the transformed data after the short-time Fourier transform of the output frequency domain feature set as pseudo data Input the frequency domain discriminator to obtain a fourth probability value; calculate the seventh loss based on the fourth probability value and the true label; input the output frequency domain feature set as pseudo data into the time domain discriminator to obtain a first output feature and a fifth probability value; calculate the eighth loss based on the fifth probability value and the true label; input the labeled time domain data corresponding to the preset frequency domain feature as true data into the time domain discriminator to obtain a second output feature; calculate the ninth loss based on the second output feature and the first output feature; combine the fifth loss, sixth loss, seventh loss, eighth loss and ninth loss to update the real-time conversion generator to obtain the real-time conversion generator after training of the round.

[0132] In some embodiments of the present application, the second training module is configured to: input the labeled time domain dataset z as true data into the time domain discriminator to obtain a sixth probability value, and calculate a tenth loss based on the sixth probability value and the true label; input the transformed data after short-time Fourier transform of the labeled time domain dataset into the time domain discriminator as true data to obtain a seventh probability value; calculate an eleventh loss based on the seventh probability value and the true label; input the preset frequency domain features into the round of post-training real-time conversion generator to obtain predicted time domain data; cache the predicted time domain data corresponding to the preset frequency domain features for a target number of frames to obtain a predicted time domain dataset; input the predicted time domain dataset as pseudo data into the time domain discriminator to obtain an eighth probability value; calculate a twelfth loss based on the eighth probability value and the false label; input the transformed data after short-time Fourier transform of the predicted time domain dataset into the time domain discriminator as pseudo data to obtain a ninth probability value; calculate a thirteenth loss based on the ninth probability value and the false label; and update the parameters of the time domain discriminator by combining the tenth loss, the eleventh loss, the twelfth loss, and the thirteenth loss.

[0133] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiment of the application, the features and functions of two or more modules or units described above can be concretized in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be concretized.

[0134] In addition, an embodiment of the present application further provides an electronic device, as shown in FIG10 . FIG10 shows a block diagram of an electronic device according to an embodiment of the present application. Specifically:

[0135] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the electronic device structure shown in FIG10 does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0136] Processor 401 is the control center of the electronic device, connecting the various components of the entire computer device using various interfaces and lines. By running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402, it performs various computer device functions and processes data, thereby monitoring the electronic device as a whole. Optionally, processor 401 may include one or more processing cores; preferably, processor 401 may integrate an application processor and a modem processor, wherein the application processor primarily processes the operating system, user interfaces, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0137] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0138] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0139] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0140] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail here. Specifically, in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more computer programs into the memory 402 according to the following instructions, and the processor 401 will run the computer programs stored in the memory 402, thereby realizing the various functions of the aforementioned embodiments of the present application. For example, the processor 401 may perform the following steps:

[0141] Acquire the audio to be repaired; extract the frequency domain features of the audio to be repaired to obtain the frequency domain features of the audio to be repaired; use a frequency domain repair generator to repair the frequency domain features to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by adversarial training of a real-time repair generator and a repair discriminator; use a signal conversion generator to perform time domain conversion processing on the repaired frequency domain features to obtain repaired audio, wherein the signal conversion generator is a trained real-time conversion generator obtained by adversarial training of a real-time conversion generator and a conversion discriminator.

[0142] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by a computer program, or by controlling related hardware through a computer program. The computer program may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0143] To this end, an embodiment of the present application further provides a storage medium storing a computer program, which can be loaded by a processor to execute the steps of any method provided in the embodiment of the present application.

[0144] The storage medium may be a computer-readable storage medium, and the storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0145] Since the computer program stored in the storage medium can execute the steps of any method provided in the embodiments of the present application, the beneficial effects that can be achieved by the method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0146] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed herein.

[0147] It should be understood that the present application is not limited to the embodiments that have been described above and shown in the accompanying drawings, but various modifications and changes may be made without departing from the scope thereof.

Claims

1. An audio repair method, wherein: include: Get the audio to be repaired; Extracting frequency domain features of the audio to be repaired to obtain frequency domain features of the audio to be repaired; A frequency domain repair generator is used to repair the frequency domain features to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by adversarially training a real-time repair generator and a repair discriminator; A signal conversion generator is used to perform time domain conversion processing on the repaired frequency domain features to obtain repaired audio. The signal conversion generator is a trained real-time conversion generator obtained by adversarially training a real-time conversion generator and a conversion discriminator.

2. The method according to claim 1, wherein: The frequency domain feature is repaired by a frequency domain repair generator to obtain a repaired frequency domain feature, including: Generate input frequency domain features based on the frequency domain features; Using the frequency domain restoration generator, encoding and decoding the input frequency domain features to obtain encoding and decoding features; and, Averaging the encoding and decoding features to obtain average features; The average feature is multiplied by the intermediate frame of the input frequency domain feature to obtain the repaired frequency domain feature.

3. The method according to claim 2, wherein: The encoding and decoding processing of the input frequency domain features to obtain encoding and decoding features includes: Performing a first convolution process on the input frequency domain feature to obtain a first convolution result; Performing a first pooling process on the first convolution result to obtain a first pooling result; Performing a second convolution process on the first pooling result to obtain a second convolution result; Performing a second pooling process on the second convolution result to obtain a second pooling result; Performing a third convolution process on the second pooling result to obtain a third convolution result; Performing recurrent neural network encoding processing on the third convolution result to obtain an encoding result; The encoding result and the third convolution result are concatenated and then subjected to a first deconvolution process to obtain a first deconvolution result; Performing a second deconvolution process on the first deconvolution result and the second deconvolution result to obtain a second deconvolution result; Performing a third deconvolution process on the second deconvolution result and the first deconvolution result to obtain a third deconvolution result; A fourth convolution process is performed on the third deconvolution result to obtain the encoding and decoding feature.

4. The method according to claim 1, wherein: The frequency domain restoration generator is trained in the following manner: Acquire a restoration training set, wherein the restoration training set includes different sample frequency domain features and corresponding label frequency domain features, wherein the label frequency domain features are extracted from sample audio, and the sample frequency domain features are extracted from damaged audio corresponding to the sample audio; According to the sample frequency domain features and the corresponding label frequency domain features in the repair training set, the first generative adversarial network is subjected to multiple rounds of iterative adversarial training to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes: Using the sample frequency domain features and the label frequency domain features as input data and label data of the real-time repair generator respectively, training the real-time repair generator adversarially based on the repair discriminator to obtain the real-time repair generator after the training of the round; and The repair discriminator is adversarially trained based on the real-time repair generator after the round of training.

5. The method according to claim 4, wherein: The method of training the real-time repair generator based on the repair discriminator adversarially to obtain the round of trained real-time repair generator includes: Inputting the sample frequency domain features into the real-time repair generator to obtain output frequency domain features; Buffering the output frequency domain features corresponding to the sample frequency domain features of a predetermined number of frames to obtain an output frequency domain feature set; Calculating a first loss according to the output frequency domain feature set and the label frequency domain feature set, wherein the label frequency domain feature set includes label frequency domain features corresponding to a predetermined number of the sample frequency domain features; Inputting the output frequency domain feature set as pseudo data into the repair discriminator to obtain a first probability value; Calculate a second loss based on the first probability value and the true label; The first loss and the second loss are combined to update the parameters of the real-time repair generator to obtain the real-time repair generator after training for the round.

6. The method according to claim 5, wherein: The real-time repair generator after the round of training performs adversarial training on the repair discriminator, comprising: Inputting the label frequency domain feature set as true data into the repair discriminator to obtain a second probability value; Calculate a third loss according to the second probability value and the false label; Inputting the sample frequency domain features into the round of post-training real-time repair generator to obtain predicted frequency domain features; Buffering the predicted frequency domain features corresponding to the sample frequency domain features of a predetermined number of frames to obtain a predicted frequency domain feature set; Inputting the predicted frequency domain feature set as pseudo data into the repair discriminator to obtain a third probability value; Calculate a fourth loss according to the third probability value and the true label; The parameters of the restoration discriminator are updated by combining the third loss and the fourth loss.

7. The method according to claim 1, wherein: The signal conversion generator is trained in the following manner: Acquire a conversion training set, wherein the conversion training set includes different preset frequency domain features and corresponding label time domain data; According to the preset frequency domain features and the corresponding label time domain data in the conversion training set, multiple rounds of iterative adversarial training are performed on the second generative adversarial network to obtain a trained signal conversion generator; wherein the iterative adversarial training of the second generative adversarial network in each round includes: Using the preset frequency domain features and the label time domain data as input data and label data of the real-time conversion generator respectively, training the real-time conversion generator adversarially based on the conversion discriminator to obtain the real-time conversion generator after training of the round; and The transformation discriminator is adversarially trained based on the round of trained real-time transformation generator.

8. The method according to claim 7, wherein: The method of training the real-time conversion generator based on the conversion discriminator adversarially to obtain the round of trained real-time conversion generator includes: Inputting the preset frequency domain features into the real-time conversion generator to obtain output time domain data; Cache the output time domain data corresponding to the preset frequency domain features of the target number of frames to obtain an output time domain data set; Calculating a fifth loss according to the output time domain data set and the label time domain data set, wherein the label time domain data set includes label time domain data corresponding to the preset frequency domain features of the target frame number; Calculate the sixth loss after performing short-time Fourier transform on the output frequency domain feature set and the label time domain data set respectively; Inputting transformed data after short-time Fourier transform of the output frequency domain feature set as pseudo data into a frequency domain discriminator to obtain a fourth probability value; Calculate a seventh loss according to the fourth probability value and the true label; Input the output frequency domain feature set as pseudo data into a time domain discriminator to obtain a first output feature and a fifth probability value; Calculate an eighth loss according to the fifth probability value and the true label; Inputting the label time domain data corresponding to the preset frequency domain feature as true data into the time domain discriminator to obtain a second output feature; Calculate a ninth loss according to the second output feature and the first output feature; The fifth loss, the sixth loss, the seventh loss, the eighth loss and the ninth loss are combined to update the real-time conversion generator to obtain the real-time conversion generator after training of the round.

9. The method according to claim 8, wherein: The real-time conversion generator after the round of training performs adversarial training on the time domain discriminator, comprising: Input the labeled time domain data set z as true data into the time domain discriminator to obtain a sixth probability value, and calculate a tenth loss based on the sixth probability value and the true label; Inputting transformed data after short-time Fourier transform of the label time domain data set as true data into the time domain discriminator to obtain a seventh probability value; Calculating an eleventh loss according to the seventh probability value and the true label; Inputting the preset frequency domain features into the round of trained real-time conversion generator to obtain predicted time domain data; Cache the predicted time domain data corresponding to the preset frequency domain features of the target number of frames to obtain a predicted time domain data set; Inputting the predicted time domain data set as pseudo data into the time domain discriminator to obtain an eighth probability value; Calculating a twelfth loss according to the eighth probability value and the false label; Inputting transformed data after short-time Fourier transform of the predicted time domain data set as pseudo data into the time domain discriminator to obtain a ninth probability value; Calculate a thirteenth loss according to the ninth probability value and the false label; The tenth loss, the eleventh loss, the twelfth loss and the thirteenth loss are combined to update the parameters of the time domain discriminator.

10. The method according to claim 1, wherein: The extracting frequency domain features of the audio to be repaired to obtain the frequency domain features of the audio to be repaired includes: Upsampling the one-dimensional audio signal of the audio to be repaired to a target sampling rate to obtain an upsampled audio; Extract the logarithmic Mel spectrum feature of the upsampled audio, and use the logarithmic Mel spectrum feature as the frequency domain feature of the audio to be repaired.

11. The method according to claim 5, wherein: The combining the first loss and the second loss to update the parameters of the real-time repair generator comprises: According to the formula LOSS G =LOSS1+λLOSS2 to calculate and get the joint loss LOSS G , wherein LOSS1 is the first loss, LOSS2 is the second loss, and λ is a preset value; Back-propagation is performed based on the joint loss to update the parameters of the real-time restoration generator.

12. The method according to claim 6, wherein: The combining the third loss and the fourth loss to update the parameters of the repair discriminator includes: According to the formula LOSS D =LOSS3+LOSS4 to calculate and get the combined loss LOSS D , wherein LOSS3 is the third loss, and LOSS4 is the fourth loss; Back-propagation is performed based on the joint loss to update the parameters of the restoration discriminator.

13. The method according to claim 8, wherein: The combining the fifth loss, the sixth loss, the seventh loss, the eighth loss and the ninth loss to update the real-time conversion generator comprises: Calculate LOSS G2 =LOSS5+LOSS6+αLOSS7+βLOSS8+γLOSS9, and according to LOSS G2 Back propagation is performed to update the parameters of the real-time conversion generator, wherein α, β, γ are preset coefficients, and LOSS5, LOSS6, LOSS7, LOSS8, and LOSS9 are the fifth loss, the sixth loss, the seventh loss, the eighth loss, and the ninth loss, respectively.

14. The method according to claim 9, wherein: The combining the tenth loss, the eleventh loss, the twelfth loss and the thirteenth loss to update the parameters of the time domain discriminator includes: According to the formula LOSS D2 =LOSS10+LOSS11+LOSS12+LOSS13 to calculate the combined loss LOSS D2 Back propagation is performed to update the parameters of the time domain discriminator, wherein LOSS10, LOSS11, LOSS12, and LOSS13 are the tenth loss, the eleventh loss, the twelfth loss, and the thirteenth loss, respectively.

15. An audio repair device, wherein: include: An acquisition module, used to acquire the audio to be repaired; An extraction module, used for extracting frequency domain features of the audio to be repaired to obtain frequency domain features of the audio to be repaired; A repair module, used to repair the frequency domain features using a frequency domain repair generator to obtain repaired frequency domain features, wherein the frequency domain repair generator is a trained real-time repair generator obtained by adversarially training a real-time repair generator and a repair discriminator; The conversion module is used to use a signal conversion generator to perform time domain conversion processing on the repaired frequency domain features to obtain repaired audio. The signal conversion generator is a trained real-time conversion generator obtained by adversarially training a real-time conversion generator and a conversion discriminator.

16. The device according to claim 15, wherein: The repair module is used to: generate input frequency domain features based on the frequency domain features; use the frequency domain repair generator to encode and decode the input frequency domain features to obtain encoding and decoding features; And, averaging the encoding and decoding features to obtain average features; The average feature is multiplied by the intermediate frame of the input frequency domain feature to obtain the repaired frequency domain feature.

17. The device according to claim 15, wherein: The repair module is used to: perform a first convolution process on the input frequency domain feature to obtain a first convolution result; perform a first pooling process on the first convolution result to obtain a first pooling result; perform a second convolution process on the first pooling result to obtain a second convolution result; perform a second pooling process on the second convolution result to obtain a second pooling result; Performing a third convolution process on the second pooling result to obtain a third convolution result; performing a recurrent neural network encoding process on the third convolution result to obtain an encoding result; and performing a first deconvolution process after splicing the encoding result and the third convolution result to obtain a first deconvolution result; Performing a second deconvolution process on the first deconvolution result and the second deconvolution result to obtain a second deconvolution result; Performing a third deconvolution process on the second deconvolution result and the first deconvolution result to obtain a third deconvolution result; A fourth convolution process is performed on the third deconvolution result to obtain the encoding and decoding feature.

18. The device according to claim 15, wherein: The device also includes a first training module, which is used to: obtain a repair training set, wherein the repair training set includes different sample frequency domain features and corresponding label frequency domain features, wherein the label frequency domain features are extracted from sample audio, and the sample frequency domain features are extracted from damaged audio corresponding to the sample audio; perform multiple rounds of iterative adversarial training on the first generative adversarial network according to the sample frequency domain features and the corresponding label frequency domain features in the repair training set to obtain a trained real-time repair generator; wherein the iterative adversarial training of the first generative adversarial network in each round includes: using the sample frequency domain features and the label frequency domain features as input data and label data of the real-time repair generator respectively, adversarially training the real-time repair generator based on the repair discriminator to obtain the real-time repair generator after training of the round; and, adversarially training the repair discriminator based on the real-time repair generator after training of the round.

19. A storage medium, wherein: A computer program is stored thereon, and when the computer program is executed by a processor of a computer, the computer is caused to execute the method according to any one of claims 1 to 14.

20. An electronic device, wherein: include: a memory storing a computer program; A processor reads a computer program stored in a memory to execute the method according to any one of claims 1 to 14.

Images