Training method and device of audio packet loss compensation model and computer equipment

Abstract

The invention relates to an audio packet loss compensation model training method and device, computer equipment, a storage medium and a computer program product. The training method of the audio packet loss compensation model comprises the following steps: performing iterative training on an original generative adversarial network based on an audio training set to obtain a trained generative adversarial network, and defining calculation related to a compensation signal for generating a packet loss frame in a generator; and reconstructing the generator of the generative adversarial network according to calculation related to the compensation signal for generating the packet loss frame in the generator to obtain an audio packet loss compensation model. Since the audio packet loss compensation model is obtained based on calculation and reconstruction related to the generation of the compensation signal of the packet loss frame in the generator, the audio packet loss compensation model only needs to carry out calculation related to the generation of the compensation signal when generating the compensation signal of the packet loss frame, the calculation amount of irrelevant calculation is reduced, the model redundancy is reduced, and meanwhile, the calculation efficiency is improved. And the speed of packet loss compensation is improved, so that the real-time performance of a packet loss compensation algorithm can be improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a training method, apparatus, computer equipment, storage medium and computer program product for an audio packet loss compensation model. Background Art

[0002] During the wireless audio transmission process, due to bandwidth or jitter, the network delay of the data packet is unstable, resulting in packet loss. Currently, netEQ uses dynamic jitter buffer, FEC, PLC and other algorithms to address packet loss caused by network jitter delay, while maintaining high-quality calls and taking into account low data latency. Among them, the two major modules are MCU and DSP. The MCU will perform DSP signal processing based on network delay and cache decisions. DSP processing includes acceleration and deceleration, packet loss compensation and other strategies. When the network jitter is large and the bandwidth is limited, packet loss is severe. When packet loss retransmission, in-band and out-of-band FEC, and acceleration and deceleration strategies cannot obtain audio playback, it is necessary to predict packet loss based on historical data, that is, packet loss compensation.

[0003] There are many packet loss compensation schemes, which can be summarized into three directions: packet loss compensation based on insertion, packet loss compensation based on reconstruction, and packet loss compensation based on interpolation. Currently, the more common scheme is the scheme based on interpolation, but the packet loss compensation based on interpolation has the problem of poor real-time performance of the algorithm. Summary of the invention

[0004] Based on this, it is necessary to provide a training method, device, computer equipment, computer-readable storage medium and computer program product for an audio packet loss compensation model that can improve real-time performance in response to the above-mentioned technical problems.

[0005] In a first aspect, the present application provides a method for training an audio packet loss compensation model. The method comprises:

[0006] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0007] Determining calculations in a trained generator of the generative adversarial network related to generating the compensation signal for the packet loss frame;

[0008] According to the calculation in the generator related to generating the compensation signal of the packet loss frame, the generator is reconstructed to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal of the audio packet loss frame.

[0009] In one of the embodiments, calculations in the generator that are irrelevant to the compensation signal for generating the packet loss frame are determined, the irrelevant calculations in the generator are deleted, and an audio packet loss compensation model is obtained based on the retained calculations related to the compensation signal for generating the packet loss frame.

[0010] In one of the embodiments, an audio training set is obtained, the audio training set comprising audio data and audio tags of the audio data, the audio tags are used to identify packet loss audio and non-packet loss audio, and the audio training set also comprises complete audio of the packet loss audio;

[0011] Inputting the audio data in the audio training set into a generator of a generative adversarial network, wherein the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio label;

[0012] Obtaining compensated audio according to the compensation signal and the packet-lost audio;

[0013] Inputting the compensated audio into the discriminator of the generative adversarial network to obtain a discrimination result;

[0014] A discriminator loss is obtained according to the discrimination result, the difference between the compensated audio and the complete audio, and a model parameter of the generative adversarial network is adjusted according to the discriminator loss.

[0015] In one embodiment, inputting the audio data in the audio training set into the generator of the original generative adversarial network includes:

[0016] Converting the audio data in the audio training set from the time domain to the frequency domain to obtain an audio frequency domain signal;

[0017] The audio frequency domain signal is input into the generator of the original generative adversarial network.

[0018] In one of the embodiments, the discriminator includes a multi-scale time domain discriminator and a multi-scale frequency domain discriminator;

[0019] The step of inputting the compensated audio into the discriminator of the generative adversarial network to obtain a discrimination result includes:

[0020] Inputting the compensated audio into the multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator;

[0021] Convert the compensated audio from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio;

[0022] The frequency domain signal of the compensated audio is input into the multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

[0023] In one of the embodiments, obtaining the lost audio data;

[0024] Acquire a previous target audio signal or a context target audio signal within a preset range of a packet loss frame from the audio data to obtain a reference audio signal;

[0025] Convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal;

[0026] The frequency domain signal of the reference audio signal is input into the audio packet loss compensation model for prediction to obtain a compensation signal of the packet loss frame.

[0027] In one of the embodiments, the above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the continuous multiple frames is obtained from the audio data to obtain a reference audio signal of the current frame to be compensated, and the step of converting the reference audio signal from the time domain to the frequency domain to obtain the frequency domain signal of the reference audio is returned to generate a compensation signal for the current frame to be compensated, until the generated compensation signal reaches the maximum compensation frame, or the compensation of the continuous multiple frames is completed.

[0028] In a second aspect, the present application also provides a training device for an audio packet loss compensation model. The device comprises:

[0029] A generative adversarial network training module, used for iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the original generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0030] A determination module, used to determine the calculation related to generating a compensation signal for a packet loss frame in a generator of the trained generative adversarial network;

[0031] The model acquisition module is used to reconstruct the generator according to the calculation related to the compensation signal of the packet loss frame generated in the generator to obtain the audio packet loss compensation model; the audio packet loss compensation model generates the compensation signal of the audio packet loss frame.

[0032] In a third aspect, the present application further provides a computer device. The computer device includes a memory and a processor, the memory stores a computer program, and the processor implements the following steps when executing the computer program:

[0033] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0034] Determining calculations in a trained generator of the generative adversarial network related to generating the compensation signal for the packet loss frame;

[0035] According to the calculation in the generator related to generating the compensation signal of the packet loss frame, the generator is reconstructed to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal of the audio packet loss frame.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0037] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0038] Determining calculations in a trained generator of the generative adversarial network related to generating the compensation signal for the packet loss frame;

[0039] According to the calculation in the generator related to generating the compensation signal of the packet loss frame, the generator is reconstructed to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal of the audio packet loss frame.

[0040] In a fifth aspect, the present application further provides a computer program product. The computer program product includes a computer program, and when the computer program is executed by a processor, the following steps are implemented:

[0041] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0042] Determining calculations in a trained generator of the generative adversarial network related to generating the compensation signal for the packet loss frame;

[0043] According to the calculation in the generator related to generating the compensation signal of the packet loss frame, the generator is reconstructed to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal of the audio packet loss frame.

[0044] The training method, device, computer equipment, storage medium and computer program product of the above-mentioned audio packet loss compensation model iteratively train the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network, and clarify the calculations in the generator related to the compensation signal for generating packet loss frames, and reconstruct the generator of the generative adversarial network according to the calculations in the generator related to the compensation signal for generating packet loss frames to obtain the audio packet loss compensation model. Since the audio packet loss compensation model is reconstructed based on the calculations in the generator related to the compensation signal for generating packet loss frames, the audio packet loss compensation model only needs to perform calculations related to generating compensation signals when generating compensation signals for packet loss frames, which reduces the amount of irrelevant calculations, reduces model redundancy, and improves the speed of packet loss compensation, thereby improving the real-time performance of the packet loss compensation algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A diagram of an application environment of a training method for an audio packet loss compensation model in one embodiment;

[0046] Figure 2 A schematic diagram of a flow chart of a method for training an audio packet loss compensation model in one embodiment;

[0047] Figure 3 is a schematic diagram of reconstruction of an audio packet loss compensation model in one embodiment;

[0048] Figure 4 A schematic diagram of a training process of an audio packet loss compensation model in another embodiment;

[0049] Figure 5 A schematic diagram of an application flow of an audio packet loss compensation model in one embodiment;

[0050] Figure 6 A schematic diagram of a process flow of packet loss compensation for continuous audio packet loss in one embodiment;

[0051] Figure 7 A schematic diagram of the effect of an audio packet loss compensation model in another embodiment;

[0052] Figure 8 A schematic diagram of the effect of an audio packet loss compensation model in another embodiment;

[0053] Fig. 9 is a structural block diagram of a training device for an audio packet loss compensation model in one embodiment;

[0054] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] There are many schemes for packet loss compensation, which can be summarized into three directions: packet loss compensation based on insertion, packet loss compensation based on reconstruction, and packet loss compensation based on interpolation. Among them, the scheme for packet loss compensation based on interpolation is to create a data packet similar to the lost packet through pattern matching and interpolation technology. At present, there are two main problems in the development of this direction. The audio generated by linear prediction based on traditional signals is severely distorted. The packet loss compensation algorithm based on neural network has the problem of poor real-time performance, which makes it impossible to apply in the actual scenario of multiple microphones.

[0057] The training method of the audio packet loss compensation model provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. Figure 1 Take the voice call scenario as an example, in which the first terminal 102 is provided to the terminal user for sending or receiving voice streams, and the server 104 is used to support the voice call. When a voice call is made on the mobile terminal, the audio packet loss compensation model performs packet loss compensation on the packet loss frames during the voice call. The voice after packet loss compensation is transmitted to the second terminal 106 through the server 104. Specifically, the training process of the audio packet loss compensation model is as follows: after the generator of the original generative adversarial network receives the audio training set, it performs packet loss compensation on the packet loss audio in the audio training set to generate compensated audio. Among them, the generator of the original generative adversarial network is used to generate a compensation signal for the packet loss frame of the packet loss audio to obtain the compensated audio, and the discriminator of the original generative adversarial network is used to discriminate the authenticity of the compensated audio; determine the calculation related to the compensation signal for generating the packet loss frame in the generator of the trained generative adversarial network; according to the calculation related to the compensation signal for generating the packet loss frame in the generator, reconstruct the generator to obtain the audio packet loss compensation model; the audio packet loss compensation model is for the compensation signal of the packet loss frame of the preset packet loss audio. The terminal may be, but is not limited to, various personal computers, laptop computers, smart phones, tablet computers, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0058] In one embodiment, Figure 2 As shown, a training method for an audio packet loss compensation model is provided, and the method is applied to Figure 1 The server in the process includes the following steps:

[0059] Step 202, iteratively train the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of the lost frame of the lost audio to obtain compensated audio.

[0060] The audio training set is used to train the original generative adversarial network, and the audio training set includes audio data, wherein the audio data may include packet loss audio, complete audio of packet loss audio, and non-packet loss audio.

[0061] The original generative adversarial network has a generator and a discriminator. The generator and discriminator of the original generative adversarial network are trained through the audio training set to generate a trained generative adversarial network. The generator of the generative adversarial network is composed of an encoder and a decoder. The audio data in the audio training set is input into the encoder of the generator, and the audio data is feature extracted. The decoder of the generator generates the compensation audio of the lost frame of the lost audio according to the feature. The compensation audio of the lost frame of the lost audio and the complete audio corresponding to the lost audio are input into the discriminator of the original generative adversarial network. The discriminator of the generative adversarial network judges the authenticity of the compensation audio of the lost frame and the complete audio, and determines the loss function of the discriminator based on the result output by the discriminator of the generative adversarial network. The parameters of the generator are adjusted according to the loss function. After multiple trainings, the generator generates compensation audio that is difficult for the discriminator to distinguish the authenticity. At this time, a trained generative adversarial network is obtained, and a trained generator is obtained.

[0062] Step 204, determining the calculations in the generator of the trained generative adversarial network related to generating a compensation signal for the packet loss frame.

[0063] In this embodiment, a compensation signal similar to a packet loss frame is generated by a generative adversarial network. In the process of training the original generative adversarial network, the generator of the original generative adversarial network receives audio data in an audio training set, and when the received audio data is packet loss audio, determines the position of the packet loss frame of the packet loss audio. The required reference audio frame is selected according to the position of the packet loss frame of the packet loss audio.

[0064] In one embodiment, 4 frames of receptive fields are selected, of which the frame after the 4th frame is a packet loss frame, that is, the 5th frame is a packet loss frame, and these four frames are historical audio data. The audio data corresponding to the 4 frames of receptive fields are input into the generator of the original generative adversarial network, and the 4 frames of audio data will generate 4 frames of waveform signals through upsampling, of which the waveform data of the 5th frame needs to be generated by the first four frames of data. However, generating the waveform data of the 5th frame only requires a part of the relevant calculations in the first 4 frames of audio, that is, a part of the calculations in the process of generating compensation signals for the first 4 frames are irrelevant to the compensation signals, and the calculations in the generator related to generating compensation signals for packet loss frames can be determined during the training process of the generator.

[0065] Step 206, reconstructing the generator according to calculations in the generator related to generating a compensation signal for a packet loss frame to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal for an audio packet loss frame.

[0066] Among them, reconstruction refers to rebuilding a new audio packet loss compensation model based on the trained generator and the calculation related to the compensation signal of the packet loss frame in the determined generator. Therefore, the new audio packet loss compensation model only involves the calculation related to the compensation signal of the packet loss frame.

[0067] Specifically, the reconstruction method can be pruning. For unstructured pruning, the branch weight is changed to 0, which is equivalent to removing the branch. If it is observed that some branches with an iterative weight of 0 have a great effect on this round, the weight caused by unstructured pruning can be dynamically restored to 0. The reconstruction method can also be to delete the calculations in the generator that are not related to the compensation signal for generating packet loss frames.

[0068] In the process of training the generator of the generative adversarial network, a calculation related to generating a compensation signal for a packet loss frame in the generator of the generative adversarial network is determined. According to the calculation related to generating a compensation signal for a packet loss frame in the generator, the trained generator of the generative adversarial network is reconstructed to obtain an audio packet loss compensation model.

[0069] In this embodiment, the original generative adversarial network is iteratively trained based on the audio training set to obtain a trained generative adversarial network, and the calculations in the generator related to the compensation signal for generating packet loss frames are clarified. The generator of the generative adversarial network is reconstructed according to the calculations in the generator related to the compensation signal for generating packet loss frames to obtain an audio packet loss compensation model. Since the audio packet loss compensation model is reconstructed based on the calculations in the generator related to the compensation signal for generating packet loss frames, the audio packet loss compensation model only needs to perform calculations related to generating compensation signals when generating compensation signals for packet loss frames, which reduces the amount of irrelevant calculations, reduces model redundancy, and improves the speed of packet loss compensation, thereby improving the real-time performance of the packet loss compensation algorithm.

[0070] In one embodiment, Figure 3 As shown, according to the calculations in the generator related to the compensation signal for generating packet loss frames, the generator is reconstructed to obtain an audio packet loss compensation model, including: determining the calculations in the generator that are not related to the compensation signal for generating packet loss frames, deleting the irrelevant calculations in the generator, and obtaining the audio packet loss compensation model based on the retained calculations related to the compensation signal for generating packet loss frames.

[0071] In this embodiment, the reconstruction method is to delete the calculations in the trained generator that are irrelevant to the generation of the compensation signal for the packet loss frame.

[0072] Specifically, when the playback audio cannot be obtained, it is necessary to predict the packet loss frame based on historical data. In this embodiment, a compensation signal similar to the packet loss frame is generated by generating an adversarial network, thereby achieving packet loss compensation.

[0073] In the training of the original generative adversarial network, the generator of the original generative adversarial network receives the audio data in the audio training set. When performing packet loss compensation according to the audio data, it is first necessary to determine the position of the packet loss frame in the received audio data. And select the receptive field to be referenced according to the position of the packet loss frame of the packet loss audio. Generally, there are two situations, one is to select several frames of historical data as the receptive field, and the other is to select several frames of history and several frames in the future as the receptive field. In this embodiment, several frames of historical data are selected as the receptive field. Generally, the more frames of the reference receptive field, the higher the accuracy of the generated packet loss frame, but at the same time, due to the large number of reference frames, the amount of calculation is large, so that the real-time performance of the model is poor. Based on this, according to the real-time requirements of packet loss compensation, a suitable number of reference receptive fields can be selected. In this embodiment, taking the selection of 4 frames of receptive fields as an example, it can be understood that the above 4 frames of receptive fields are four consecutive frames of audio data before the packet loss frame, and the packet loss frame corresponds to the frame after the 4 frames of receptive fields, that is, the packet loss frame is the fifth frame, and the above five frames are continuous in time.

[0074] When compensating for packet loss based on the 4-frame receptive field before the packet loss frame, the compensation signal of the packet loss frame is generated by the generator of the generative adversarial network. The packet loss frame needs to be predicted based on the 4-frame receptive field, but there will be calculations that are irrelevant to the prediction of the packet loss frame. Therefore, in the process of training the generator of the generative adversarial network based on the audio training set, it is necessary to gradually clarify the calculations in the generator that are irrelevant to the generation of the compensation signal of the packet loss frame. Figure 3 As shown in the figure, after the training of the generator is completed, the calculations in the generator that are not related to the compensation signal for generating packet loss frames that are clear in the model training process are used to determine the above-mentioned irrelevant calculations that need to be deleted in the generator, and the calculations related to the compensation signal for generating packet loss frames are retained. The redundant calculations are deleted, which can reduce the amount of calculation by 1 / 3, complete the reconstruction of the generator, and obtain the audio packet loss compensation model. Among them, the obtained audio packet loss compensation model is a recursive model that can continuously compensate for audio with continuous packet loss.

[0075] In this embodiment, the generator of the trained generative adversarial network is reconstructed according to the calculations in the determined generator that are irrelevant to the compensation signal for generating packet loss frames, that is, the calculations in the generator that are irrelevant to the compensation signal for generating packet loss frames are deleted, and the calculations related to the compensation signal for generating packet loss frames are retained, thereby reducing redundant calculations of the model and improving the real-time performance of audio packet loss compensation.

[0076] In one embodiment, Figure 4 As shown in Figure 1, each iterative training of the original generative adversarial network based on the audio training set includes the following steps:

[0077] Step 402: Obtain an audio training set, where the audio training set includes audio data and audio tags of the audio data, where the audio tags are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of the packet loss audio.

[0078] The audio training set includes audio data and audio labels of the audio data. Whether the audio data has packet loss can be determined based on the audio labels of the audio data.

[0079] Step 404: input the audio data in the audio training set into the generator of the original generative adversarial network, and the generator generates a compensation signal for the lost frame of the lost audio based on the audio label.

[0080] Specifically, the audio data in the audio training set and the audio labels of the audio data are input into the generator of the generative adversarial network. When the audio label of the audio data is packet loss audio, the generator of the generative adversarial network generates a compensation signal of the packet loss frame of the packet loss audio according to the packet loss audio. When the audio label of the audio data is non-packet loss audio, the audio data is directly output to the discriminator, so that when the model generates packet loss frames, it only generates and calculates the loss of packet loss frames, which reduces the parameters of the model. The model parameter amount is only 240k, which is a very small parameter amount, and improves the parallel capability of the model.

[0081] Step 406: Obtain compensated audio according to the compensation signal and the packet-lost audio.

[0082] The compensated audio is audio data after compensating for the packet loss frame.

[0083] According to the compensation signal generated by the generator of the generative adversarial network, compensation data of the packet loss frame is obtained, and the compensation data of the packet loss frame is spliced ​​into the packet loss audio to obtain the compensated audio.

[0084] Step 408: Input the compensated audio into the discriminator of the generative adversarial network to obtain a discrimination result.

[0085] The discriminator of the adversarial network judges the compensated audio generated by the generator to obtain the discriminator output result of the compensated audio. The discriminator of the adversarial network judges the complete audio corresponding to the compensated audio to obtain the discriminator output result of the complete audio.

[0086] Step 410, obtain the discriminator loss according to the discriminant result, the difference between the compensated audio and the complete audio, and adjust the model parameters of the generative adversarial network according to the discriminator loss.

[0087] Among them, the discriminator loss can include adversarial loss, feature loss and multi-scale Mel spectrum loss.

[0088] Fighting Losses

[0089] Among them, D k,t (x) represents the output of the complete audio corresponding to the compensated audio after the discriminator, D k,t (g(x)) represents the output of the generated compensated audio after the discriminator, k represents the number of discriminators, and T represents the output length of the discriminator.

[0090] Feature loss

[0091] Among them, l represents the output of the feature map of each layer.

[0092] Multi-scale Mel spectrum loss:

[0093] in, Represents the Mel spectrum output of the complete audio frame t corresponding to the compensated audio, The Mel spectrum output of the tth frame of the generated compensated audio. The Mel spectrum of the audio is obtained from the audio.

[0094] The loss of the discriminator is calculated based on the discriminator output results of the compensated audio and the complete audio. The model parameters of the generative adversarial network are adjusted based on the calculated model loss. After one iteration of training is completed, the step 402 is returned again to train the generative adversarial network using the audio data in the audio training set until the discriminator of the generative adversarial network cannot distinguish between the compensated audio and the complete audio.

[0095] In this embodiment, the original generative adversarial network is iteratively trained using audio data in an audio training set. During the training process, the model parameters of the generative adversarial network are adjusted according to the calculated discriminator loss until the discriminator of the generative adversarial network cannot distinguish between the compensated audio and the complete audio, thereby completing the training of the original generative adversarial network.

[0096] In one embodiment, audio data in an audio training set is input into a generator of a generative adversarial network, including: converting audio signals in the audio training set from time domain to frequency domain to obtain audio frequency domain signals; and inputting the audio frequency domain signals into the generator of the generative adversarial network.

[0097] Among them, the time domain is an expression form that uses time as the independent variable to describe the change of physical quantities, and the frequency domain is an expression form that uses frequency as the independent variable to describe the change of physical quantities.

[0098] In this embodiment, considering that the spectrum input can reduce the computational complexity of downsampling feature extraction caused by the speech waveform input, the audio signal in the audio training set is converted. Specifically, the audio is converted from a time domain signal to a frequency domain signal through a fast Fourier transform, thereby obtaining an audio frequency domain signal in the audio training set. Therefore, when training the generative adversarial network, the audio frequency domain signal is input into the generator of the generative adversarial network.

[0099] In this embodiment, by converting the audio signals in the audio training set from the time domain to the frequency domain, the amount of calculation caused by downsampling of the audio packet loss compensation model is reduced, and the real-time compensation performance of the audio packet loss compensation model is improved.

[0100] In one embodiment, the discriminator includes a multi-scale time domain discriminator and a multi-scale spectrum discriminator;

[0101] Inputting the compensated audio into a discriminator of a generative adversarial network to obtain a discrimination result includes: inputting the compensated audio into a multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator; converting the compensated audio from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio; inputting the frequency domain signal of the compensated audio into a multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

[0102] In order to improve the granularity of the discriminator, the packet loss audio generated by the generator is discriminated from the time domain and the frequency domain. The multi-scale time domain discriminator is used to discriminate the time domain of the packet loss audio generated by the generator, and the multi-scale frequency domain discriminator is used to discriminate the frequency domain of the packet loss audio generated by the generator. Specifically, the time domain waveform of the packet loss audio generated by the generator is downsampled by 1, 2 and 4 times, and the downsampled time domain waveform is respectively input into three time domain discriminators with different parameters, and the parameters of the generator are adjusted according to the discrimination results of the time domain discriminators with different granularities. Similarly, the compensated audio is transformed from the time domain to the frequency domain by fast Fourier transform to obtain the frequency domain signal of the compensated audio, and the frequency domain signal is downsampled by 1, 2 and 4 times, and the downsampled frequency domain waveform is respectively input into three frequency domain discriminators with different parameters, and the parameters of the generator are adjusted according to the discrimination results of different frequency domain discriminators. There are different choices for the scales of downsampling the time domain waveform and the frequency domain waveform, which can be selected according to actual conditions.

[0103] In this embodiment, the packet loss audio is discriminated by a multi-scale time domain discriminator and a multi-scale frequency domain discriminator, and the parameters of the generator are adjusted according to the discrimination results of the multi-scale time domain discriminator and the multi-scale frequency domain discriminator, that is, the discrimination ability of the discriminator in the generative adversarial network is improved from the time domain and frequency domain.

[0104] Among them, the audio packet loss compensation model can be applied to compensate for packet loss audio in various audio transmission scenarios, such as Figure 5 As shown, the application method of the audio packet loss compensation model is:

[0105] Step 502: Acquire the lost audio data.

[0106] The audio data with packet loss may be the audio data with packet loss in the live stream in the live broadcast scenario of the network, or the audio data with packet loss sent in a voice call.

[0107] The audio packet loss model provided in this application only runs the model when packet loss occurs, thereby reducing the amount of computation caused by continuous model reasoning.

[0108] Step 504: Acquire a previous target audio signal or a context target audio signal within a preset range of the packet loss frame from the audio data to obtain a reference audio signal.

[0109] Among them, the reference audio signal refers to the audio signal used as a reference for predicting the packet loss frame audio. There are generally two situations for selecting the reference audio signal. One is to select the audio signal before the packet loss frame (historical frame) as the reference audio signal, that is, to select the previous few frames of data as the reference audio signal. The other is to select a few frames before the packet loss frame (historical frame) and after the packet loss frame (future frame) as the reference audio signal, that is, to select the previous few frames and the following few frames as the reference audio signal. In this case, packet loss compensation refers to the future frame. The preset range is a preset number of frames, which can be selected according to the actual situation, such as selecting 3 frames or 5 frames, etc.

[0110] When performing packet loss compensation based on audio data, it is first necessary to determine the packet loss frame in the received audio data, and select an audio signal to be referenced based on the packet loss frame of the packet loss audio, that is, select a reference audio signal to be referenced.

[0111] Step 506: Convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal.

[0112] The reference audio signal is converted from the time domain to the frequency domain through the fast Fourier transform, thereby obtaining the frequency domain signal of the reference audio signal. Because the frequency domain has fast algorithms such as fast Fourier transform, the time complexity and space complexity can be reduced, making the frequency domain signal conducive to real-time processing and reducing the required storage amount.

[0113] Step 508: Input the frequency domain signal of the reference audio signal into the audio packet loss compensation model to obtain a compensation signal of the packet loss frame.

[0114] After the frequency domain signal of the reference signal is input into the audio packet loss compensation model, the frequency domain signal of the reference audio signal is upsampled to generate a waveform signal, which is compensated by the audio packet loss compensation model to obtain a compensation signal of the lost frame of the lost audio, and the audio data of the lost frame is obtained according to the compensation signal of the lost frame.

[0115] In this embodiment, after obtaining the packet-lost audio data, the reference audio signal of the packet-lost frame in the audio data is determined, the reference audio signal is converted to the frequency domain, and the audio packet-lost compensation model is input to obtain the compensation data of the packet-lost frame, thereby completing the compensation of the packet-lost frame. Since the audio packet-lost compensation model deletes and reduces the amount of irrelevant calculations, and only retains the calculations related to generating the compensation signal of the packet-lost frame, in the scenario of multiple microphones, the packet-lost audio of multiple channels is calculated in parallel for packet-lost compensation, and its real-time rate is still relatively high. And converting the input of the audio packet-lost compensation model into a frequency domain signal is more conducive to real-time processing of packet-lost audio.

[0116] In one embodiment, the application method of the audio packet loss compensation model is as follows: Figure 6 As shown, including:

[0117] Step 602: Acquire the lost audio data.

[0118] Step 604: Acquire the previous target audio signal or context target audio signal within a preset range of the current frame to be compensated in the continuous packet loss frames from the audio data to obtain a reference audio signal of the current frame to be compensated.

[0119] Step 606: Convert the reference audio signal from the time domain to the frequency domain to obtain a reference audio time domain signal.

[0120] Step 608: Input the reference audio time domain signal into the audio packet loss compensation model to obtain a compensation signal of the current frame to be compensated.

[0121] Step 610 , determining whether the generated compensation signal reaches the maximum compensation frame, or whether the compensation for the continuous packet loss frames is completed. If not, executing step 612 .

[0122] Step 612: The next frame of the current compensation frame is used as a new current frame to be compensated, and the process returns to step 602 until the generated compensation signal reaches the maximum compensation frame, or the compensation for the continuous packet loss frames is completed.

[0123] like Figure 6As shown, when the packet loss frames of the packet loss audio are multiple consecutive frames, the audio packet loss compensation model is used to generate a compensation signal for the first frame of the multiple consecutive frames, and then packet loss compensation is performed on the second frame in the packet loss frame, and the second frame in the packet loss frame is determined as the current frame to be compensated, and the above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the multiple consecutive frames is obtained from the audio data to obtain a reference audio signal of the current frame to be compensated, and the reference audio signal is converted from the time domain to the frequency domain to obtain a reference audio time domain signal to generate a compensation signal for the current frame to be compensated, that is, a compensation signal for the second frame in the packet loss frame is generated, and recursive iterations are performed in sequence until the generated packet loss data reaches the maximum compensation frame, or the compensation for continuous packet loss is completed.

[0124] In one embodiment, the present application provides a method for training an audio packet loss compensation model, including two stages: training the audio packet loss compensation model and applying the audio packet loss compensation model.

[0125] The training process of the audio packet loss compensation model includes:

[0126] Obtain an audio training set, which includes audio data and audio labels of the audio data; convert the audio data in the audio training set from the time domain to the frequency domain to obtain an audio frequency domain signal; input the audio frequency domain signal in the audio training set into the generator of the generative adversarial network, and the generator generates a compensation signal of the lost frame of the lost audio based on the audio label; obtain the compensated audio according to the compensation signal and the lost audio; input the compensated audio into the discriminator of the generative adversarial network to obtain a discrimination result; obtain the discriminator loss according to the discrimination result, the difference between the compensated audio and the complete audio, and adjust the model parameters of the generative adversarial network according to the discriminator loss. After multiple trainings, the discriminator has difficulty distinguishing the authenticity of the generated compensated audio, and the generative adversarial network training is completed at this time. After the generative adversarial network training is completed, determine the calculations related to the compensation signal for generating the lost frame in the generator of the trained generative adversarial network, and reconstruct the generator according to the calculations related to the compensation signal for generating the lost frame to obtain an audio packet loss compensation model.

[0127] The application process of the audio packet loss compensation model includes:

[0128] After the audio packet loss compensation model is obtained, the above target audio signal or the context target audio signal in the preset range of the packet loss frame of the packet loss audio is used as the reference audio signal of the packet loss frame. The reference audio signal is converted from the time domain to the frequency domain to obtain a reference audio time domain signal, and the reference audio time domain signal is input into the audio packet loss compensation model to obtain a compensation signal of the packet loss frame.

[0129] In order to determine the effect of the audio packet loss compensation model, the audio packet loss compensation model provided by the present application and the packet loss compensation effect of the jitter buffer are scored. Figure 7 As shown in the figure, taking the compensation of two frames of audio data as an example, according to the scores in the figure, it can be known that the audio packet loss compensation model provided by the present application has a good packet loss compensation effect. And the audio packet loss compensation model is recursive and can compensate for multiple consecutive frames. Figure 8 As shown in the figure, it can specifically support large packet loss within 80ms and random packet loss with a packet loss rate of 20%. The audio packet loss compensation model was tested, and the test results showed that the real-time rate of a single core on the window side was 0.05, and the CPU occupancy of each channel during the stress test was about 1%. The real-time rate of the stress test on the iPhone 7 was within 0.2, which improved the real-time performance of the packet loss compensation of the audio packet loss compensation model.

[0130] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0131] Based on the same inventive concept, the embodiment of the present application also provides a training device for an audio packet loss compensation model for implementing the training method for the audio packet loss compensation model involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the embodiments of the training device for one or more audio packet loss compensation models provided below can refer to the limitations of the training method for the audio packet loss compensation model above, and will not be repeated here.

[0132] In one embodiment, Fig. 9 As shown, a training device for an audio packet loss compensation model is provided, comprising: a generative adversarial network training module 902, a determination module 904 and a model acquisition module 906, wherein:

[0133] The generative adversarial network training module 902 is used to iteratively train the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the original generative adversarial network is used to generate a compensation signal for the lost frame of the lost audio to obtain compensated audio.

[0134] The determination module 904 is used to determine the calculations related to generating the compensation signal of the packet loss frame in the generator of the trained generative adversarial network.

[0135] The model acquisition module 906 is used to reconstruct the generator according to the calculation related to generating the compensation signal of the packet loss frame in the generator to obtain the audio packet loss compensation model; the audio packet loss compensation model generates the compensation signal of the audio packet loss frame.

[0136] In one embodiment, a model acquisition module is used to determine calculations in a generator that are not related to a compensation signal for generating packet loss frames, delete irrelevant calculations in the generator, retain calculations related to a compensation signal for generating packet loss frames, and obtain an audio packet loss compensation model.

[0137] In one embodiment, a generative adversarial network training module is used to obtain an audio training set, where the audio training set includes audio data and audio tags of the audio data, where the audio tags are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of packet loss audio; the audio data in the audio training set is input into a generator of a generative adversarial network, where the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio tags; compensated audio is obtained based on the compensation signal and the packet loss audio; the compensated audio is input into a discriminator of the generative adversarial network to obtain a discrimination result; the discriminator loss is obtained based on the discrimination result, the difference between the compensated audio and the complete audio, and the model parameters of the generative adversarial network are adjusted based on the discriminator loss.

[0138] In one embodiment, the training device of the audio packet loss compensation model includes a conversion module, which is used to convert the audio signal in the audio training set from the time domain to the frequency domain to obtain an audio frequency domain signal; and input the audio frequency domain signal into the generator of the generative adversarial network.

[0139] In one embodiment, the training device of the audio packet loss compensation model includes a discrimination module, which is used to input the compensated audio into a multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator; convert the compensated audio from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio; input the frequency domain signal of the compensated audio into the multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

[0140] Each module in the training device of the above-mentioned audio packet loss compensation model can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Fig.10As shown. The computer device includes a processor, a memory, a communication interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. When the computer program is executed by the processor, a training method for an audio packet loss compensation model is implemented.

[0142] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0143] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:

[0144] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0145] Determining computations in a generator of a trained generative adversarial network related to generating a compensation signal for a lost frame;

[0146] The generator is reconstructed according to calculations in the generator related to generating a compensation signal for a packet loss frame to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal for an audio packet loss frame.

[0147] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0148] Calculations in the generator that are irrelevant to the compensation signal for generating packet loss frames are determined, and irrelevant calculations in the generator are deleted. An audio packet loss compensation model is obtained based on the retained calculations that are relevant to the compensation signal for generating packet loss frames.

[0149] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0150] An audio training set is obtained, where the audio training set includes audio data and audio labels of the audio data, where the audio labels are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of packet loss audio; the audio data in the audio training set is input into a generator of a generative adversarial network, where the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio labels; compensated audio is obtained based on the compensation signal and the packet loss audio; the compensated audio is input into a discriminator of the generative adversarial network to obtain a discrimination result; a discriminator loss is obtained based on the difference between the discrimination result, the compensated audio and the complete audio, and model parameters of the generative adversarial network are adjusted based on the discriminator loss.

[0151] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0152] The audio data in the audio training set is converted from the time domain to the frequency domain to obtain an audio frequency domain signal; the audio frequency domain signal is input into the generator of the original generative adversarial network.

[0153] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0154] Input the compensated audio into the multi-scale time domain discriminator to obtain the discrimination result of the multi-scale time domain discriminator; convert the compensated audio from the time domain to the frequency domain to obtain the frequency domain signal of the compensated audio;

[0155] The frequency domain signal of the compensated audio is input into the multi-scale frequency domain discriminator to obtain the discrimination result of the multi-scale frequency domain discriminator.

[0156] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0157] Acquire the lost audio data; acquire the above target audio signal or context target audio signal within a preset range of the lost frame from the audio data to obtain a reference audio signal; convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal; input the frequency domain signal of the reference audio signal into an audio packet loss compensation model for prediction to obtain a compensation signal of the lost frame.

[0158] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:

[0159] The above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the continuous multiple frames is obtained from the audio data to obtain the reference audio signal of the current frame to be compensated, and the step of converting the reference audio signal from the time domain to the frequency domain to obtain the frequency domain signal of the reference audio is returned to generate the compensation signal of the current frame to be compensated, until the generated compensation signal reaches the maximum compensation frame, or the compensation of the continuous multiple frames is completed.

[0160] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:

[0161] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0162] Determining computations in a generator of a trained generative adversarial network related to generating a compensation signal for a lost frame;

[0163] The generator is reconstructed according to calculations in the generator related to generating a compensation signal for a packet loss frame to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal for an audio packet loss frame.

[0164] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0165] Calculations in the generator that are irrelevant to the compensation signal for generating packet loss frames are determined, and irrelevant calculations in the generator are deleted. An audio packet loss compensation model is obtained based on the retained calculations that are relevant to the compensation signal for generating packet loss frames.

[0166] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0167] An audio training set is obtained, where the audio training set includes audio data and audio labels of the audio data, where the audio labels are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of packet loss audio; the audio data in the audio training set is input into a generator of a generative adversarial network, where the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio labels; compensated audio is obtained based on the compensation signal and the packet loss audio; the compensated audio is input into a discriminator of the generative adversarial network to obtain a discrimination result; a discriminator loss is obtained based on the difference between the discrimination result, the compensated audio and the complete audio, and model parameters of the generative adversarial network are adjusted based on the discriminator loss.

[0168] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0169] The audio data in the audio training set is converted from the time domain to the frequency domain to obtain an audio frequency domain signal; the audio frequency domain signal is input into the generator of the original generative adversarial network.

[0170] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0171] The compensated audio is input into a multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator; the compensated audio is converted from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio; the frequency domain signal of the compensated audio is input into a multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

[0172] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0173] Acquire the lost audio data; acquire the above target audio signal or context target audio signal within a preset range of the lost frame from the audio data to obtain a reference audio signal; convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal; input the frequency domain signal of the reference audio signal into an audio packet loss compensation model for prediction to obtain a compensation signal of the lost frame.

[0174] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0175] The above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the continuous multiple frames is obtained from the audio data to obtain the reference audio signal of the current frame to be compensated, and the step of converting the reference audio signal from the time domain to the frequency domain to obtain the frequency domain signal of the reference audio is returned to generate the compensation signal of the current frame to be compensated, until the generated compensation signal reaches the maximum compensation frame, or the compensation of the continuous multiple frames is completed.

[0176] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:

[0177] Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio;

[0178] Determining computations in a generator of a trained generative adversarial network related to generating a compensation signal for a lost frame;

[0179] The generator is reconstructed according to calculations in the generator related to generating a compensation signal for a packet loss frame to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal for an audio packet loss frame.

[0180] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0181] Calculations in the generator that are irrelevant to the compensation signal for generating packet loss frames are determined, and irrelevant calculations in the generator are deleted. An audio packet loss compensation model is obtained based on the retained calculations that are relevant to the compensation signal for generating packet loss frames.

[0182] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0183] An audio training set is obtained, where the audio training set includes audio data and audio labels of the audio data, where the audio labels are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of packet loss audio; the audio data in the audio training set is input into a generator of a generative adversarial network, where the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio labels; compensated audio is obtained based on the compensation signal and the packet loss audio; the compensated audio is input into a discriminator of the generative adversarial network to obtain a discrimination result; a discriminator loss is obtained based on the difference between the discrimination result, the compensated audio and the complete audio, and model parameters of the generative adversarial network are adjusted based on the discriminator loss.

[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0185] The audio data in the audio training set is converted from the time domain to the frequency domain to obtain an audio frequency domain signal; the audio frequency domain signal is input into the generator of the original generative adversarial network.

[0186] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0187] The compensated audio is input into a multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator; the compensated audio is converted from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio; the frequency domain signal of the compensated audio is input into a multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

[0188] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0189] Acquire the lost audio data; acquire the above target audio signal or context target audio signal within a preset range of the lost frame from the audio data to obtain a reference audio signal; convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal; input the frequency domain signal of the reference audio signal into an audio packet loss compensation model for prediction to obtain a compensation signal of the lost frame.

[0190] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:

[0191] The above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the continuous multiple frames is obtained from the audio data to obtain the reference audio signal of the current frame to be compensated, and the step of converting the reference audio signal from the time domain to the frequency domain to obtain the frequency domain signal of the reference audio is returned to generate the compensation signal of the current frame to be compensated, until the generated compensation signal reaches the maximum compensation frame, or the compensation of the continuous multiple frames is completed.

[0192] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.

[0193] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0194] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A training method for an audio packet loss compensation model, characterized in that: The method comprises: Iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio; Determining calculations in a trained generator of the generative adversarial network related to generating the compensation signal for the packet loss frame; According to the calculation in the generator related to generating the compensation signal of the packet loss frame, the generator is reconstructed to obtain an audio packet loss compensation model; the audio packet loss compensation model generates a compensation signal of the audio packet loss frame.

2. The method according to claim 1, characterized in that The step of reconstructing the generator according to the calculation related to generating the compensation signal of the packet loss frame to obtain the audio packet loss compensation model comprises: Determine calculations in the generator that are irrelevant to the compensation signal for generating the packet loss frame, delete the irrelevant calculations in the generator, and obtain an audio packet loss compensation model based on the retained calculations related to the compensation signal for generating the packet loss frame.

3. The method according to claim 1, characterized in that Each iteration of training the original generative adversarial network based on the audio training set includes: Acquire an audio training set, wherein the audio training set includes audio data and audio tags of the audio data, wherein the audio tags are used to identify packet loss audio and non-packet loss audio, and the audio training set also includes complete audio of the packet loss audio; Inputting the audio data in the audio training set into a generator of a generative adversarial network, wherein the generator generates a compensation signal of a packet loss frame of the packet loss audio based on the audio label; Obtaining compensated audio according to the compensation signal and the packet-lost audio; Inputting the compensated audio into the discriminator of the generative adversarial network to obtain a discrimination result; A discriminator loss is obtained according to the discrimination result, the difference between the compensated audio and the complete audio, and a model parameter of the generative adversarial network is adjusted according to the discriminator loss.

4. The method according to claim 3, characterized in that Inputting the audio data in the audio training set into the generator of the original generative adversarial network comprises: Converting the audio data in the audio training set from the time domain to the frequency domain to obtain an audio frequency domain signal; The audio frequency domain signal is input into the generator of the original generative adversarial network.

5. The method according to claim 4, characterized in that The discriminator includes a multi-scale time domain discriminator and a multi-scale frequency domain discriminator; The step of inputting the compensated audio into a discriminator of the generative adversarial network to obtain a discrimination result includes: Inputting the compensated audio into the multi-scale time domain discriminator to obtain a discrimination result of the multi-scale time domain discriminator; Convert the compensated audio from the time domain to the frequency domain to obtain a frequency domain signal of the compensated audio; The frequency domain signal of the compensated audio is input into the multi-scale frequency domain discriminator to obtain a discrimination result of the multi-scale frequency domain discriminator.

6. The method according to claim 1, characterized in that The method further comprises: Get the lost audio data; Acquire a previous target audio signal or a context target audio signal within a preset range of a packet loss frame from the audio data to obtain a reference audio signal; Convert the reference audio signal from the time domain to the frequency domain to obtain a frequency domain signal of the reference audio signal; The frequency domain signal of the reference audio signal is input into the audio packet loss compensation model for prediction to obtain a compensation signal of the packet loss frame.

7. The method according to claim 6, characterized in that If the packet loss frame includes multiple consecutive frames, after generating a compensation signal of a first frame of the multiple consecutive frames, the method further includes: The above target audio signal or context target audio signal of a preset range of the current frame to be compensated in the continuous multiple frames is obtained from the audio data to obtain a reference audio signal of the current frame to be compensated, and the step of converting the reference audio signal from the time domain to the frequency domain to obtain the frequency domain signal of the reference audio signal is returned to generate a compensation signal for the current frame to be compensated, until the generated compensation signal reaches the maximum compensation frame, or the compensation of the continuous multiple frames is completed.

8. A training device for an audio packet loss compensation model, characterized in that: The device comprises: A generative adversarial network training module, used for iteratively training the original generative adversarial network based on the audio training set to obtain a trained generative adversarial network; wherein the generator of the original generative adversarial network is used to generate a compensation signal of a packet loss frame of the packet loss audio to obtain compensated audio; A determination module, used to determine the calculation related to generating a compensation signal for a packet loss frame in a generator of the trained generative adversarial network; The model acquisition module is used to reconstruct the generator according to the calculation related to the compensation signal of the packet loss frame generated in the generator to obtain the audio packet loss compensation model; the audio packet loss compensation model generates the compensation signal of the audio packet loss frame.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images