Neural network-based voice packet loss concealment method and device

Abstract

A neural network-based voice packet loss concealment method and device. The method comprises: acquiring a pre-trained neural network used for voice packet loss concealment; receiving an audio to be processed and frame loss position information corresponding to said audio; and inputting an input feature generated on the basis of said audio and the frame loss position information corresponding to said audio into the neural network to obtain a packet-loss-concealed audio corresponding to said audio.

Description

Speech packet loss compensation method and device based on neural network

[0001] The present application claims priority to the Chinese patent application No. 202410411048.5, filed on April 7, 2024, entitled "Speech packet loss compensation method and device based on neural network", the whole content of which is incorporated herein by reference. TECHNICAL FIELD

[0002] Embodiments of the present disclosure relate to the technical field of computer, and particularly, to a speech packet loss compensation method and device based on neural network. BACKGROUND

[0003] With the development of Internet technology and communication technology, audio stream transmission has become a very important part of network communication. In practice, due to various factors such as network congestion, bandwidth limitation, hardware failure, etc., audio data packets may be lost during transmission, which will have a serious impact on the quality of voice communication and cause the user experience to decline. Therefore, it is urgent to solve the problem of repairing and recovering audio packet loss. SUMMARY

[0004] Embodiments of the present disclosure describe a speech packet loss compensation method and device based on neural network. The neural network trained by the method can more accurately perform speech packet loss compensation.

[0005] According to a first aspect, a method for training a neural network for speech packet loss compensation is provided, wherein the neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, the intermediate layer is connected between the encoder layer and the decoder layer, and the method includes: obtaining a training sample set, wherein each training sample includes a sample packet loss audio, sample frame loss position information corresponding to the sample packet loss audio, and a sample non-packet loss audio; generating an input feature based on the sample packet loss audio and the sample frame loss position information corresponding to the sample packet loss audio; inputting the input feature into the neural network to be trained; inputting a feature output by the intermediate layer into a pre-trained fundamental frequency prediction network, and outputting a predicted fundamental frequency from the fundamental frequency prediction network; and adjusting network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and a true fundamental frequency calculated based on the sample non-packet loss audio.

[0006] In one embodiment, the neural network to be trained is a neural network of U-Net structure, and the intermediate layer is a bottleneck layer in the U-Net structure. In this way, the speech packet loss compensation can be realized by the neural network of U-Net structure.

[0007] In an embodiment, the base frequency prediction network comprises a bidirectional long short-term memory network. The bidirectional long short-term memory network can fully consider the front and back information in the audio data when processing the audio data, thereby improving the processing capability of the audio data.

[0008] In an embodiment, the features output by the intermediate layer comprise features of frames corresponding to frame loss positions corresponding to the sample loss frame position information, and the predicted base frequency output by the base frequency prediction network comprises a base frequency of a frame corresponding to a frame loss position.

[0009] In an embodiment, the input features are generated based on the sample packet loss audio and the corresponding sample loss frame position information, comprising: performing sub-band decomposition on the sample packet loss audio to obtain a plurality of sub-bands; and generating the input features based on a conversion result of the plurality of sub-bands converted to a time-frequency domain and the sample loss frame position information. In this way, the sample packet loss audio can be decomposed into a plurality of sub-bands for processing, thereby greatly reducing the computational complexity.

[0010] In an embodiment, the encoder layer comprises a plurality of encoders, each of which comprises a gated convolution layer and a time-frequency space convolution layer, and the time-frequency space convolution layer is configured to extract features through time-dimension and frequency-dimension space convolution. In this way, the receptive field of the convolution layer can be effectively improved.

[0011] In an embodiment, the decoder layer comprises a plurality of decoders, each of which comprises a first branch and a second branch in parallel, the first branch is configured to predict a real part of the audio, and the second branch is configured to predict an imaginary part of the audio; the to-be-trained neural network is configured to output a sample predicted audio based on the real part and the imaginary part of the predicted audio output by the encoder layer; and the method further comprises: inputting the sample predicted audio into at least one discriminator pre-trained, outputting a discrimination result of the sample predicted audio by each discriminator; calculating a loss based on the at least one discrimination result, the sample predicted audio and the sample non-packet loss audio, and adjusting network parameters of the to-be-trained neural network based on the loss. In this way, the to-be-trained neural network can be trained using a generative adversarial network (GAN).

[0012] In an embodiment, the at least one discriminator comprises a first discriminator configured to discriminate a probability that the sample predicted audio is real audio, and a second discriminator configured to discriminate an audio quality of the sample predicted audio. In this way, the accuracy of the generator can be improved by using multiple discriminators.

[0013] In an embodiment, the method further comprises: inputting the sample predicted audio output by the to-be-trained neural network and the corresponding sample non-packet-loss audio into a pre-trained speech recognition model respectively; obtaining the encoding layer features of the sample predicted audio and the corresponding sample non-packet-loss audio in the speech recognition model; and adjusting the network parameters of the to-be-trained neural network based on the difference loss of the two obtained encoding layer features. In this way, the network parameters of the to-be-trained neural network can be adjusted through the pre-trained speech recognition model, and the accuracy of the to-be-trained neural network can be improved.

[0014] According to a second aspect, a neural network-based speech packet loss compensation method is provided, comprising: obtaining a neural network for speech packet loss compensation trained according to any of the methods of the first aspect; receiving to-be-processed audio and frame loss position information corresponding to the to-be-processed audio; inputting input features generated based on the to-be-processed audio and the frame loss position information corresponding to the to-be-processed audio into the neural network to obtain audio after packet loss compensation corresponding to the to-be-processed audio.

[0015] According to a third aspect, an apparatus for training a neural network for speech packet loss compensation is provided, wherein the to-be-trained neural network comprises an encoder layer, an intermediate layer, and a decoder layer, the intermediate layer is connected between the encoder layer and the decoder layer, and the apparatus comprises: an obtaining unit configured to obtain a training sample set, wherein each training sample comprises sample packet-loss audio, sample frame loss position information corresponding to the sample packet-loss audio, and sample non-packet-loss audio; a generating unit configured to generate input features based on the sample packet-loss audio and the sample frame loss position information corresponding to the sample packet-loss audio; a first input unit configured to input the input features into the to-be-trained neural network; a second input unit configured to input features output by the intermediate layer into a pre-trained fundamental frequency prediction network to output predicted fundamental frequencies by the fundamental frequency prediction network; and an adjusting unit configured to adjust network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequencies and true fundamental frequencies calculated based on the sample non-packet-loss audio.

[0016] According to a fourth aspect, a neural network-based speech packet loss compensation apparatus is provided, comprising: a model obtaining unit configured to obtain a neural network for speech packet loss compensation trained according to any of the methods of the first aspect; a receiving unit configured to receive to-be-processed audio and frame loss position information corresponding to the to-be-processed audio; and a feature input unit configured to input input features generated based on the to-be-processed audio and the frame loss position information corresponding to the to-be-processed audio into the neural network to obtain audio after packet loss compensation corresponding to the to-be-processed audio.

[0017] According to a fifth aspect, a computer program product is provided, comprising a computer program which, when executed by a processor, implements the method according to any one of the first aspect.

[0018] According to a sixth aspect, a computer-readable storage medium is provided, having stored thereon a computer program which, when executed in a computer, causes the computer to perform the method according to any one of the first aspect.

[0019] According to a seventh aspect, an electronic device is provided, comprising a memory and a processor, the memory having stored thereon executable code which, when executed by the processor, implements the method according to any one of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0020] Fig. 1 shows a schematic diagram of an application scenario in which embodiments of the present disclosure can be applied;

[0021] Fig. 2 shows a flow diagram of a method of training a neural network for speech packet loss compensation according to an embodiment;

[0022] Fig. 3 shows a schematic diagram of an example of an encoder structure;

[0023] Fig. 4 shows a schematic diagram of an example of a base frequency prediction network structure;

[0024] Fig. 5 shows a schematic diagram of an example of a neural network structure to be trained;

[0025] Fig. 6 shows a schematic diagram of an example of a frequency domain multi-resolution discriminator;

[0026] Fig. 7 shows a schematic diagram of an example of a time domain multi-period discriminator;

[0027] Fig. 8 shows a schematic diagram of an example of MetricGAN discriminator training;

[0028] Fig. 9 shows a neural network based speech packet loss compensation method according to an embodiment;

[0029] Fig. 10 shows a schematic block diagram of an apparatus for training a neural network for speech packet loss compensation according to an embodiment;

[0030] Fig. 11 shows a schematic block diagram of a neural network based speech packet loss compensation apparatus according to an embodiment;

[0031] Fig. 12 shows a schematic diagram of the structure of an electronic device suitable for implementing embodiments of the present application. DETAILED DESCRIPTION

[0032] It can be understood that, before using the technical solutions disclosed in the embodiments of the present disclosure, the type of personal information involved in the present disclosure, the use range, the use scenario, etc. should be informed to the user and the authorization of the user should be obtained through appropriate means according to relevant laws and regulations.

[0033] For example, in response to receiving an active request of a user, a prompt information is sent to the user to explicitly prompt the user that the operation requested to be performed will require obtaining and using personal information of the user. Thus, the user can voluntarily choose whether to provide personal information to the software or hardware such as an electronic device, an application program, a server or a storage medium, etc. performing the operation of the technical solutions of the present disclosure according to the prompt information.

[0034] As an optional but non-limiting implementation manner, in response to receiving an active request of a user, the manner of sending a prompt information to the user may, for example, be a pop-up window manner, in which the prompt information can be presented in a text manner. In addition, the pop-up window can also carry a selection control for the user to select "agree" or "disagree" to provide personal information to the electronic device.

[0035] It can be understood that the above notification and obtaining of user authorization process is only illustrative and does not limit the implementation manners of the present disclosure, and other manners meeting relevant laws and regulations can also be applied to the implementation manners of the present disclosure.

[0036] The technical solutions provided by the present disclosure will be further described in detail below in combination with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the related invention, but not to limit the invention. In addition, it should be noted that, for the convenience of description, only the parts related to the invention are shown in the drawings. It should be noted that, in the case of no conflict, the embodiments of the present disclosure and the features in the embodiments can be combined with each other.

[0037] As described above, the audio data packets may be lost during transmission, which will seriously affect the quality of voice communication and cause the user experience to decrease. The main goal of the voice packet loss concealment (PLC) technology is to recover or conceal the lost data packets as much as possible through various ways, so as to maintain the continuity and clarity of voice communication. The traditional packet loss concealment technology can be based on redundant coding or signal processing interpolation to supplement the lost packets. For example, in many codecs, the forward error correction technology is used. When detecting poor network conditions, the sender can transmit redundant information about past frames to recover short packet loss. However, this way will introduce additional network overhead and additional delay, and at the same time, it cannot handle long packet loss.

[0038] To this end, an embodiment of the present disclosure provides a neural network-based voice packet loss compensation method. First, a neural network for voice packet loss compensation can be trained by using a method provided by an embodiment of the present disclosure, so that the trained neural network can more accurately perform voice packet loss compensation without introducing additional network overhead and additional delay.

[0039] According to the neural network-based voice packet loss compensation method and device provided by an embodiment of the present disclosure, first, a neural network needs to be trained. The neural network to be trained can include an encoder layer, an intermediate layer, and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. The training samples used include sample packet loss audio, corresponding sample frame loss position information, and sample non-packet loss audio. During the training process, input features can be generated based on the sample packet loss audio and the corresponding sample frame loss position information, and the input features are input into the neural network to be trained. Then, the features output by the intermediate layer are input into a pre-trained fundamental frequency prediction network, and the fundamental frequency prediction network outputs a predicted fundamental frequency. Then, the network parameters of the encoder layer and the intermediate layer are adjusted based on the predicted fundamental frequency and the real fundamental frequency calculated based on the sample non-packet loss audio. In this way, the output of the encoder layer and the intermediate layer can be more accurate, and thus the trained neural network can more accurately perform voice packet loss compensation.

[0040] FIG. 1 shows a schematic diagram of an application scenario to which an embodiment of the present disclosure can be applied. As shown in FIG. 1, in the application scenario shown in FIG. 1, an electronic device 10 can first train a neural network 101 for voice packet loss compensation. Then, an electronic device 20 can obtain the neural network 101 from the electronic device 10, and use the trained neural network 101 to perform voice packet loss compensation.

[0041] Specifically, the process of training the neural network 101 by the electronic device 10 can include the following steps 1-5: Step 1, obtaining a training sample set. Here, each training sample in the training sample set can include sample packet loss audio, corresponding sample frame loss position information, and sample non-packet loss audio. Step 2, generating input features based on the sample packet loss audio and the corresponding sample frame loss position information. Step 3, inputting the input features into the neural network to be trained. In the present application scenario, the neural network to be trained can include an encoder layer, an intermediate layer, and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. Step 4, inputting the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and outputting a predicted fundamental frequency by the fundamental frequency prediction network. Step 5, updating the network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the real fundamental frequency calculated based on the sample non-packet loss audio. In addition, the network parameters of the entire neural network to be trained can also be updated based on the predicted results output by the decoder layer and the sample non-packet loss audio. Thus, the trained neural network 101 is obtained.

[0042] Afterwards, the electronic device 20 can obtain the trained neural network 101 from the electronic device 10, and input the to-be-processed audio and the frame loss position information corresponding to the to-be-processed audio into the neural network 101, so as to obtain the audio after packet loss compensation corresponding to the to-be-processed audio.

[0043] Referring to FIG. 2, FIG. 2 shows a flowchart of a method for training a neural network for voice packet loss compensation according to an embodiment. The method can be executed by any device, apparatus, platform, cluster of devices having computing and processing capabilities. As shown in FIG. 2, the method for training a neural network for voice packet loss compensation can include the following steps 201-205, in particular:

[0044] Step 201, obtaining a training sample set. In this embodiment, each training sample in the training sample set can include a sample packet loss audio, sample frame loss position information corresponding to the sample packet loss audio, and sample non-packet loss audio corresponding to the sample packet loss audio. The sample frame loss position information can be used to indicate the position of the frame in which data loss occurs in the sample packet loss audio. In this example, the sample packet loss audio can be audio at various sampling rates, for example, audio at a sampling rate of 48 kHz.

[0045] In practice, audio data is usually processed and transmitted in units of frames, and each frame can contain a certain length of audio. During network transmission, due to network congestion, delay or other reasons, some frames may not successfully reach their destination, resulting in packet loss. In order to detect and handle these packet loss situations, a flag bit is usually set in each frame of audio data. For example, this flag bit can be a binary bit, which is used to indicate whether the frame is complete and arrives. If the flag bit is set to indicate packet loss, the receiving end can detect the loss of this frame of data. Thus, the position of the frame in which data loss occurs in the audio can be determined according to the flag bit.

[0046] Step 202, generating input features based on the sample packet loss audio and the sample frame loss position information corresponding thereto. In this embodiment, the input features can be generated according to the sample packet loss audio and the sample frame loss position information corresponding thereto. For example, the sample packet loss audio can be first converted to the frequency domain, and then spliced with the sample frame loss position information to obtain the input features.

[0047] In some implementations, the above step 202 can include the following steps 1 and 2, in particular: Step 1, performing sub-band decomposition on the sample packet loss audio to obtain a plurality of sub-bands. Step 2, generating input features based on the conversion result of the plurality of sub-bands converted to the time-frequency domain and the sample frame loss position information.

[0048] In the present implementation, the sample loss audio can be subband-decomposed in various ways. For example, the sample loss audio can be subband-decomposed using a stable and efficient Pseudo-Quadrature Mirror Filter Bank (PQMF) to divide the sample loss audio into multiple subbands. In the PQMF, the subband decomposition can include a set of K FIR (Finite Impulse Response) filters. The subband decomposition process can include FIR analysis, down-sampling, and Short-Time Fourier Transform (STFT).

[0049] For example, assume y represents the original audio to be input to the neural network to be trained, y k represents the audio after subband analysis and down-sampling, k ∈ [1, K] is the subband number, y k may represent the kth subband. The sampling rate of y k is y may represent the frequency domain corresponding to y k after STFT. Y k is stacked along the channel dimension to obtain the frequency domain subband features After that, the input features can be directly concatenated with the frame loss position information, or the input features can be obtained after compression and concatenation with the frame loss position information. For example, the input features can be obtained after amplitude spectrum compression with a scaling factor of 0.5 and concatenation with the frame loss position information.

[0050] It can be understood that the subband decoding process corresponding to the foregoing subband decomposition process can include iSTFT (Inverse Short-Time Fourier Transform), up-sampling, and FIR synthesis. Corresponding to the subband analysis process, the output of the neural network is After inverse Fourier transform on each subband , the time domain subband audio is recovered to the audio In the present implementation, K can be 4, i.e., the audio signal is divided into 4 subbands for processing.

[0051] Through the present implementation, the sample loss audio can be decomposed into multiple subbands for processing, thereby greatly reducing the computational complexity. Therefore, the present implementation is particularly suitable for audio with a high sampling rate, e.g., audio with a sampling rate greater than or equal to 48 k Hz.

[0052] Step 203, input the input features into the neural network to be trained. In the embodiment, the obtained input features can be input into the neural network to be trained and processed by the neural network to be trained. Here, the neural network to be trained can include an encoder layer, an intermediate layer and a decoder layer. The intermediate layer is connected to the encoder layer and the decoder layer. The encoder layer and the decoder layer can be connected or not connected.

[0053] In some implementations, the neural network to be trained can be a neural network of U-Net structure, which mainly includes an encoder, a bottleneck layer and a decoder, and the bottleneck layer in the U-Net structure can be the intermediate layer. In the U-Net structure, the encoder and the decoder can be connected by skip connection.

[0054] In some implementations, the encoder layer can include several encoders. According to needs, each encoder can include various layers. For example, it can include convolution layers, pooling layers, etc. For example, each encoder can include a gated convolution layer and a time-frequency hole convolution layer, which can be used to extract features by time dimension and frequency dimension hole convolution. As shown in FIG. 3, FIG. 3 shows a schematic diagram of one example of an encoder structure. In the example shown in FIG. 3, the encoder can include a gated convolution layer and a time-frequency hole convolution layer in sequence, and the time-frequency hole convolution layer can include frequency dimension and time dimension dilated convolution FConv and TConv. FConv and TConv can be regarded as multi-scale modeling along the frequency axis and the time axis, which can fully perceive historical information. In addition, the time-frequency hole convolution layer can further include a BN (Batch Normalization) layer, a PReLU (Parametric Rectified Linear Unit) activation function, a PWConv (Pointwise convolution) and the like, and the PWConv can be used to align the input and output dimensions. Here, b dilated degrees from 1 to 2 b-1 convolution layers are spliced together to form a time-frequency hole convolution layer, which can effectively improve the receptive field of the convolution layer. It can be understood that the encoder structure shown in FIG. 3 is only schematic and is not a limitation on the encoder structure. In practice, the layers included in the encoder can be set according to actual needs.

[0055] At step 204, the features output by the intermediate layer are input into the pre-trained pitch prediction network, and a predicted pitch is output by the pitch prediction network. In the present embodiment, the intermediate layer can be used for packet loss compensation, and the intermediate layer can be implemented by various neural networks. For example, the intermediate layer can be implemented by a bidirectional GRU (Gate Recurrent Unit), which can extract the correlation between the frequency and time dimensions and perform packet loss compensation.

[0056] Thus, the features output by the intermediate layer can include the features of the frame corresponding to the frame loss position corresponding to the sample frame loss position information, and the pre-trained pitch prediction network can be used to predict the pitch according to the input features. Therefore, the features output by the intermediate layer are input into the pitch prediction network, and a predicted pitch can be output by the pitch prediction network. The predicted pitch can include the pitch of the frame corresponding to the frame loss position. In the present embodiment, the pitch prediction network can be various neural networks.

[0057] In some implementations, the pitch prediction network can include a Bi-LSTM (Bi-directional Long Short-Term Memory). As shown in FIG. 4, FIG. 4 shows a schematic diagram of one example of the pitch prediction network structure. In the example shown in FIG. 4, the pitch prediction network can include, in sequence, a reshape layer, a Maxpool layer, a reshape layer, a Bi-LSTM layer, a Linear layer, a reshape layer, a Linear_C layer, a Linear_F layer, and the like. Here, the output of the first reshape layer and the output of the Linear layer form a skip connection. The reshape layer can be used for data dimension adjustment, the Maxpool layer can be used as down-sampling, the Linear layer can be used for linear mapping, the Linear_C layer can represent linear mapping in the channel dimension, and the Linear_F can represent linear mapping in the frequency dimension. It can be understood that the pitch prediction network structure in FIG. 4 is merely schematic and is not a limitation on the pitch prediction network structure. In practice, the layers included in the pitch prediction network can be set according to actual needs.

[0058] As an example, the fundamental frequency prediction network can be trained in various ways. For example, the training can be performed in a supervised manner. As an example, in the supervised manner, a sample set can be first obtained, each sample in the sample set can include a sample audio feature and a fundamental frequency label corresponding to the sample audio feature. Here, the sample audio feature can be obtained through the network structure of the neural network to be trained. Then, the sample audio feature in the sample can be input into the fundamental frequency prediction network to be trained, and a predicted fundamental frequency can be output by the fundamental frequency prediction network to be trained. Then, a difference loss between the predicted fundamental frequency and the fundamental frequency label is calculated based on a predefined loss function, and the network parameters of the fundamental frequency prediction network are adjusted to minimize the difference loss.

[0059] At step 205, the network parameters of the encoder layer and the intermediate layer are adjusted based on the predicted fundamental frequency and the real fundamental frequency calculated based on the sample non-packet loss audio. In the embodiment, the real fundamental frequency of each frame can be calculated using the sample non-packet loss audio, and then the network parameters of the encoder layer and the intermediate layer can be adjusted based on the predicted fundamental frequency and the real fundamental frequency. As an example, the following formula can be used as a loss function:

[0060] where T can represent the number of frames, f i can represent the fundamental frequency of the i-th frame predicted by the fundamental frequency prediction network, i f i can represent the real fundamental frequency of the i-th frame, f0 L i can represent the difference loss of the fundamental frequency prediction. Then, the network parameters of the encoder layer and the intermediate layer can be adjusted to minimize L i. f0

[0061] In some implementations, corresponding to the encoder layer, the decoder layer can also include a plurality of decoders, each of which can include a first branch and a second branch in parallel. The first branch can be used to predict the real part of the audio, and the second branch can be used to predict the imaginary part of the audio. The neural network to be trained can output the sample predicted audio based on the predicted real part and imaginary part of the audio output by the encoder layer.

[0062] As shown in FIG. 5, FIG. 5 shows a schematic diagram of an example of a neural network structure to be trained. In the example shown in FIG. 5, the input can include the sample packet loss audio and the sample packet loss position information m(t), t∈T, B can represent the batch size, N can represent the audio length, and T can represent the number of frames. The output can include the sample predicted audio after packet loss compensation

[0063] In the example shown in FIG. 5, the sample packet loss audio ​The sample prediction audio output by the trained neural network can be input into at least one discriminator pre-trained to distinguish the sample prediction audio from the sample non-loss audio. The discriminator can output a discrimination result for the sample prediction audio. The discrimination result can be used to adjust the network parameters of the trained neural network. The trained neural network can be used to perform packet loss compensation on the audio signal.

[0064] It can be understood that the parts included in the trained neural network structure shown in FIG. 5 are only illustrative, and are not a limitation on the trained neural network structure. In practice, the structure of the trained neural network can be designed according to actual needs.

[0065] Based on the sample prediction audio output by the trained neural network, the method of training the neural network for performing packet loss compensation on the audio signal can further include the following steps S1 and S2. Specifically, in step S1, the sample prediction audio is input into at least one discriminator pre-trained, and each discriminator outputs a discrimination result for the sample prediction audio. In step S2, based on the at least one discrimination result, the sample prediction audio and the sample non-loss audio, a loss is calculated, and the network parameters of the trained neural network are adjusted based on the loss.

[0066] In the present implementation, the neural network to be trained can be trained by using a generative adversarial network (GAN). The neural network to be trained can be used as a generator, and the sample predicted audio output by the generator can be input to a discriminator, and the discriminator can output a discrimination result for the sample predicted audio. Then, the network parameters of the neural network to be trained can be adjusted based on the discrimination result output by the discriminator. In the present example, one or more discriminators can be used.

[0067] In some implementations, the at least one discriminator described above can include a first discriminator configured to discriminate a probability that the sample predicted audio is real audio, and can further include a second discriminator configured to discriminate an audio quality of the sample predicted audio.

[0068] For example, the first discriminator can include a frequency domain multi-resolution discriminator, a time domain multi-period discriminator, or the like. During training, the real non-packet loss audio and the audio generated by the generator can be input to the first discriminator, and the first discriminator can determine whether the input audio is the audio generated by the generator. The discriminator loss can be calculated, and the network parameters of the first discriminator can be updated by back propagation. In the present example, the loss of the first discriminator can be calculated by the following formula:

[0069] wherein s can represent the real non-packet loss audio, x can represent the packet loss audio corresponding to s, G can represent the generator, D can represent the discriminator, G(x) can represent the result of processing x by the generator. E[·] can represent an expected value, and the subscript thereof can represent an object of the expected value.

[0070] As described above, the first discriminator can include a frequency domain multi-resolution discriminator. As shown in FIG. 6, FIG. 6 shows a schematic diagram of an example of the frequency domain multi-resolution discriminator. In the example shown in FIG. 6, after the audio waveform is input to the frequency domain multi-resolution discriminator, the frequency domain multi-resolution discriminator can use a short-time Fourier transform (STFT) with different window lengths and window shifts to transform the time domain waveform. A two-dimensional convolution can be used to downsample the features of different frequency resolutions. For example, the window lengths can be [30, 60, 120, 240, 480, 960], and the input audio can be discriminated from different frequency resolutions. It can be understood that the frequency domain multi-resolution discriminator shown in FIG. 6 is a known existing discriminator, and the contents contained in each part thereof are as shown in the figure, which will not be described herein again.

[0071] The first discriminator can further include a time domain multi-period discriminator (MPD). As shown in FIG. 7, FIG. 7 shows a schematic diagram of an example of a time domain multi-period discriminator. In the example shown in FIG. 7, the time domain multi-period discriminator can fold the input one-dimensional sample point sequence into a two-dimensional plane with a certain period, and then apply two-dimensional convolution for processing. Specifically, each sub-discriminator of a specific period first performs padding to ensure that the number of sample points is an integer multiple of the period, so as to facilitate folding into a two-dimensional plane. Next, a plurality of convolution layers are entered, and the number of output channels can be [32, 128, 512, 1024], respectively, for example. After convolution, a leaky_relu activation is used. Finally, a convolution layer with an input channel number of 1024, for example, and an output channel of 1 is used for post-processing, and is flattened as the final output of the time domain multi-period discriminator. It can be understood that the time domain multi-period discriminator shown in FIG. 7 is a known existing discriminator, and the contents contained in each part thereof are as shown in the figure, which will not be described here.

[0072] As an example, the second discriminator can be a MetricGAN discriminator. The audio input after packet loss compensation can be input into the MetricGAN discriminator to estimate PESQ (Perceptual evaluation of speech quality), and the loss is calculated with the real PESQ obtained through a python library, and the MetricGAN discriminator is updated according to the loss. For example, the loss of the MetricGAN discriminator can be calculated through the following loss function:

[0073] where s can represent the real non-packet loss audio; x can represent the packet loss audio corresponding to s; G can represent the generator; D can represent the discriminator; G(x) can represent the result of processing x by the generator. Q' can represent the pypesq library function, may represent the calculation expectation.

[0074] As shown in FIG. 8, FIG. 8 shows a schematic diagram of an example of the training of the MetricGAN discriminator. In the example shown in FIG. 8, the MetricGAN discriminator can output a predicted PESQ score for an input. As an example, the MetricGAN discriminator can include four layers of down-sampling two-dimensional convolution followed by three layers of fully connected layers. The output of the MetricGAN discriminator is a predicted PESQ score. During the training phase, first, the predicted PESQ score and the real PESQ score output by the pypesq library function are used to calculate a MSE loss, and the network parameters of the MetricGAN discriminator are updated. Then, the MSE loss is calculated with the maximum PESQ, and the network parameters of the generator are updated.

[0075] In some implementations, the method of training the neural network for speech packet loss compensation described above can further include the following steps one to three, specifically: step one, input the sample predicted audio output by the neural network to be trained and its corresponding sample non-packet loss audio into a pre-trained speech recognition model respectively. In this example, the speech recognition model can be any existing speech recognition model, for example, it can be a whisper model. Step two, obtain the encoding layer features of the sample predicted audio and its corresponding sample non-packet loss audio in the speech recognition model. Step three, adjust the network parameters of the neural network to be trained based on the difference loss of the two obtained encoding layer features. For example, the network parameters of the neural network to be trained can be adjusted with the goal of minimizing the difference loss.

[0076] In this implementation, taking the whisper model as an example of the speech recognition model, the difference loss of the two encoding layer features can be calculated by the following formula:

[0077] wherein, W(s) can represent the encoding layer features of the sample non-packet loss audio corresponding to the whisper model.

[0078] It can be understood that the network parameters of the neural network to be trained can also be adjusted based on the sample predicted audio and its corresponding sample non-packet loss audio. For example, the difference loss of the sample predicted audio and its corresponding sample non-packet loss audio can be calculated, and the network parameters of the neural network to be trained can be adjusted with the goal of minimizing the difference loss. For example, the difference loss can be calculated by the following formula:

[0079] wherein, L maeThe mae (Mean Absolute Error) loss in the time domain can be represented as N can represent the audio length, s i The sample non-lossy audio at the i-th sampling point can be represented as The sample predicted audio at the i-th sampling point can be represented as

[0080] Here, L plcpa The MSE of the amplitude spectrum compression in the frequency domain can be represented as S(t, f) can represent the spectrum corresponding to the sample non-lossy audio, The spectrum corresponding to the sample predicted audio can be represented as t and f can represent the indices of the time dimension and the frequency dimension, respectively, p can represent the compression coefficient, j and The phase related to the Fourier transform can be represented as The MSE of the amplitude spectrum can be represented as The MSE of the phase can be represented as

[0081] For another example, the difference loss can also be calculated by the following formula:

[0082] Where s can represent the real non-lossy audio; x can represent the lossy audio corresponding to s; G can represent the generator; D can represent the discriminator. G(x) can represent the result after the generator processes x, PESQ max The maximum value of the PESQ index can be represented as

[0083] The above describes the training process of the neural network for voice lossy compensation, and the neural network thus obtained can perform lossy compensation on the input audio.

[0084] Next, please continue to refer to FIG. 9, which shows a neural network-based voice packet loss compensation method according to an embodiment. The method can be performed by any device, equipment, platform, cluster of equipment, etc. with computing and processing capabilities. As shown in FIG. 9, the neural network-based voice packet loss compensation method can include the following steps 901 to 903, specifically: step 901, obtaining a neural network pre-trained for voice packet loss compensation. In this embodiment, the neural network pre-trained for voice packet loss compensation by the method described in FIG. 2 can be obtained, which can perform packet loss compensation based on the packet loss audio and the frame loss position information corresponding to the packet loss audio. Step 902, receiving the audio to be processed and the frame loss position information corresponding to the audio to be processed. Step 903, inputting the input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain the audio after packet loss compensation corresponding to the audio to be processed. According to another aspect of the embodiment, a device for training a neural network for voice packet loss compensation is provided. The device for training a neural network for voice packet loss compensation described above can be deployed on any device, equipment, platform, cluster of equipment, etc. with computing and processing capabilities.

[0085] FIG. 10 shows a schematic block diagram of a device for training a neural network for voice packet loss compensation according to an embodiment. The device shown in FIG. 10 is used to perform the method shown in FIG. 2. The neural network to be trained includes an encoder layer, an intermediate layer, and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer. As shown in FIG. 10, the device 100 for training a neural network for voice packet loss compensation includes: an obtaining unit 1001 configured to obtain a training sample set, each training sample including a sample packet loss audio, sample frame loss position information corresponding to the sample packet loss audio, and a sample non-packet loss audio; a generating unit 1002 configured to generate input features based on the sample packet loss audio and the sample frame loss position information corresponding to the sample packet loss audio; a first input unit 1003 configured to input the input features into the neural network to be trained; a second input unit 1004 configured to input features output by the intermediate layer into a pre-trained fundamental frequency prediction network to output a predicted fundamental frequency by the fundamental frequency prediction network; and an adjusting unit 1005 configured to adjust network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and a true fundamental frequency calculated based on the sample non-packet loss audio.

[0086] In some optional implementations of the present embodiment, the neural network to be trained is a neural network of U-Net structure, and the intermediate layer is a bottleneck layer in the U-Net structure.

[0087] In some optional implementations of the present embodiment, the fundamental frequency prediction network includes a bidirectional long short-term memory network.

[0088] In some optional implementations of the present embodiment, the features output by the intermediate layer include features of the frame corresponding to the frame loss position corresponding to the sample loss frame position information, and the predicted fundamental frequency output by the fundamental frequency prediction network includes the fundamental frequency of the frame corresponding to the frame loss position.

[0089] In some optional implementations of the present embodiment, the generating unit 1002 is further configured to perform subband decomposition on the sample loss packet audio to obtain a plurality of subbands; and generate input features based on the conversion result of the plurality of subbands converted to the time-frequency domain and the sample loss frame position information.

[0090] In some optional implementations of the present embodiment, the encoder layer includes a plurality of encoders, each of which includes a gated convolution layer and a time-frequency space convolution layer, and the time-frequency space convolution layer is configured to extract features through time-dimension and frequency-dimension space convolution.

[0091] In some optional implementations of the present embodiment, the decoder layer includes a plurality of decoders, each of which includes a first branch and a second branch in parallel, the first branch is configured to predict the real part of the audio, and the second branch is configured to predict the imaginary part of the audio; the to-be-trained neural network outputs a sample predicted audio based on the predicted real part and imaginary part of the audio output by the encoder layer; and the apparatus 1000 further includes: a third input unit (not shown in the figure) configured to input the sample predicted audio into at least one discriminator pre-trained to output a discrimination result for the sample predicted audio from each discriminator; a computing unit (not shown in the figure) configured to calculate a loss based on the at least one discrimination result, the sample predicted audio and the sample non-loss packet audio, and adjust the network parameters of the to-be-trained neural network based on the loss.

[0092] In some optional implementations of the present embodiment, the at least one discriminator includes a first discriminator configured to discriminate the probability that the sample predicted audio is real audio and a second discriminator configured to discriminate the audio quality of the sample predicted audio.

[0093] In some optional implementations of the present embodiment, the apparatus 1000 further includes: a fourth input unit (not shown in the figure) configured to input the sample predicted audio output by the to-be-trained neural network and its corresponding sample non-loss packet audio into a pre-trained speech recognition model respectively; an encoding layer feature acquisition unit (not shown in the figure) configured to acquire the encoding layer features of the sample predicted audio and its corresponding sample non-loss packet audio in the speech recognition model; and a network parameter adjustment unit (not shown in the figure) configured to adjust the network parameters of the to-be-trained neural network based on the difference loss of the two acquired encoding layer features.

[0094] According to an embodiment of another aspect, a neural network based voice packet loss compensation apparatus is provided. The above-mentioned neural network based voice packet loss compensation apparatus can be deployed in any apparatus, device, platform, cluster of devices, etc. having computing and processing capabilities.

[0095] FIG. 11 shows a schematic block diagram of a neural network based voice packet loss compensation apparatus according to an embodiment. The apparatus shown in FIG. 11 is configured to perform the method shown in FIG. 9. As shown in FIG. 11, the neural network based voice packet loss compensation apparatus 1100 comprises: a model obtaining unit 1101 configured to obtain a neural network trained according to the method described in FIG. 2 for performing voice packet loss compensation; a receiving unit 1102 configured to receive a to-be-processed audio and frame loss position information corresponding to the to-be-processed audio; and a feature input unit 1103 configured to input, into the neural network, input features generated based on the to-be-processed audio and the frame loss position information corresponding to the to-be-processed audio, to obtain an audio after packet loss compensation corresponding to the to-be-processed audio.

[0096] The above-mentioned apparatus embodiments correspond to the method embodiments, and specific descriptions can be made with reference to the descriptions of the method embodiments, which will not be repeated here. The apparatus embodiments are based on the corresponding method embodiments and have the same technical effects as the corresponding method embodiments. Specific descriptions can be made with reference to the corresponding method embodiments.

[0097] According to an embodiment of another aspect, a computer readable storage medium having a computer program stored thereon is also provided. When the computer program is executed in a computer, the computer is caused to perform the method described in FIG. 2 or FIG. 9.

[0098] According to an embodiment of yet another aspect, an electronic device is also provided, which comprises a memory and a processor. The memory has executable codes stored therein. When the processor executes the executable codes, the method described in FIG. 2 or FIG. 9 is implemented.

[0099] The above describes specific embodiments of the present disclosure, and other embodiments are within the scope of the appended claims. In some cases, the acts or steps in a claim can be performed in an order different from the order in the embodiments, and a desired result can still be achieved. In addition, the processes depicted in the accompanying drawings do not necessarily have to be performed in the specific order or sequential order shown, in order to achieve a desired result. In some embodiments, multi-task processing and parallel processing are possible, or can be advantageous.

[0100] Reference is made to FIG. 12, which shows a structural schematic diagram of an electronic device 1200 suitable for implementing the embodiments of the present application. The electronic device shown in FIG. 12 is merely an example, and should not impose any limitation on the functions and use range of the embodiments of the present application.

[0101] As shown in FIG. 12, the electronic device 1200 can include a processing apparatus (e.g., a central processing unit, a graphics processing unit, etc.) 1201 that can perform various appropriate actions and processes according to programs stored in a read-only memory (ROM) 1202 or loaded into a random access memory (RAM) 1203 from a storage apparatus 1208. Various programs and data required for the operation of the electronic device 1200 are also stored in the RAM 1203. The processing apparatus 1201, the ROM 1202, and the RAM 1203 are connected to each other through a bus 1204. An input / output (I / O) interface 1205 is also connected to the bus 1204.

[0102] In general, the following devices can be connected to the I / O interface 1205: input devices 1206 including, for example, a touch screen, a touch pad, a keyboard, a mouse, etc.; output devices 1207 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage apparatuses 1208 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1209. The communication devices 1209 can allow the electronic device 1200 to perform wireless or wired communication with other devices to exchange data. Although FIG. 12 shows the electronic device 1200 having various devices, it should be understood that all of the illustrated devices are not required to be implemented or possessed. More or fewer devices can be alternatively implemented or possessed. Each block shown in FIG. 12 can represent a device or, as the case can be, multiple devices.

[0103] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods illustrated by the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network through the communication devices 1209, or installed from the storage apparatus 1208, or installed from the ROM 1202. When the computer program is executed by the processing apparatus 1201, the above-described functions defined in the methods of embodiments of the present application are performed.

[0104] The embodiments of the present disclosure also provide a computer readable storage medium, having stored thereon a computer program, which, when executed in a computer, causes the computer to perform the method provided by the present disclosure.

[0105] It should be noted that the computer readable medium in the embodiments of the present disclosure can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination of the above. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the embodiments of the present disclosure, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus. In the embodiments of the present disclosure, the computer readable signal medium can include a data signal carried in a baseband or as a part of a carrier wave, which carries computer readable program code. Such a propagated data signal can take various forms, including but not limited to an electromagnetic signal, an optical signal or any suitable combination of the above. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit a program for use by or in conjunction with an instruction execution system, device or apparatus. The program code contained in the computer readable medium can be transmitted by any suitable medium, including but not limited to a wire, an optical cable, an RF (Radio Frequency) or the like, or any suitable combination of the above.

[0106] The computer readable medium described above can be contained in the electronic device described above; or can exist separately and not be assembled into the electronic device. The computer readable medium described above carries one or more programs, which, when executed by the electronic device, cause the electronic device to: obtain a training sample set, wherein each training sample includes a sample packet loss audio and corresponding sample frame loss position information and a sample non-packet loss audio; generate an input feature based on the sample packet loss audio and the corresponding sample frame loss position information; input the input feature into the neural network to be trained, wherein the neural network to be trained includes an encoder layer, an intermediate layer and a decoder layer, and the intermediate layer is connected between the encoder layer and the decoder layer; input a feature output by the intermediate layer into a pre-trained fundamental frequency prediction network, and output a predicted fundamental frequency by the fundamental frequency prediction network; and adjust network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and a real fundamental frequency calculated based on the sample non-packet loss audio.

[0107] Computer program code for carrying out operations of embodiments of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or electronic device. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0108] The various embodiments in the present disclosure are described in progressive manner, and the same or similar parts among the various embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, the storage medium and computing device embodiments are described simply because they are substantially similar to the method embodiments, and the relevant parts can be referred to the description of the method embodiments.

[0109] Those skilled in the art will appreciate that the functions described in the above one or more examples can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium.

[0110] The above detailed description has disclosed the purpose, technical solutions and advantages of the embodiments of the present disclosure. It should be understood that the above description is only a specific implementation of the embodiments of the present disclosure, and is not intended to limit the protection scope of the present disclosure. Any modification, equivalent replacement, improvement, etc. made on the basis of the technical solutions of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for compensating speech packet loss based on a neural network, comprising: Obtaining a pre-trained neural network for voice packet loss compensation; Receiving audio to be processed and frame loss position information corresponding to the audio to be processed; as well as Input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed are input into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed.

2. A method for training a neural network for speech packet loss compensation, wherein: The neural network to be trained includes an encoder layer, an intermediate layer and a decoder layer, wherein the intermediate layer is connected between the encoder layer and the decoder layer, and the method includes: Obtaining a training sample set, wherein each training sample includes sample packet loss audio, its corresponding sample frame loss position information, and sample non-packet loss audio; Generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; Inputting the input features into the neural network to be trained; Inputting the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, and having the fundamental frequency prediction network output a predicted fundamental frequency; and Based on the predicted fundamental frequency and the actual fundamental frequency calculated based on the sampled unpacked audio, network parameters of the encoder layer and the intermediate layer are adjusted.

3. The method according to claim 2, wherein: The neural network to be trained is a neural network with a U-Net structure, wherein the intermediate layer is a bottleneck layer in the U-Net structure.

4. The method according to claim 2, wherein: The fundamental frequency prediction network includes a bidirectional long short-term memory network.

5. The method according to claim 2, wherein: The features output by the intermediate layer include features of the frame corresponding to the frame loss position corresponding to the sample frame loss position information, and the predicted fundamental frequency output by the fundamental frequency prediction network includes the fundamental frequency of the frame corresponding to the frame loss position.

6. The method according to claim 2, wherein: The generating of input features based on the sample packet loss audio and its corresponding sample frame loss position information includes: Performing sub-band decomposition on the sample packet loss audio to obtain multiple sub-bands; An input feature is generated based on the conversion results of the multiple sub-bands into the time-frequency domain and the sample frame loss position information.

7. The method according to claim 2, wherein: The encoder layer includes multiple encoders, each encoder includes a gated convolution layer and a time-frequency dilated convolution layer, and the time-frequency dilated convolution layer is used to extract features through dilated convolution in the time dimension and the frequency dimension.

8. The method according to claim 2, wherein: The decoder layer includes a plurality of decoders, each decoder includes a first branch and a second branch in parallel, the first branch is used to predict the real part of the audio, and the second branch is used to predict the imaginary part of the audio; The neural network to be trained outputs sample predicted audio based on the real part and the imaginary part of the predicted audio output by the encoder layer; And, the method further comprises: Inputting the sample prediction audio into at least one pre-trained discriminator, and having each discriminator output a discrimination result for the sample prediction audio; Based on at least one discrimination result, the sample predicted audio and the sample non-packet-loss audio, a loss is calculated, and network parameters of the neural network to be trained are adjusted based on the loss.

9. The method according to claim 8, wherein The at least one discriminator includes a first discriminator for discriminating the probability that the sample predicted audio is real audio and a second discriminator for discriminating the audio quality of the sample predicted audio.

10. The method according to any one of claims 2 to 9, wherein: The method further comprises: Inputting the sample prediction audio output by the neural network to be trained and its corresponding sample unpacked audio into the pre-trained speech recognition model respectively; Obtaining coding layer features of the sample predicted audio and its corresponding sample unpacked audio in the speech recognition model; and Based on the obtained difference loss of the two coding layer features, the network parameters of the neural network to be trained are adjusted.

11. A device for training a neural network for voice packet loss compensation, wherein: The neural network to be trained includes an encoder layer, an intermediate layer and a decoder layer, wherein the intermediate layer is connected between the encoder layer and the decoder layer, and the apparatus includes: An acquisition unit is configured to acquire a training sample set, wherein each training sample includes a sample packet loss audio, its corresponding sample frame loss position information, and a sample non-packet loss audio; A generating unit configured to generate input features based on the sample packet loss audio and its corresponding sample frame loss position information; A first input unit is configured to input the input feature into the neural network to be trained; A second input unit is configured to input the features output by the intermediate layer into a pre-trained fundamental frequency prediction network, so that the fundamental frequency prediction network outputs a predicted fundamental frequency; and The adjustment unit is configured to adjust network parameters of the encoder layer and the intermediate layer based on the predicted fundamental frequency and the real fundamental frequency calculated based on the sampled unpacked audio.

12. A speech packet loss compensation device based on a neural network, comprising: A model acquisition unit configured to acquire a pre-trained neural network for voice packet loss compensation; A receiving unit configured to receive audio to be processed and frame loss position information corresponding to the audio to be processed; as well as The feature input unit is configured to input input features generated based on the audio to be processed and the frame loss position information corresponding to the audio to be processed into the neural network to obtain audio after packet loss compensation corresponding to the audio to be processed.

13. A computer-readable storage medium having a computer program stored thereon, which, when executed in a computer, causes the computer to execute the method according to any one of claims 1 to 10.

14. An electronic device comprising a memory and a processor, wherein the memory stores executable code, and when the processor executes the executable code, the method according to any one of claims 1 to 10 is implemented.

Images