Real-time packet loss concealment using deep generative networks

Abstract

To provide a method and system for executing packet loss concealment using a neural network system.SOLUTION: A method includes the steps of obtaining representation of an incomplete audio signal; inputting the representation of the incomplete audio signal to an encoder neural network; and outputting latent representation of a predicted complete audio signal. The latent representation is input to a decoder neural network, which outputs the representation of the predicted complete audio signal including reconfiguration of an original part of the complete audio signal, wherein the encoder neural network and decoder neural network are trained with adversarial networks.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] [Cross - reference to Related Applications] This application claims priority to the following prior applications: Spanish application P202031040 (reference number: D20093ES) filed on October 15, 2020; US provisional application 63 / 126,123 (reference number: D20093USP1), filed on December 16, 2020; Spanish application P202130258 (reference number: D20093ESP2), filed on March 24, 2021, US provisional application 63 / 195,831 (reference number: D20093USP2), filed on June 2, 2021. These applications are hereby incorporated by reference.

[0002] [Technical Field] This disclosure relates to a method of performing packet loss concealment using a neural network system, a method of training a neural network system for packet loss concealment, and a computer - implemented neural network system implementing the above methods.

Background Art

[0003] Most implementations within the field of communication technology operate under constrained real - time conditions so that users do not experience delays or interruptions in communication. Voice over Internet Protocol (VoIP) is an example of a communication protocol that operates under strict real - time conditions and enables users to have natural conversations. To meet strict conditions, VoIP and similar communication protocols rely on a steady stream of packets, each transmitting a part of the communication signal, which are transmitted continuously and without interruption from a transmitting entity to a receiving entity. However, in reality, packets are often delayed, delivered to the receiving entity in the wrong order, or completely lost, resulting in distortions or interruptions of the communication signal that significantly degrade the communication quality experienced by the user.

[0004] For this reason, an improved method for performing packet loss concealment (PLC) is needed. [Overview of the project]

[0005] Previous solutions to conceal packet loss involved duplicating the structure of the most recent packet and employing audio processing to attenuate signal energy, naturally extending the duration of the most recent packet instead of the next one. However, while these previous solutions reduced the visibility of lost packets to some extent, the interruption of the communication signal still degrades the quality of the communication signal, and users can still perceive distortion in the communication signal even after PLC processing, especially in the case of prolonged interruptions.

[0006] For this purpose, the objective of this disclosure is to provide improved methods and systems for performing packet loss concealment.

[0007] According to a first aspect of the disclosure, a method is provided for packet loss concealment of an incomplete audio signal, wherein the incomplete audio signal includes an alternative signal portion that replaces the original signal portion of the complete audio signal. The method includes the steps of: obtaining a representation of the incomplete audio signal; and inputting the representation of the incomplete audio signal into an encoder neural network, which is trained to predict a latent representation of the complete audio signal given the representation of the incomplete audio signal. The encoder neural network outputs a predicted latent representation of the complete audio signal, which is input into a decoder neural network, which is trained to predict a representation of the complete audio signal given the latent representation of the complete audio signal. The decoder neural network outputs a predicted representation of the complete audio signal, which includes a reconstruction of the original portion of the complete audio signal. The encoder neural network and the decoder neural network are trained with adversarial neural networks.

[0008] Alternate signal portions refer to signal portions that replace the corresponding (actual) portion of a complete audio signal. For example, an alternate signal portion may be a silent (zero) signal portion indicating that part of the signal is lost, corrupted, or missing. While zero signals are commonly used to indicate parts of the signal that are not received or are missing, other alternate signal portions may also be used. For example, a sawtooth signal at a specific frequency, or other predetermined types of signals established to represent alternate signal portions in place of the actual signal portions of a complete audio signal. In some implementations, missing signal portions are indicated as metadata to distinguish alternate signal portions from, for example, actual completely silent (zero) signal portions.

[0009] In some implementations, the method involves the steps of: quantizing the latent representation of a complete audio signal to obtain a quantized latent representation, the quantized latent representation being formed by selecting a set of tokens from a given lexical token set; The steps include inputting the quantized latent representation to the decoder neural network, The process further includes: At least one token of the quantized latent representation is used to tune a generative neural network, which is trained to predict a token in the token set given at least one different token in the token set, and the generative latent model outputs the predicted token of the latent representation and the confidence metric associated with the predicted token. Based on the confidence of the predicted token, the corresponding token in the quantized latent representation is replaced with the predicted token of the generative model, forming a modified token set (modified quantized representation), which is then provided to a decoder neural network.

[0010] The neural network system described above includes a deep causal adversarial autoencoder formed by encoder and decoder neural networks that have learned together to generate a reconstructed, complete audio signal representation when an incomplete audio signal is provided. A causal autoencoder is a non-autoregressive model that can predict any long signal portion (e.g., spanning multiple packets) in a single inference step. In some implementations, the decoder directly outputs a waveform representation, and for adversarial training purposes, the output waveform may be a highly accurate reconstruction of the complete audio signal. Unlike most existing models that use autoregressive loops to generate frame-by-frame, the causal autoencoder can generate a reconstructed, complete audio signal with alternative signal portions exceeding 100 milliseconds.

[0011] Because causal autoencoders are time-independent, the model can output a reconstructed audio signal of any length (at any sample rate) in a single feedforward step. This is in contrast to the vast majority of state-of-the-art packet loss hiding solutions, which employ some form of autoregressive loop from output to input. Using an optional generative latent model can enhance packet loss hiding performance against long-term losses. Furthermore, causal autoencoders have a deterministic meaning, where the same input produces the same output. Nevertheless, because causal autoencoders produce data that emulates real data, they are sometimes called generators in adversarial generative training settings. This is in contrast to other generators, which rely on random variables to perform the generation of new data, which is not a deterministic process.

[0012] In some implementations, causal autoencoders are supported by generative latent models that operate based on the quantized latent representation of the causal autoencoder. Generative latent models enable signal reconstruction over longer time periods, such as over 100 milliseconds, but also allow for easy reconstruction for arbitrary lengths.

[0013] According to a second aspect of the disclosure, a computer-implemented neural network system is provided for packet loss concealment of an audio signal, wherein the audio signal includes an alternate signal portion that replaces the original signal portion of the complete audio signal. The system includes an input unit configured to obtain a representation of an incomplete audio signal, and an encoder neural network configured to receive the incomplete audio signal representation and output the predicted latent representation of the complete audio signal, given the incomplete audio signal representation. The neural network system further includes a decoder neural network configured to receive the predicted latent representation of the complete audio signal and output the reconstructed representation of the complete audio signal, given the reconstructed latent representation of the complete audio signal, and an output unit configured to output the predicted representation of the complete audio signal, including a reconstruction of the original portion of the complete audio signal, wherein the encoder neural network and the decoder neural network are trained with adversarial neural networks.

[0014] According to a third aspect of this disclosure, a method for training a neural network system for packet loss concealment, the method is: Steps to obtain a neural network system for packet loss concealment, Steps to obtain a discriminative neural network, Steps to obtain a set of training data, The process involves training a neural network system in a generative adversarial training mode using a set of training data, by providing the neural network system with a set of training data and providing the output of the neural network system to a discriminative neural network, thereby training the neural network system in conjunction with the discriminative neural network. A method including this is provided.

[0015] In some implementations, the classifier includes at least two classifier branches operating at different sampling rates, and the method further includes the step of determining an aggregated likelihood index based on the individual indicators of the at least two classifier branches.

[0016] The disclosures in the second and third embodiments feature the same or equivalent embodiments and advantages as those in the disclosure in the first embodiment. For example, the encoders and decoders of a neural network system may be trained using a classifier having at least two classifier branches operating at different sample rates. Furthermore, the functions described in relation to the method may have corresponding structural features in a system or code that performs such functions in a computer program product. [Brief explanation of the drawing]

[0017] The following describes an embodiment with reference to the attached drawings. [Figure 1] This disclosure shows a causal autoencoder that implements an encoder and decoder neural network according to the embodiments of this disclosure. [Figure 2] Detailed diagrams of generative models operating on latent representations output by encoder neural networks are shown according to some embodiments of this disclosure. [Figure 3A] A detailed diagram of a causal autoencoder is shown according to one exemplary embodiment of the present disclosure. [Figure 3B] Detailed diagrams of residual causal convolution blocks in decoder neural networks are shown according to some embodiments of this disclosure. [Figure 3C] According to some embodiments of the present disclosure, a detailed diagram of an alternative residual causal convolutional block of a decoder neural network is shown. [Figure 4A] According to some embodiments of the present disclosure, an identifier with three discriminator branches is shown. [Figure 4B] According to an exemplary embodiment of the present disclosure, a detailed diagram of an identifier is shown. [Figure 5] According to some embodiments of the present disclosure, a training process employed during generator - identifier training is shown. [Figure 6] According to some embodiments of the present disclosure, an exemplary training setting for a causal auto - encoder is shown. [Figure 7] According to some embodiments of the present disclosure, a flowchart showing a method of performing packet loss concealment is shown. [Figure 8] According to some embodiments of the present disclosure, a flowchart showing a method of training a causal auto - encoder with an identifier is shown.

Modes for Carrying Out the Invention

[0018] Figure 1 schematically shows a causal auto-encoder. An incomplete audio signal 10 including an alternative signal portion 11 is input into the causal auto-encoder optionally via a conversion unit 110. The conversion unit 110 converts the incomplete audio signal 10 into a representation of the incomplete audio signal 10. For example, the conversion unit 110 can convert the time-domain (waveform) representation of the incomplete audio signal 10 into a representation format selected from a group including a frequency-domain representation, a time representation, a filter bank representation, and a feature-domain representation. Examples of such representations include a Mel-frequency cepstral coefficient representation and a short-time Fourier transform representation. The encoder neural network 120 can obtain the representation of the incomplete audio signal 10 from the conversion unit 110. Alternatively, the incomplete audio signal 10 can be directly obtained by the encoder neural network 120, for example, in a waveform time-domain representation.

[0019] The incomplete audio signal 10 may be subdivided into one or more frames such that the encoder neural network 120 accepts one or more frames for each inference step. For example, the encoder neural network 120 can have a 600-millisecond reception field, and short-time Fourier transform (STFT) spectral frames are generated from the incomplete audio signal 10 at 10-millisecond intervals with some overlap (resulting in a signal sampled at 100 Hz), meaning that the encoder neural network 120 accepts 60 frames for each inference step. The incomplete audio signal 10 is further divided into a set of packets, and each packet contains a representation of one or more frames or a part of the complete audio signal. If one or more packets from the set of packets are omitted (which occurs during packet loss), the signal portion and / or one or more frames that were present in the complete audio signal are not available, and the incomplete audio signal 10 becomes a representation of available information having an alternative signal portion 11 that replaces the signal portion of the complete audio signal that is thereby unavailable.

[0020] The encoder neural network 120 is trained to predict and output a latent representation of the reconstructed complete audio signal 20. That is, the latent representation of the reconstructed complete audio signal 20 is a prediction of the original complete audio signal given an incomplete audio signal 10 (or its representation), and the reconstructed complete audio signal 20 includes a reconstructed signal portion 21 that replaces the alternative signal portion 11 of the incomplete audio signal 10. The latent representation can be quantized and processed using a generative model 130, which is described in detail in relation to Figure 2. Optionally, the latent representation is provided as input to the decoder neural network 140. The decoder neural network 140 can be trained to directly predict the temporal representation (e.g., waveform) of the reconstructed complete audio signal 20, regardless of whether the encoder neural network 120 has received a waveform representation of the incomplete audio signal 10 or any other representation. Alternatively, the decoder neural network 140 can output a representation of the reconstructed complete audio signal 20, which is converted to a waveform representation by an optional inverse unit 150.

[0021] The receive field of the encoder neural network 120 should ideally be wide enough to capture a long context so that it is resilient to recent signal loss that may occur in close proximity to the current signal portion being reconstructed. The receive field may be approximately 600 milliseconds, and reducing the receive field (e.g., to 100 milliseconds) may degrade the reconstruction quality.

[0022] Figure 2 shows optional quantization and generative models 130 of the causal autoencoder. Optional generative latent models (GLMs) 160a and 160b operate autoregressively on the quantized representations 22a and 22b of the latent representation. As is clear from Figure 2, GLMs 160a and 160b operate in the latent representation domain. The latent representation is sometimes called the context domain, where the audio signal is represented, for example, by a context vector.

[0023] The latent representation output by the encoder neural network 120 is supplied to the quantization block 131. The quantization block 131 performs at least one transformation on the latent representation to form at least one quantized latent representation 22a, 22b. In some implementations, the quantization block 131 performs at least one linear transformation on the latent representation to form at least one quantized latent representation 22a, 22b. In some implementations, the quantization block 131 performs quantization of the latent representation by selecting a predetermined number of tokens to represent the latent representation, the tokens being selected from a predetermined set of possible lexical tokens. For example, the quantization may be vector quantization, where a predetermined number of quantization vectors are selected from a predetermined codebook of quantization vectors to describe the latent representation as quantized latent representations 22a, 22b.

[0024] In some implementations, quantization of quantization block 131 involves selecting a first set of tokens from a first vocabulary to form a first quantized representation 22a, and selecting a second set of tokens from a second (different) vocabulary to form a second quantized representation 22b. The number of tokens in each of the first and second sets may be the same. The first and second token sets are alternative representations of the same latent representation. Thus, quantization block 131 can provide 1, 2, 3, or more quantized latent representations 22a, 22b as outputs. For example, quantization block 131 can be a multi-head vector (VQ) quantization block.

[0025] Quantized latent representations 22a and 22b may be provided to the decoder neural network 140. The quantized latent representation can be of any dimension, for example, using 64, 512, 800, or 1024 tokens or quantization vectors.

[0026] In some implementations, each of at least one quantized latent representations 22a, 22b is associated with each of the GLMs 160a, 160b that operate autoregressively on the associated set of tokens 22a, 22b. GLM 160a is trained to predict the likelihood of at least one token given at least one other token in the token set that forms the quantized representation 22a. For example, GLM 160a may be trained to predict at least one future token (selected from lexical tokens) given at least one previous token. GLM 160a may be trained to predict the likelihood associated with each token in the token set from the quantized latent representation 22a, where the likelihood indicates the probability that the token is at least one specific token from the associated lexical tokens. In other words, GLM 160a can continuously predict new tokens given past tokens, or predict modifications to the current token set, where the new or modified predicted token is the same as or different from the token output by quantization block 131. In comparison block 132, the predicted token sequence 23a is compared with the token sequence 22a output by quantization block 131. If there is a difference of at least one predicted token between the GLM predicted token set 23a and the token set 22a output from quantization block 131, a selection is made to use one of the predicted tokens from either token set 22a or 23a. For example, token selection is based on the likelihood of the token predicted by GLM 160a and / or encoder 120. For example, if the GLM predicted likelihood is below a predetermined likelihood threshold, the token in quantization block 131 is a block and is used. As a further example, token selection is based on the likelihood of each token predicted by GLM 160a and / or the Argmin distance with respect to the unquantized latent representation.

[0027] Similarly, in parallel, given at least one token of the second latent representation 22b, the second GLM 160b can predict at least one future / modified token so as to form the predicted second token set 23b. Likewise, based on the likelihood of the token predicted by the second GLM 160b, the second token output by the quantized block 131 is compared to the second predicted token output by the second GLM 160b. If a difference is detected, a token selection is made based on the likelihood of the second token predicted by the second GLM 160b and / or the Argmin distance to the unquantized latent representation.

[0028] Similarly, three or more quantized latent representations are obtained, each by an associated GLM that performs predictions about possible continuations / modifications of the token sequence, which may differ from the actual sequence output by quantization block 131.

[0029] When a single quantized latent representation 22a and GLM160a are used, the most likely token sequence can be transferred to the decoder neural network 140 to perform waveform prediction based on the quantized latent representation selected by the comparison block 132.

[0030] When multiple quantized latent representations 22a, 22b and GLMs 160a, 160b are used, each quantized latent representation 22a, 23b selected by the comparison block 131 can be concatenated or added to in the concatenation block 133 to form an aggregated representation 24. Therefore, the bitrate can be increased by concatenating additional quantized representations 22a, 23b and transferring the aggregated representation 24 to the decoder neural network 140.

[0031] GLM160a and 160b are discrete autoregressive models trained on the maximum likelihood criterion. Each GLM160a and 160b can be configured to operate similarly to language models in the natural language processing domain. Thus, several neural architectures can be used to perform the tasks of GLM160a and 160b, such as one or more causal convolutional networks, recurrent neural networks, or self-attention models. Due to their generative nature, GLM160a and 160b can perform longer-term predictions in addition to causal adversarial autoencoders. Any large continuation of a latent representation can be predicted by GLM160a and 160b. Note that quantization and GLM160a and 160b are optional and may be added to causal adversarial autoencoders to facilitate longer-term predictions. For example, GLM160a and 160b may operate adaptively and be activated only in response to the alternative signal portion exceeding a threshold duration.

[0032] Figure 3A shows a detailed exemplary implementation of a causal adversarial autoencoder. The encoder neural network 120 can contain one or more encoder blocks 120', each containing one or more neural network layers 121, 122, 123. Multiple encoder blocks 120' can be cascaded with each encoder block 120', each having the same structure. In one implementation, one encoder block 120' may contain a normalization layer 121. Normalization of the layer can help accelerate the convergence of the model and obtain better signal reconstruction quality. The normalization layer 121 may be followed by a causal convolution layer 122 configured to perform causal convolution. The causal convolution layer 122 may use weight normalization because it is a simple stabilization mechanism for the adversarial model that generates the waveform. The causal convolution layer 122 may be an extended causal convolution layer with an extension coefficient d. Therefore, encoder 120 and encoder block 120' can be easily adapted with respect to modifications of the received field thanks to the expansion factor d per encoder block 120'. In some implementations, the received field can be widened to improve prediction quality. For example, the received field can be longer than 1 second, longer than 2 seconds, or longer than 3 seconds. Each encoder block 120' can include skip connections to facilitate training.

[0033] Some implementations include quantization of the latent representation output by encoder block 120'. The quantization may include one or more linear transformations of the output of encoder block 120'. The linear transformations can be performed by a linear transformation block 131' which outputs at least one quantized latent representation 22 representing the reconstructed complete audio signal.

[0034] The latent representation or quantized latent representation 22 is provided to the decoder neural network 140. The decoder neural network may include one or more cascaded decoder blocks 140', each decoder block 140' containing one or more neural network layers. Optionally, there is a causal convolution layer 133 prior to the decoder blocks 140' that performs initial upsampling of the latent representation or quantized latent representation 22.

[0035] In one implementation, the decoder block 140' includes a Leaky ReLU (Rectified Linear Unit) layer 141. Using the Leaky ReLU layer 141 as a nonlinear activation can mitigate gradient flow problems. Following the Leaky ReLU layer 141 is a transposed convolution layer 142, which may be followed by one or more residual causal convolutional blocks 143a, 143b, 143c with different expansion coefficients D. In one implementation, the expansion coefficients D of the residual causal convolutional layers 143a, 143b, 143c increase, for example, the first expansion coefficient is 1, the second is D, the third is D 2 Thereafter, D is an integer. Examples of residual causal convolution blocks 143a, 143b, and 143c are shown in detail in Figures 3B and 3C. Thus, the decoder block 140' converts the latent representation or quantized latent representation 22 back into a waveform representation of the reconstructed complete audio signal 20 through a series of transposed convolutions (i.e., learnable upsampling filters) and residual causal convolution blocks 143a, 143b, and 143c with different expansion coefficients. Alternatively, the decoder block outputs a representation of the reconstructed complete audio signal 20, where the waveform representation of the reconstructed complete audio signal is obtained by an inverse transform unit (not shown).

[0036] In some implementations, the output of decoder block 140' is provided to one or more post-processing layers 171, 172. In one exemplary implementation, the post-processing layers 171, 172 include a linear transformation layer 171 with a nonlinear (e.g., Tanh) activation 172.

[0037] The final sampling rate of the reconstructed complete audio signal 20 is determined by the number of transposed convolutions 142 (i.e., the number of cascaded decoder blocks 140') and their striding coefficients. In one implementation, the decoder 140 includes one or more decoder blocks 140', such as four decoder blocks 140' with different upsampling coefficients. For example, the upsampling coefficients could be 5, 4, 4, and 2 for each decoder block 140'. However, other coefficients may be adopted and fewer or more decoder blocks 140' may be stacked to obtain any sampling rate of the output reconstructed audio signal 20. The transposed convolutions can be restricted to not overlap so that causality is not broken during upsampling (i.e., there is no overlap between transposed convolution outputs from future data).

[0038] In some implementations, the causal adversarial autoencoder may include an optional crossphasing post-filtering module configured to receive at least one reconstructed signal portion 21 and subsequent signal portions (e.g., signal portions shown to be representations of the complete audio signal portion, rather than alternative signal portions) and apply a crossphasing filter (e.g., a window function) to ensure a smooth transition between the reconstructed audio signal portion 21 and the complete audio signal portion. The crossphasing filter can then be applied to the reconstructed audio signal 20. In some implementations, the optional crossphasing post-filtering module may include one or more neural networks trained to predict a crossphasing filter, given at least the reconstructed signal portion and the subsequent and / or preceding portions of the complete audio signal. The advantage of using neural networks is that they can be trained to adapt the predicted crossphasing filter after different acoustic conditions present in the training data (e.g., noise, codec artifacts, reverberation effects, etc.).

[0039] Figure 3B shows an exemplary residual causal convolution block 143, which includes a 1x1 causal convolution block 1432 with an expansion coefficient D and skip connections. As shown in Figure 3C, the residual causal convolution block 143 may include Leaky ReLU activation via a Leaky ReLU layer 1431 preceding the causal convolution layer 1432. Optionally, the causal convolution layer 1432 may be followed by a second Leaky ReLU layer 1433, then a linear transformation layer 1434. Each of the one or more causal convolution blocks 143 may include its own skip connections to facilitate model training.

[0040] Figure 4A shows a classifier 200 according to some embodiments of the present disclosure. The classifier 200 includes a classifier neural network, which may include multiple neural network layers. The classifier 200 is trained to predict an indicator 25 indicating whether an input complete audio signal 20 represents a (fake) reconstructed audio signal or a (real) complete audio signal. The classifier 200 may be configured to output an indicator 25 indicating whether the input data 20 represents a complete audio signal or an audio signal that includes at least a reconstructed portion. The indicator 25 may be a Boolean variable or a value between 0 and 1, where 0 indicates the input audio signal is fake and 1 indicates the input audio signal is real.

[0041] In some implementations, the classifier 200 includes two, three, or more classifier branches 210a, 210b, 210c, each trained to predict individual indicators 25a, 25b, 25c indicating whether the input data represents a complete audio signal or a reconstructed audio signal. In one example, the first classifier branch 210a obtains a representation of the input audio signal 20, while the second classifier branch 210b obtains a downsampled representation of the same input audio signal 20. Furthermore, the third classifier branch 210c can obtain a further downsampled representation of the same input audio signal 20. For this purpose, there is a downsampling stage 211 before the second classifier branch 210b, while there are two downsampling stages 211 before the third classifier branch 210c. Each downsampling stage can perform downsampling using the same coefficient S or individual coefficients S1, S2, where S2 ≠ S1.

[0042] Therefore, each discriminator branch 210a, 210b, 210c predicts individual indicators 25a, 25b, 25c indicating whether the input audio signal 20 appears as a complete audio signal or a reconstructed audio signal at different sampling rates. Each indicator 25a, 25b, 25c is aggregated in the indicator aggregation stage 212 to form a composite indicator 25 indicating whether the input audio signal 20 is a complete audio signal or a reconstructed audio signal. The indicator aggregation stage 212 can determine the composite indicator 25 based on the number of discriminator branches 210a, 210b, 210c indicating whether the input audio signal is real or fake. The indicator aggregation stage 212 can determine the composite indicator 25 based on a weighted sum of the individual indicators 25a, 25b, 25c of each discriminator branch 210a, 210b, 210c. The weighted sum can be weighted by the likelihood associated with each individual indicator 25a, 25b, 25c. Other aggregation or pooling strategies can be employed to generate a composite indicator 25 from the individual indicators 25a, 25b, and 25c. For example, the most reliable individual indicator of the individual indicators 25a, 25b, and 25c can be considered as the composite indicator 25 (maximum pooling). In other words, the composite indicator can be, for example, the average, weighted average, or maximum of the individual indicators 25a, 25b, and 25c.

[0043] Figure 4b shows a detailed diagram of the classifier 210 or classifier branch 210. A reconstructed complete audio signal or a complete audio signal is provided as input to the input classifier 210. The input classifier 210 is configured to predict an indicator 25 that indicates whether the input is a reconstructed complete audio signal 20 or a complete audio signal that has not been reconstructed using the neural system described above. The classifier may include multiple neural network layers 211a, 212a, 211b, 212b, 211c, 212c, 211d, 212d. For example, the multiple neural network layers 211a, 212a, 211b, 212b, 211c, 212c, 211d, 212d may be a stack of subblocks, each subblock containing a convolutional layer 211a and a Leaky ReLU activation layer 212a. Each convolutional layer 211a, 211b, 211c, and 211d can be configured to downsample by a coefficient S. The convolutional layers 211a, 211b, 211c, and 211d of the classifier can have relatively large kernels, for example, the kernel can be 10 times the stride + 1. In classifier 210, the time resolution of each feature map is reduced, and the number of features is increased to compensate for the reduction in time dimension. The output convolutional layer 213 can be configured to perform the final prediction of the classifier 210's indicator 25. All convolutional layers of the classifier can use weight normalization to facilitate training stabilization.

[0044] Referring to Figure 5, an exemplary Generative Adversarial Network (GAN) is shown, which includes two neural networks, a generative neural network 100 and a discriminative neural network 200, trained together for opposing purposes. The generative neural network 100 is called the generator 100, and learns to mimic the structures embedded in the training data 310b that are provided to the generator 100, so that the generator 100 can generate more samples that are reasonable substitutes for those present in the training data 310b.

[0045] In GAN training, the training data 310b input to the generator 100 may be a vector of random noise samples z with a certain distribution Z. The distribution may be a uniform distribution or a Gaussian distribution, but other distributions are also possible. This makes the generator 100 a generator of random samples similar to the samples in the training data 310b. As an addition or alternative, training data 310a, which includes examples of audio data (e.g., recorded speech or general speech), can also be used as training data during GAN training. For example, using incomplete training data, the generator 100 may be tasked with predicting the continuation of the training signal as realistically as possible or filling in alternative (missing) signal portions of the training data. For example, if the training data 310a and 310b include the current Mel spectrogram, the generator 100 may generate a future Mel spectrogram that fits as a realistic continuation of the current Mel spectrogram.

[0046] On the other hand, a discriminative neural network 200, called the classifier 200, is trained to detect whether the generated data output by the generator 100 is a reconstructed (fake) version of the original (real) data. The classifier 200 can simply be considered a classifier that identifies whether the input signal is real or fake. The classifier 200 can be considered a learnable (nonlinear) loss function, as it replaces and / or complements the loss function specified for use during training.

[0047] The training process is summarized in Figure 5. In the first training mode 300b, the classifier 200 learns to classify whether the input comes from the training datasets 310a, 310b, or the generator 100. In the first training mode, the internal weights of the generator 100 are not changed (i.e., the generator 100 has not learned and is in a frozen state), while the internal weights of the classifier are updated to recognize that the output from the generator 100 is fake. In the second training mode 300c, the classifier 200 is frozen while the internal weights of the generator 100 are updated so that the output of the classifier 200 misclassifies the predicted output from the generator 100 as the actual signal. In other words, if the generator 100 is successful in the second training mode 300c, the generator 100 can reconstruct the signal to a sufficient degree to cause the classifier 200 to misclassify the input as the original (real) signal. The generator 100 and the discriminator 200 are trained together by alternating between the first training mode 300b and the second training mode 300c, and / or by using different training data.

[0048] Optionally, the classifier 200 can be trained in a third training mode 300a using training data 310a representing a complete audio signal. In the third training mode, the internal weights of the classifier 200 are updated, and the training data 310a representing a complete signal is classified as genuine.

[0049] The training data 310a and 310b may include at least one example of an audio signal containing speech. For example, the training data 310a and 310b may include variations or mixtures of publicly available datasets for speech synthesis, such as VCTK or LibriTTS. Both of these can be resampled at 16kHz, but the model may be adaptable to work at higher and lower sampling rates, for example, by adjusting the decoder stride. The training data 310a and 310b may include clean speech, but additional training data 310a and 310b may be obtained by extending the clean speech to introduce codec artifacts that can emulate artifacts that may be present in real-world communication scenarios. For example, for each utterance in the training data 310a and 310b, one of the following codecs may be randomly applied at a random bitrate from among the possible ones. -Opus:1.6, 3.2, 24, 32kbps -G722.1:24, 32kbps -AMRWB:6.6, 8.85, 12.65, 14.25, 15.85, 18.25, 19.85, 23.05, 23.85kbps

[0050] The codecs listed above are merely examples, and additional or other codecs can be used as alternatives. For example, codecs with possible bitrates of 6.4, 8, 9.6, 11.2, 12.8, 14.4, 16, 17.6, 19.2, 20.8, 22.4, 24, and 32 kbps can be used.

[0051] Furthermore, the training data 310a and 310b may be further enhanced by adding noise, reverberation, and other acoustic variations such as speaker count, accent, or language from other dataset sources.

[0052] The training data 310a and 310b may be enhanced by randomly replacing portions of the training data audio signal with alternative signal portions of random length. The portions replaced by alternative signal portions may correspond to audio signal portions of one or more packets and / or frames. For example, the training data 310a and 310b can be augmented by omitting one or more packets and / or frames from the packetization or frame representation of the training data audio signal and replacing each omitted packet and / or frame with an alternative signal portion of the corresponding length. Additionally or alternatively, two portions of the training data audio signal may be swapped, or the audio signal of a second training data audio signal may be added as an alternative audio signal portion of the first training data audio signal. In other words, the training data 310a and 310b may contain concatenations of two chunks belonging to different utterances, thus causing abrupt discrepancies in language. Therefore, the generator and discriminator may be trained with different losses that implement continuity of language content. Preferably, the two mismatched audio signal segments are actual signal segments such that the discriminator learns to detect incoherent content and the generator learns to produce reality (in signal quality) and coherence (in language).

[0053] The training data 310a and 310b can define past audio signals, where the generator 100 is trained to construct a plausible future continuation given past audio signals, so as to successfully cause the discriminator 200 to misclassify the constructed future signal as an actual audio signal. For example, when operating in a Mel spectrogram representation, the generator 100 needs to generate a future trajectory that looks like an actual Mel spectrogram and fits as a continuation of a past or present Mel spectrogram. Additionally or alternatively, the training data 310a and 310b may also be random noise samples extracted from a predetermined noise distribution (e.g., a Gaussian or uniform distribution), where the generator 100 is responsible for reconstructing alternative signal portions or constructing future signals given noise training data, so as to successfully cause the discriminator 200 to misclassify the reconstructed noise data.

[0054] Figure 6 shows an exemplary training setup for a causal adversarial autoencoder. The causal adversarial autoencoder includes an encoder neural network 110, optionally a generative model 130, and a decoder neural network 140. The generative model 130 may include a quantization block to operate on a quantized latent representation and may include at least one generative latent model (GLM) which is trained together with the rest of the causal adversarial autoencoder. During training, the causal adversarial autoencoder learns to reconstruct an incomplete audio signal 10, which includes an alternate signal portion 11, to form a reconstructed complete audio signal 20 which includes a reconstructed signal portion 21 that replaces the alternate signal portion 11. Ideally, the reconstructed audio signal 20 is similar and / or identical to the complete audio signal 1. These reconstructions can be learned through two losses acting on the reconstructed audio signal 20, namely one of a multi-STFT loss 400 and an adversarial (discriminator) loss 200, or a combination thereof. Multi-STFT minimizes the magnitude difference between the predicted reconstructed audio signal 20 and the original audio signal 1. This difference is preferably calculated with different time-frequency resolutions. A multi-STFT loss of 400 helps the model converge faster and achieve better quality than using only the adversarial loss obtained from the discriminator 200. However, a multi-STFT loss of 400 may not be sufficient to produce good quality speech because it lacks the purpose of phase reconstruction. Therefore, a model trained with multi-STFT alone may sound robotic and exhibit speaker identity and prosodic artifacts, even though the reconstructed signal may be perfectly understandable.

[0055] In some implementations, adversarial loss may be obtained for the reconstructed audio signal 20. Furthermore, training may be performed based on a combination of the multi-STFT400 loss and adversarial loss obtained by the classifier 200. Other regression losses, such as L1 loss, may be used as an alternative to or in addition to the adversarial loss and / or multi-STFT loss 400. Other deep feature losses may also be utilized, such as regression of features extracted by a deep neural speech encoder like PASE, or contrast loss that enriches contextual knowledge, such as contrast prediction coding.

[0056] For example, the classifier 200 may include multiple branches, the indicators of each branch being aggregated to form the final aggregated loss. The mean squared error is used to aggregate the loss and can provide a value of 0 if the input is fake and 1 if the input is real. In this case, the model is a least squares GAN formulation, and other formulations such as Hinge and Wasserstein can also be applied.

[0057] Once the causal adversarial autoencoder is trained, it can be used in inference mode with or without quantization and / or generative latent models. The alternative signal portion 11 is shown in inference mode in the same pattern as in training mode (e.g., by zeros covering the gap length). However, a different type of alternative signal portion pattern can be used by training the causal adversarial autoencoder with a different type of alternative for the signal portion 10.

[0058] Additional alternative indicators can be received along with the alternative signal portion 10 itself. These additional alternative signal indicators can be included as metadata to indicate that the alternative signal portion 10 is an actual alternative and not part of the complete audio signal 1. If a Mel spectrogram is used to represent the audio signal, the additional alternative indicators may be concatenated as additional frequency channels within the Mel spectrogram.

[0059] Using additional alternative directives, a causally adversarial autoencoder may choose to replace only the alternative signal portion with the reconstructed signal portion while preserving the rest of the incomplete audio signal. In this case, the causally adversarial autoencoder can perform a crossfading operation by applying a windowing function that smoothly transitions between the incomplete audio signal and the reconstructed signal portion of the incomplete audio signal to form the reconstructed audio signal 20. In some implementations, the crossfading operation is performed when future portions of the complete audio signal 1 are available, in which case zeros (or other alternative signal patterns) are added to the incomplete audio signal 20. Alternatively, the causally adversarial autoencoder may reconstruct several frames of the audio signal in a single inference step and buffer the frames until the next packet (containing one or more frames) is received. During this time, the windowing function can be slid across the buffered frames to save the number of interference steps per unit time of the audio signal.

[0060] Figure 7 shows a flowchart describing a method according to an embodiment of the present disclosure. In S10, an incomplete audio signal, or a representation of an incomplete audio signal, is received, and the incomplete audio signal includes an alternative signal portion that replaces the original signal portion. The incomplete audio signal (or its representation) is provided to an encoder neural network in S11, which outputs a reconstructed latent representation of the audio signal in S12. The reconstructed complete audio signal can be quantized in an optional step S13 to form a quantized representation of the reconstructed audio signal, which includes a token set. At least one quantized latent representation is provided to each GLM in S14, which predicts an updated (more likely) token set given at least one token of the quantized representation. In S15, the tokens of the quantized (original) representation are compared to the tokens predicted by the GLM in S14. If there is a difference, the most likely token is selected to form the final predicted token set. The final set of predicted tokens is input to the decoder neural network in S16, and in S17 the decoder neural network outputs the audio signal (waveform) or a representation of the reconstructed audio signal in the time domain. Optionally, post-processing follows S17, including performing inverse transform or crossfading filtering as described above.

[0061] Figure 8 shows a flowchart describing a method for training a causal autoencoder according to an embodiment of the present disclosure. In S20, a neural network system (e.g., an implementation of the causal autoencoder) is provided along with a discriminative neural network and training data. In S21, the training data is provided to the neural network system, and in S22, a reconstructed audio signal (or its representation) is output by the neural network system. For example, the neural network system may replace alternative signal portions of the training data signal with reconstructed signal portions, or predict the reconstructed future continuation of the training data audio signal. In S24a, the reconstructed audio signal is provided to a discriminator that outputs a first likelihood (indicator) in S25a indicating whether the reconstructed audio signal is a complete audio signal. Optionally, a second discriminator branch receives the reconstructed audio signal output in S22, downsamples the reconstructed audio signal (or its representation) in S23, and then provides the downsampled reconstructed audio signal (or its representation) to the second discriminator (discriminator branch) in S24b. In S25a, the second classifier outputs a second likelihood indicating whether the downsampled audio signal (or its representation) is the complete audio signal or the reconstructed audio signal. In S26, the likelihoods of each classifier branch are aggregated to form a final likelihood indicating whether the reconstructed audio signal is the complete audio signal or the reconstructed audio signal. In S27, the internal weights of either the classifier or the neural network system are updated according to adversarial generative training as described above. As can be understood, this method is repeated after S27, in the next pass, which includes the step of updating the internal weights of the other of the classifier and the neural network system, and / or providing and using new training data in the next pass.

[0062] The above describes possible methods for training and operating a deep learning-based system for determining the speech quality indication of input speech samples, and possible implementations of such a system. Furthermore, this disclosure also relates to apparatus for performing these methods. Examples of such apparatus may include a processor (e.g., a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), one or more application-specific integrated circuits (ASICs), one or more high-frequency integrated circuits (RFICs), or any combination thereof) and memory coupled to the processor. The processor can be adapted to perform some or all of the steps of the methods described throughout the disclosure.

[0063] This device may be a server computer, a client computer, a personal computer (PC), a tablet PC, a set-top box (STB), a personal digital assistant (PDA), a mobile phone, a smartphone, a web appliance, a network router, a switch or bridge, or any machine capable of executing instructions (sequential or otherwise) that specify the actions the device should take. Furthermore, this disclosure must relate to a set of devices that individually or collectively execute instructions that perform one or more of the methodologies discussed herein.

[0064] This disclosure further relates to a program (e.g., a computer program) that, when executed by a processor, includes instructions causing the processor to perform some or all of the steps of the method described herein.

[0065] Furthermore, this disclosure relates to a computer-readable (or machine-readable) storage medium for storing the aforementioned program. The term “computer-readable storage medium” includes, but is not limited to, data repositories in the form of, for example, solid memory, optical media, or magnetic media.

[0066] Unless otherwise specified, as will be apparent from the following discussion, any discussion throughout this disclosure using terms such as “processing,” “computing, calculating,” “determining,” and “analyzing” refers to the operation and / or processing of a computer or computing system, or similar electronic computing device that manipulates and / or converts data presented as physical, for example, electronic quantities into other data similarly presented as physical quantities.

[0067] Similarly, the term “processor” may refer to any device or part of a device that processes electronic data from, for example, registers and / or memory and converts said electronic data into other electronic data that can be stored, for example, in registers and / or memory. “Computer” or “computing device” or “computing platform” may include one or more processors.

[0068] In one exemplary embodiment, the methods described herein are executable by one or more processors that accept computer-readable (also called machine-readable) code, which includes an instruction set that, when executed by one or more of the processors, performs at least one of the methods described herein. This includes any processor capable of executing an instruction set (sequential or otherwise) that specifies the operation to be performed. Thus, an example is a standard processing system comprising one or more processors. Each processor may include one or more of the following: a CPU, an image processing unit, and a programmable DSP unit. The processing system may further include a memory subsystem, which includes main RAM and / or static RAM and / or ROM. A bus subsystem may be included for communication between components. The processing system may further be a distributed processing system having networked processors. If the processing system requires a display, such a display may include, for example, a liquid crystal display (LCD) or a cathode ray tube (CRT) display. If manual data entry is required, the processing system also includes an input device, such as one or more of the following: an alphanumeric input unit, such as a keyboard, a pointing control device, such as a mouse, etc. The processing system may also include a storage system such as a disk drive unit. In some configurations, the processing system may include an audio output device and a network interface device. The memory subsystem therefore includes a computer-readable carrying medium that carries computer-readable code (e.g., software) containing an instruction set that, when executed by one or more processors, causes one or more of the methods described herein to be performed. Note that when the method includes several elements, e.g., several steps, the order of these elements is not implied unless otherwise specified. The software may reside in a hard disk, or may reside entirely or partially in RAM and / or in a processor during execution by a computer system. Thus, the memory and processor also constitute a computer-readable medium that carries computer-readable code. Furthermore, the computer-readable carrying medium may form or be contained within a computer program product.

[0069] In alternative exemplary embodiments, one or more processors may operate as standalone devices or be connected, for example, to other processors in a networked deployment; one or more processors may operate as a server or user device in a server-user network environment, or as a peer device in a peer-to-peer or distributed network environment. One or more processors may form a personal computer (PC), tablet PC, personal digital assistant (PDA), mobile phone, web device, network router, switch or bridge, or any machine capable of executing a set of instructions (sequential or otherwise) that specify the actions to be taken by the machine.

[0070] The term “machine” should be understood to include any set of machines that individually or collectively perform a set (or set) of instructions for performing any one or more of the methods described herein.

[0071] Accordingly, each exemplary embodiment of the method described herein is in the form of a computer-readable carrying medium that carries an instruction set, a computer program intended for execution on one or more processors, for example, one or more processors which are part of a web server configuration. Accordingly, as will be understood by those skilled in the art, exemplary embodiments of the disclosure may be embodied as a method, equipment such as a dedicated device, equipment such as a data processing system, or a computer-readable carrying medium, for example, a computer program product. The computer-readable carrying medium carries computer-readable code which, when executed on one or more processors, causes the processor or the number of processors to implement the method. Accordingly, embodiments of the disclosure may take the form of a method, an exemplary embodiment entirely of hardware, an exemplary embodiment entirely of software, or an exemplary embodiment which combines embodiments of software and hardware. Furthermore, the disclosure may take the form of a carrying medium that carries computer-readable program code which is embodied in the medium (for example, a computer program product on a computer-readable storage medium).

[0072] The software may further be transmitted or received over a network by a network interface device. The carrying medium is a signaling medium in an exemplary embodiment, but the term “carrying medium” should be interpreted to include signaling mediums or multiple mediums (e.g., central or distributed databases, and / or associated caches and servers) that store one or more instruction sets. The term “carrying medium” should also be interpreted to include any medium capable of storing, encoding, or carrying instruction sets for execution by one or more processors, and causing one or more processors to execute any one or more of the methods of the Disclosure. The carrying medium may take many forms, but is not limited to non-volatile mediums, volatile mediums, and transmission mediums. Non-volatile mediums include, for example, optical, magnetic disks, or magneto-optical disks. Volatile mediums include dynamic memory such as main memory. Transmission mediums include coaxial cables, copper wires, and optical fibers, including wires that include bus subsystems. Transmission mediums may also take the form of acoustic or optical waves, such as those generated between radio waves and infrared data communications. For example, the term “carrying medium” should therefore be interpreted to include, but not limited to, solid memory, computer products embodied in optical and magnetic media, media that carry propagating signals representing instruction sets that, when detected and executed by at least one or more processors, perform a method, and transmission media in a network that carry propagating signals and represent instruction sets that, when executed, are detectable by at least one of the one or more processors.

[0073] The steps of the method discussed are understood to be performed in one exemplary embodiment by a suitable processor (or more processors) of a processing (e.g., computer) system that executes instructions (computer-readable code) stored in storage. It is also understood that this disclosure is not limited to any particular implementation or programming technique, and that this disclosure may be implemented using any suitable technique to perform the functions described in this specification. This disclosure is not limited to any particular programming language or operating system.

[0074] Throughout this disclosure, any reference to “one embodiment,” “several embodiments,” or “exemplary embodiments” means that any particular feature, structure, or characteristic described in relation to an exemplary embodiment is included in at least one exemplary embodiment of this disclosure. Therefore, occurrences of the phrases “in one exemplary embodiment,” “several exemplary embodiments,” or “exemplary embodiments” in various places throughout this disclosure do not necessarily all refer to the same exemplary embodiment. Furthermore, any particular feature, structure, or characteristic may be combined in any suitable manner as will be apparent to those skilled in the art from this disclosure in one or more exemplary embodiments.

[0075] When used in this specification, unless otherwise specified, the use of ordinal numbers such as "first," "second," "third," etc., to describe a common object merely indicates that different instances of similar objects are being referred to, and does not imply that the objects being described must be in a given sequence in time, space, rank, or any other way.

[0076] In the following claims and descriptions in this specification, any one of the terms: includes, has, constitutes, or constitutes is broad in meaning, including at least the elements / features that follow it, but not excluding others. Accordingly, the term: includes, when used in a claim, should not be construed as being limited to the means, elements, or steps listed thereafter. For example, the expression "the apparatus includes A and B" should not be limited to the apparatus being composed solely of elements A and B. The term: has, when used in this specification, is also broad in meaning, including at least the elements / features that follow it, but not excluding others. Accordingly, has is synonymous with including and means including.

[0077] It should be understood that in the above description of exemplary embodiments of the Disclosure, various features of the Disclosure are sometimes grouped together in a single exemplary embodiment, figure, or description for the purpose of streamlining the Disclosure and aiding in the understanding of one or more of the various aspects of the Invention. However, this method of the Disclosure should not be construed as reflecting an intention that the claims require more features than expressly described in each claim. Rather, as reflected in the following claims, the aspects of the Invention are fewer than all the features of a single, aforementioned exemplary embodiment; therefore, the claims following the description are expressly incorporated into this description, and each claim stands independently as a separate exemplary embodiment of the Disclosure.

[0078] Furthermore, while some exemplary embodiments described herein include some features but not others included in other exemplary embodiments, as will be understood by those skilled in the art, combinations of features from different exemplary embodiments are within the scope of the disclosure and constitute different exemplary embodiments. For example, any of the claimed exemplary embodiments may be used in any combination within the following claims.

[0079] Numerous specific details are described in the description provided in this specification. However, it is understood that exemplary embodiments of this disclosure may be carried out without these specific details. In other examples, well-known methods, structures, and techniques have not been shown so as not to obscure the understanding of the description of the invention.

[0080] Accordingly, while what is believed to be the best mode of this disclosure is described, those skilled in the art will understand that other and further modifications may be made without departing from the spirit of this disclosure, and that all such changes and modifications are intended to be within the scope of this disclosure. For example, any formulas described above are merely expressions of procedures that may be used. Functions may be added to or removed from block diagrams, and operations may be interchanged between function blocks. Steps may be added to or removed from methods described within the scope of this disclosure.

[0081] Various aspects of the present invention may be apparent from the enumerated example embodiments (EEE) listed below.

[0082] (EEE1) A deep learning generator for real-time packet loss concealment for lossy audio signals, wherein the generator comprises: An encoder stage comprising multiple encoder layers, wherein the first encoder layer of the multiple encoder layers is configured to feedforward connect to the second encoder layer of the multiple encoder layers, and the first encoder layer is further configured to skip connect to the summation block in the output of the encoder, The encoder stage is configured to receive a waveform audio signal as input, extract multiple features of the waveform audio signal, and output the features to a fixed representation via a context vector. A decoder stage comprising multiple decoder layers, wherein a first decoder layer of the multiple decoder layers is feedforward connected to a second decoder layer of the multiple decoder layers, and the first decoder layer is further configured to skip-connect to a summation block at the output of the decoder, The decoder stage is configured to convert the context vector into a sequence of N predicted output frames, A deep learning generator that includes this.

[0083] (EEE2) The aforementioned features include an EEE1 deep learning generator comprising at least one of audio content, prosody, and speaker identity.

[0084] (EEE3) A deep learning generator of EEE1, configured to operate autoregressively such that each frame of the sequence of N predicted output frames is fed back as input to the encoder stage at each of the future time steps of a plurality of future time steps.

[0085] (EEE4) A deep learning generator of EEE1, in which each of the aforementioned plurality of encoder layers is composed of a convolutional layer.

[0086] (EEE5) A deep learning generator for EEE4 in which the aforementioned convolutional layer is CausalConvBlock. (EEE6) A deep learning generator of EEE1, in which each layer of the aforementioned plurality of decoder layers consists of a DecoderBlock.

[0087] (EEE7) The aforementioned DecoderBlock is an EEE6 deep learning generator that includes a deep residual fully coupled structure.

[0088] (EEE8) The aforementioned structure is an EEE6 deep learning generator that includes normalization, activation functions, and fully connected layers.

[0089] (EEE9) A deep learning generator of EEE1, wherein a reconstructed signal is output from the decoder stage, and the reconstructed signal includes the N predicted output frames added together with the overlapping frames from the waveform audio signal input.

[0090] (EEE10) A deep learning system for real-time packet loss concealment of an audio signal, including the loss portion of the audio signal, wherein the system An encoder stage comprising a plurality of convolutional blocks, wherein the plurality of convolutional blocks are configured to extract one or more features of the waveform representation of the audio signal, and the one or more features are represented in latent space, A decoder stage comprising a plurality of transposed convolution and residual causal convolution layers, wherein the plurality of transposed convolution and residual causal convolution layers are configured to convert one or more features into waveform samples including waveform reconstruction of the loss portion of the audio signal, A deep learning system that includes [this].

[0091] (EEE11) A deep learning system of EEE1, wherein the encoder stage further includes a receive field of at least 600 milliseconds.

[0092] (EEE12) A deep learning system of EEE1 or 2, wherein the encoder stage further includes a multi-head vector quantization block configured to quantize the latent spatial representation of one or more features.

[0093] (EEE13) A method for real-time packet loss concealment of an audio signal, including the loss portion of the audio signal, wherein the method is A step of performing a transformation on the audio signal in order to determine the waveform representation of the audio signal, The steps include converting the waveform representation into the latent representation of the audio signal, A step of reconstructing the waveform representation of the audio signal from the latent representation of the audio signal, wherein the reconstructed waveform representation includes a learned reconstruction of the loss portion of the audio signal. The steps include performing an inverse transform on the reconstruction of the waveform representation to output the audio signal including the reconstructed loss portion of the audio signal, A method that includes this.

[0094] (EEE14) The EEE4 method further includes the step of quantizing the latent representation of the audio signal before reconstruction.

[0095] (EEE15) The EEE4 or 5 method, wherein the step of reconstructing the waveform representation of the audio signal includes the step of determining a learned reconstruction of the loss portion of the audio signal by conjugating a plurality of loss functions acting on the reconstruction of the waveform representation of the audio signal.

[0096] (EEE16) The EEE6 method, wherein the aforementioned multiple loss functions include multi-STFT loss and adversarial loss.

[0097] (EEE17) The transformation is a Short-Time Fourier Transform (STFT), as described in any of EEE4 to 7.

Claims

1. A computer-based method for concealing packet loss in a first audio signal, wherein the first audio signal includes a substitute signal portion that replaces the original portion of a second audio signal, and the method is performed by a computer. A step to obtain a representation of the first audio signal, The steps include predicting the latent representation of the second audio signal based on at least the representation of the first audio signal, A step of quantizing the predicted latent representation of the second audio signal, wherein the quantized latent representation is formed by selecting a set of tokens from a predetermined set of lexical tokens, The first neural network generates prediction tokens for the quantized latent representation, The steps include replacing the corresponding token in the quantized latent representation with the prediction token, Given the quantized latent representation of the second audio signal, the second neural network predicts a representation of the second audio signal, wherein the predicted representation includes a reconstruction of the original portion of the second audio signal. Includes, A method wherein the first audio signal includes an incomplete audio signal, and the second audio signal includes a complete audio signal.

2. The method according to claim 1, wherein the encoder neural network is configured to predict the latent representation of the second audio signal based on at least the representation of the first audio signal.

3. The quantized latent representation is a first quantized latent representation, and the method is A step of obtaining a second quantized latent representation by quantizing the latent representation of the second audio signal, wherein the second quantized latent representation is formed by selecting a second set of tokens from a second predetermined set of lexical tokens. The steps include inputting the first and second quantized latent representations into the second neural network, The method according to any one of claims 1 to 2, further comprising:

4. Each token is represented by a quantization vector, The method according to claim 3, wherein the step of inputting the first and second quantized latent representations into the second neural network includes the step of concatenating or adding the quantization vector of the first quantized latent representation with the corresponding quantization vector of the second quantized latent representation.

5. A step of providing the second quantized latent representation to a second generative neural network, wherein the second generative neural network is trained to predict a different token from the second token set when at least one token from the second token set is provided. The second generative neural network outputs a prediction token of the second quantized latent representation and a confidence metric associated with the prediction token. The steps include replacing the corresponding tokens of the second quantized latent representation with the predicted tokens of the second generative neural network based on the confidence level of the predicted tokens of the second quantized latent representation, The method according to claim 3 or 4, further comprising:

6. The steps include obtaining at least one random variable, A step of inputting the at least one random variable into the second neural network, wherein the second neural network is trained to predict a representation of the second audio signal given the predicted latent representation of the second audio signal and the at least one random variable, The method according to any one of claims 1 to 5, further comprising:

7. The method according to any one of claims 1 to 6, further comprising the step of converting the first audio signal into a representation of the first audio signal.

8. The method according to any one of claims 1 to 7, further comprising the step of outputting a predicted representation of the second audio signal by the second neural network.

9. The method according to any one of claims 1 to 8, further comprising the step of inversely converting the predicted representation of the second audio signal into a time-domain audio signal.

10. The method according to any one of claims 1 to 9, wherein the representation is one of a frequency representation, a time representation, a filter bank representation, and / or a feature domain representation.

11. The method according to claim 10, wherein the expression is a Mel-frequency cepstrum coefficient or a short-time Fourier transform.

12. The second neural network includes at least one causal convolutional layer and at least one subsequent residual layer, and the method is The steps include: performing causal upsampling using the aforementioned causal convolutional layer to form an upsampled representation; The steps include transforming the upsampled representation using at least one residual layer, The method according to any one of claims 1 to 11, further comprising:

13. The steps include determining a window function based on the reconstructed original portion of the predicted second audio signal and the immediate subsequent portion of the predicted second audio signal, The steps include applying the determined window function to the reconstructed original portion of the predicted second audio signal, The method according to any one of claims 1 to 12, further comprising:

14. The step of determining the window function is: The method according to claim 13, comprising the step of inputting a representation of the reconstructed original portion of the predicted second audio signal and the direct subsequent portion of the predicted second audio signal into a neural network, wherein the neural network is trained to predict a window function given the reconstructed audio signal portion and the subsequent portion of the audio signal portion.

15. A computer-implemented neural network system for packet loss concealment of a first audio signal, wherein the first audio signal includes an alternative signal portion that replaces the original portion of a second audio signal, and the system An input unit configured to acquire a representation of the first audio signal, An encoder neural network trained to predict the latent representation of a second audio signal based on the representation of the first audio signal, A quantization block configured to quantize the latent representation of the second audio signal and obtain a quantized latent representation, wherein the quantized latent representation is formed by selecting a set of tokens from a predetermined lexical token set, and the quantization block A generative neural network trained to generate prediction tokens for the quantized latent representation and replace corresponding tokens in the quantized latent representation with the prediction tokens, Given a quantized latent representation of the second audio signal, a decoder neural network trained to predict a representation of the second audio signal, wherein the predicted representation of the second audio signal includes a reconstruction of the original portion of the second audio signal, Includes, A neural network system in which the first audio signal includes an incomplete audio signal and the second audio signal includes a complete audio signal.

16. The neural network system according to claim 15, wherein the encoder neural network and the decoder neural network are trained with adversarial neural networks.

Images