Audio coding and decoding based on artificial intelligence

Abstract

Techniques for encoding and decoding an audio signal are described. For example, using a neural network, a residual signal of samples of the audio signal is generated based on an input of the neural network. The residual signal is configured to excite the long-term prediction filter and / or the short-term prediction filter. Samples of the reconstructed audio signal are determined using a long-term prediction filter and / or a short-term prediction filter. Samples of a reconstructed audio signal are determined based on a residual signal generated for the samples of the audio signal using a neural network.

Description

[0001] This application is a divisional application of the following invention patent application: Application number: 202080061632.2, Application date: June 29, 2020, Invention name: Audio codec based on artificial intelligence. Technical Field

[0002] This application is related to speech coding and decoding. For example, a system and method for performing artificial intelligence-based speech coding and decoding are described. Background Art

[0003] A speech codec (also called an audio codec and / or a voice codec) is a technique for representing a digitized audio signal using as few bits as possible (thus compressing the voice data) while attempting to maintain a certain level of audio quality. A speech coder is used to encode (or compress) a digitized audio (e.g., speech, music, etc.) signal into a lower bit rate data stream. The lower bit rate data stream can be input to a speech decoder, which decodes the data stream and constructs an approximation or reconstruction of the original signal. A speech encoder-decoder structure may be referred to as a speech codec (or speech codec or audio codec) or a speech / speech / audio encoder-decoder (codec).

[0004] Speech codecs exploit the fact that speech signals are highly correlated waveforms. Some speech codecs are based on the source-filter model of speech production, which assumes that the vocal cords are the source of spectrally flat sound (excitation signal), while the vocal tract acts as a filter, spectrally shaping the various sounds of speech. Different phonemes, such as vowels, fricatives, and speech fricatives, can be distinguished by their excitation (source) and spectral shape (filter). Summary of the invention

[0005] This article describes techniques and systems for providing an artificial intelligence-based audio codec (e.g., a speech codec, a music codec, or other types of audio codecs). In some examples, the audio decoder can utilize a neural network model to decode certain parts of an audio signal. Examples of audio signals include speech or voice signals, music signals, and / or other sound signals. Although some of the examples described herein use speech or sound signals as examples, the techniques described herein can be used for any type of audio signal.

[0006] In some embodiments, the speech codec may include a code-excited linear prediction (CELP) codec or an algebraic-CELP (ACELP) codec. The CELP model is based on a source filter model of speech production, which uses a linear prediction (LP) model to model the vocal tract and takes entries of a fixed codebook (FCB) as input to the LP model. For example, the pitch of a speech signal can be modeled using long-term linear prediction, and the spectral shape (phonemes) of the speech signal can be modeled using short-term linear prediction. The entries in the FCB are based on the encoding and decoding of the residual signal remaining after performing long-term and short-term linear prediction modeling. The most bit-hungry stage in a CELP-based codec is the FCB, which is based on brute-force encoding and decoding of the residual signal without attempting to model the signal.

[0007] The techniques and systems described herein include replacing the FCB of a traditional speech decoder with an artificial intelligence model (e.g., a neural network model). For example, the neural network model can be trained to act as a model of a residual signal that includes speech components not captured by the long-term and short-term linear prediction models. In such an example, the output of the neural network model includes a residual signal that can be used to excite the long-term linear prediction model or the short-term linear prediction model. Using the residual signal, the speech signal can be reconstructed by the long-term and short-term linear prediction models.

[0008] In some cases, by modeling the residual signal using a neural network in a speech decoder, a speech codec suitable for applications and / or devices with low bit rate, complexity and / or memory constraints is provided. For example, the neural network model requires significantly fewer bits for residual coding than the FCB in a conventional speech codec.

[0009] This article also provides other techniques for improving speech encoding and decoding.

[0010] According to at least one example, there is provided an apparatus for reconstructing one or more audio signals, comprising: a memory configured to store audio data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors being configured to: generate (902) a first residual signal of at least one sample of the audio signal using a neural network, the residual signal being configured to excite at least a first filter; determine, using the first filter and based on the first residual signal generated for at least one sample of the audio signal using the neural network, a second residual signal for input to a second filter, wherein the first filter is a long-term prediction filter and the second filter is a short-term prediction filter, or wherein the first filter is a short-term prediction filter and the second filter is a long-term prediction filter; and determine (904) at least one sample of the reconstructed audio signal using the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the second residual signal.

[0011] According to at least one example, a method for reconstructing one or more audio signals is provided, the method comprising: generating (902) a first residual signal of at least one sample of an audio signal using a neural network, the residual signal being configured to excite at least a first filter; determining, using the first filter and based on a first residual signal generated for at least one sample of the audio signal using the neural network, a second residual signal for input to a second filter, wherein the first filter is a long-term prediction filter and the second filter is a short-term prediction filter, or wherein the first filter is a short-term prediction filter and the second filter is a long-term prediction filter; and determining (904) at least one sample of a reconstructed audio signal using the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the second residual signal.

[0012] According to at least one example, a method for reconstructing one or more audio signals is provided. The method includes using a neural network to generate a residual signal for at least one sample of the audio signal based on one or more inputs to the neural network. The residual signal is configured to excite at least one of a long-term prediction filter and a short-term prediction filter. The method also includes using at least one of the long-term prediction filter and the short-term prediction filter to determine at least one sample of the reconstructed audio signal. Determine at least one sample of the reconstructed audio signal based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0013] In another example, an apparatus for reconstructing one or more audio signals is provided, the apparatus comprising a memory configured to store audio data, and one or more processors implemented in a circuit and coupled to the memory. In some examples, the one or more processors are configured to generate a residual signal for at least one sample of the audio signal based on one or more inputs to the neural network using a neural network. The residual signal is configured to excite at least one of a long-term prediction filter and a short-term prediction filter. The one or more processors are also configured to determine at least one sample of the reconstructed audio signal using at least one of the long-term prediction filter and the short-term prediction filter. The at least one sample of the reconstructed audio signal is determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0014] In another example, a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device for reconstructing one or more audio signals to: generate, using a neural network, a residual signal for at least one sample of the audio signal based on one or more inputs to the neural network, the residual signal configured to excite at least one of a long-term prediction filter and a short-term prediction filter; and determine, using at least one of the long-term prediction filter and the short-term prediction filter, at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0015] In another example, an apparatus for reconstructing one or more audio signals is provided. The apparatus includes components for generating a residual signal for at least one sample of the audio signal using a neural network based on one or more inputs to the neural network. The residual signal is configured to excite at least one of a long-term prediction filter and a short-term prediction filter. The apparatus also includes components for determining at least one sample of the reconstructed audio signal using at least one of the long-term prediction filter and the short-term prediction filter. The at least one sample of the reconstructed audio signal is determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0016] In certain aspects, the long term prediction filter is a long term linear prediction (LTP) filter. In some cases, the residual signal comprises a long term linear prediction (LTP) residual configured to excite the LTP filter.

[0017] In some examples, the above methods, apparatus, and computer-readable media may include determining a linear prediction (LP) residual of at least one sample of an audio signal using an LTP filter, the LP residual of at least one sample of the audio signal being determined based on an LTP residual generated for at least one sample of the audio signal using a neural network and an LTP prediction of at least one sample of the audio signal determined using a gain and an LP residual of at least one previous sample of the audio signal. In some aspects, the gain is applied to the LP residual of at least one previous sample of the audio signal.

[0018] In some aspects, the short-term prediction filter is a short-term linear prediction (LP) filter. In some examples, the above methods, apparatus, and computer-readable media may include: determining at least one sample of the reconstructed audio signal using the short-term LP filter, the at least one sample of the reconstructed audio signal being determined based on the LP prediction and the LP residual determined for at least one sample of the audio signal. In some examples, one or more inputs to the neural network include at least one of an LTP prediction for at least one sample of the audio signal, an LP prediction for at least one sample of the audio signal, a previous LTP residual determined by the neural network for at least one previous sample of the audio signal, and at least one previous sample of the reconstructed audio signal.

[0019] In some examples, an LTP residual generated for at least one sample of the audio signal using a neural network is combined with an LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal.

[0020] In some examples, the LP residual determined for at least one sample of the audio signal is combined with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal.

[0021] In some aspects, the short-term prediction filter is a short-term linear prediction (LP) filter. In some examples, the residual signal includes a linear prediction (LP) residual configured to excite the short-term LP filter. In some examples, the above methods, apparatus, and computer-readable media may include: determining a long-term linear prediction (LTP) residual of at least one sample of the audio signal using the short-term LP filter, the LTP residual of the at least one sample being determined based on the LP prediction of the at least one sample of the audio signal and the LP residual generated using a neural network.

[0022] In some examples, the long term prediction filter is a long term linear prediction (LTP) filter. In some examples, the above methods, apparatus, and computer readable media may include: using the LTP filter to determine at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on an LTP residual of at least one sample of the audio signal and an LTP prediction of at least one sample of the audio signal determined using a gain and at least one previous sample of the reconstructed audio signal.

[0023] In some aspects, the at least one sample of the audio signal comprises a single sample.

[0024] In some aspects, at least one sample of the audio signal comprises a plurality of samples. In some examples, the plurality of samples are included in a frame of the audio signal.

[0025] In some aspects, the audio signal comprises a speech signal, and reconstructing the audio signal comprises reconstructing the speech signal. In some examples, the audio signal comprises a music signal, and reconstructing the audio signal comprises reconstructing the music signal.

[0026] The above-described aspects relating to any of the methods, apparatus, and computer-readable media may be used alone or in any suitable combination.

[0027] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used alone to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings and each claim.

[0028] The above and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Examples of various implementations are described in detail below with reference to the following drawings:

[0030] Figure 1 is a block diagram illustrating an example of a speech codec system according to some examples;

[0031] Figure 2 is a block diagram illustrating an example of a CELP-based speech codec system utilizing a fixed codebook (FCB) according to some examples;

[0032] Figure 3 is a block diagram illustrating an example of a speech codec system utilizing a neural network model according to some examples;

[0033] Figure 4 is a block diagram illustrating an example of a neural network model according to some examples;

[0034] Figure 5A is a block diagram illustrating another example of a speech encoding and decoding system utilizing a neural network model according to some examples;

[0035] Figure 5B is a block diagram illustrating another example of a speech encoding and decoding system utilizing a neural network model according to some examples;

[0036] Figure 6 is a block diagram illustrating another example of a speech encoding and decoding system utilizing a neural network model according to some examples;

[0037] Figure 7 is a block diagram illustrating another example of a speech encoding and decoding system utilizing a neural network model according to some examples;

[0038] Figure 8 is a block diagram illustrating an example of a neural network-based speech codec system operating on a frame-by-frame basis according to some examples;

[0039] Fig. 9 is a flow chart illustrating an example of a process for reconstructing one or more audio signals according to some examples;

[0040] Fig.10 is a flowchart illustrating an example of a process for reconstructing one or more audio signals according to some examples; and

[0041] Fig.11 is an example computing device architecture of an example computing device that may implement the various techniques described herein. DETAILED DESCRIPTION

[0042] Certain aspects and embodiments of the present disclosure are provided below. Some of these aspects and embodiments can be applied independently, and some of them can be applied in combination, as will be apparent to those skilled in the art. In the following description, for the purpose of explanation, specific details are set forth to provide a thorough understanding of the embodiments of the present application. However, it is apparent that various embodiments can be implemented without these specific details. The drawings and description are not intended to be limiting.

[0043] The following description provides only example embodiments and is not intended to limit the scope, applicability or configuration of the present disclosure. Instead, the following description of the exemplary embodiments will provide an enabling description for implementing the exemplary embodiments for those skilled in the art. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the spirit and scope of the application as described in the appended claims.

[0044] As described above, this article describes techniques and systems for providing an audio codec based on artificial intelligence. In some examples, the audio decoder can utilize a neural network model to decode certain parts of the audio signal. Although some examples described herein use voice signals or speech signals as examples, the techniques described herein can be used for any type of audio signal, such as music signals and / or other sound signals.

[0045] An audio codec (eg, a voice codec, a music signal codec, or other types of audio codecs) may be performed on a digitized audio signal (eg, a speech signal) to compress the amount of data for storage, transmission, and / or other uses. Figure 1 1 is a block diagram illustrating an example of a speech codec system 100 (also referred to as a speech or voice codec or speech coder-decoder (codec)). A speech encoder 102 of the speech codec system 100 may process a speech signal 101 using a speech codec algorithm. The speech signal 101 may include a digitized speech signal generated from an analog speech signal from a given source. For example, a digitized speech signal may be generated using a filter to eliminate aliasing, a sampler to convert to discrete time, and an analog-to-digital converter to convert the analog signal to the digital domain. The resulting digitized speech signal (e.g., speech signal 101) is a discrete-time speech signal having sample values ​​(referred to herein as samples) that are also discretized.

[0046] Using a speech codec algorithm, the speech encoder 102 can generate a compressed signal (including a lower bit rate data stream) that represents the voice signal 101 using as few bits as possible while attempting to maintain a certain quality level of the voice. The speech encoder 102 can use any suitable speech codec algorithm, such as a linear prediction codec algorithm (e.g., code excited linear prediction (CELP), algebraic-CELP (ACELP) or other linear prediction techniques) or other speech codec algorithms.

[0047] The speech encoder 102 attempts to reduce the bit rate of the speech signal 101. The bit rate of the signal is based on the sampling frequency and the number of bits per sample. For example, the bit rate of the speech signal can be determined as follows:

[0048]

[0049] Where BR is the bit rate, S is the sampling frequency, and b is the number of bits per sample. In one illustrative example, at a sampling frequency (S) of 8 kilohertz (kHZ) and 16 bits per sample (b), the bit rate of the signal would be a bit rate of 128 kilobits per second (kbps).

[0050] The compressed speech signal may then be sent to and processed by the speech decoder 104. In some examples, the speech decoder 104 may communicate with the speech encoder 102 to, for example, request speech data, send feedback information, and / or provide other communications to the speech encoder 102. In some examples, the speech encoder 102 or a channel encoder may perform channel coding on the compressed speech signal before the compressed speech signal is sent to the speech decoder 104. For example, the channel coding may provide error protection to the bit stream of the compressed speech signal to protect the bit stream from noise and / or interference that may occur during transmission over a communication channel.

[0051] The speech decoder 104 decodes the data of the compressed speech signal and constructs a reconstructed speech signal 105 that approximates the original speech signal 101. The reconstructed speech signal 105 includes a digitized discrete-time signal that may have the same bit rate as the original speech signal 101. The speech decoder 104 may use the inverse of the speech coding algorithm used by the speech encoder 102, which, as described above, may include any suitable speech codec algorithm, such as a linear prediction codec algorithm (e.g., CELP, ACELP, or other suitable linear prediction techniques) or other speech codec algorithms. In some cases, the reconstructed speech signal 105 may be converted to a continuous-time analog signal, such as by performing digital-to-analog conversion and anti-aliasing filtering.

[0052] Speech codecs can take advantage of the fact that speech signals are highly correlated waveforms. Samples of the input speech signal can be divided into blocks of N samples each, where a block of N samples is called a frame. In an illustrative example, each frame can be 10-20 milliseconds (ms) long.

[0053] Code Excited Linear Prediction (CELP) is an example of a speech codec algorithm. The CELP model is based on the source-filter model of speech production, which assumes that the vocal cords are the source of spectrally flat sound (excitation signal), while the vocal tract acts as a filter, spectrally shaping the various sounds of speech. Different phonemes (such as vowels, fricatives, and speech fricatives) can be distinguished by their excitation (source) and spectral shape (filter).

[0054] Typically, CELP uses a linear prediction (LP) model to model the vocal tract and uses entries of a fixed codebook (FCB) as input to the LP model. For example, long-term linear prediction can be used to model the pitch of a speech signal, and short-term linear prediction can be used to model the spectral shape (phoneme) of a speech signal. The entries in the FCB are based on the encoding and decoding of the residual signal remaining after performing long-term and short-term linear prediction modeling. For example, long-term linear prediction and short-term linear prediction models can be used for speech synthesis, and a fixed codebook (FCB) can be searched during encoding to locate the best residual for input to the long-term and short-term linear prediction models. The FCB provides residual speech components that are not captured by the short-term and long-term linear prediction models. The residual and the corresponding index can be selected based on a comprehensive analysis process at the encoder, which is performed to select the best parameters to match the original speech signal as closely as possible. The index can be sent to the decoder, and the decoder can extract the corresponding LTP residual from the FCB based on the index.

[0055] Figure 2 2 is a block diagram showing an example of a CELP-based speech coding and decoding system 200 including a speech encoder 202 and a speech decoder 204. The speech encoder 202 can obtain a speech signal 201 and can segment samples of the speech signal into frames and subframes. For example, a frame of N samples can be divided into subframes. In an illustrative example, a frame of 240 samples can be divided into four subframes, each subframe having 60 samples. For each frame, subframe or sample, the speech encoder 202 selects parameters (e.g., gain, filter coefficients or linear prediction (LP) coefficients, etc.) for the synthesized speech signal so that the synthesized speech signal matches the original speech signal as much as possible.

[0056] The speech encoder 202 may include a short-term linear prediction (LP) engine 210, a long-term linear prediction (LTP) engine 212, and a fixed codebook (FCB) 214. The short-term LP engine 210 models the spectral shape (phonemes) of the speech signal. For example, the short-term LP engine 210 may perform a short-term LP analysis on each frame to generate linear prediction (LP) coefficients. In some examples, the input to the short-term LP engine 210 may be an original speech signal or a preprocessed version of the original speech signal. In some embodiments, the short-term LP engine 210 may perform linear prediction on each frame by estimating the value of the current speech sample based on a linear combination of past speech samples. For example, the speech signal s(n) may be modeled using an autoregressive (AR) model (e.g., ), where each sample is represented as the previous m samples plus a prediction error term The linear combination of . Weighting coefficient a 1 、a 2 to a mis called the LP coefficient. The prediction error term can be found as : The short-term LP engine 210 can obtain the LP coefficients by minimizing the mean square prediction error with respect to the filter coefficients. The LP coefficients can be used to form the analysis filter:

[0057]

[0058]

[0059] The short-term LP engine 210 can calculate the above AR model equation ( ) or other error metrics to minimize the LP coefficients ( ) to solve (may be referred to as a transfer function). In some implementations, the LP coefficients may be determined using a Levinson-Durbin method, a Leroux-Gueguen algorithm, or other suitable techniques. In some examples, the speech encoder 202 may send the LP coefficients to the speech decoder 204. In some examples, the speech decoder 204 may determine the LP coefficients, in which case the speech encoder 202 may not send the LP coefficients to the speech decoder 204. In some examples, line spectral frequencies (LSFs) may be calculated instead of or in addition to the LP coefficients.

[0060] The LTP engine 212 models the pitch of the speech signal. Pitch is a characteristic that determines the spacing or period of pulses in a speech signal. For example, a speech signal is produced when airflow from the lungs is periodically interrupted by the movement of the vocal cords. The time between consecutive vocal cord openings corresponds to the pitch period. After the short-term LP engine 210 is applied to a frame, the LTP engine 212 can be applied to each frame or each subframe of the frame. The LTP engine 212 can predict the current signal sample from past samples that are one or more pitch periods (hence the term "long-term") away from the current sample. For example, the current signal sample can be predicted as , where T represents the pitch period, represents the tone gain, and Represents the LP residual of the previous sample that is one or more pitch periods away from the current sample. The pitch period can be estimated at each frame. By comparing the frame with past samples, the period at which the signal repeats itself can be identified, thereby estimating the actual pitch period. The LTP engine 212 can be applied to each subframe separately.

[0061] The FCB 214 may include a number of long-term linear prediction (LTP) residuals (denoted as L). The LTP residuals include speech signal components remaining after performing long-term and short-term linear prediction modeling. The LTP residuals may be, for example, fixed or adaptive, and may contain deterministic pulses or random noise (e.g., white noise samples). For each segment of the input speech signal (e.g., each frame or other sample group), the speech encoder 202 may pass through the number of LTP residuals L in the FCB 214 multiple times, and an error value (e.g., a mean square error value) may be calculated after each pass. The LTP residuals may be represented using a code vector. The length of each code vector may be equal to the length of each subframe, in which case a search of the FCB 214 is performed once per subframe. The speech encoder 202 may select the LTP residual that provides the lowest error. The speech encoder 202 may select an index corresponding to the LTP residual selected from the FCB 214 for a given subframe or frame. The speech encoder 202 may send an index indicating which LTP residual is selected from the FCB 214 for a given subframe or frame to the speech decoder 204. The gain associated with the lowest error may also be selected and sent to the speech decoder 204 .

[0062] The speech decoder 204 includes a FCB 224, an LTP engine 222, and a short-term LP engine 220. The FCB 224 has the same LTP residual (e.g., code vector) as the FCB 214. The speech decoder 204 can extract the LTP residual from the FCB 224 using the index sent from the speech encoder 202 to the speech decoder 204. The extracted LTP residual can be scaled to an appropriate level and filtered by the LTP engine 222 and the short-term LP engine 220 to generate a reconstructed speech signal 205. The LTP engine 222 generates periodicity in the signal associated with the fundamental pitch frequency, and the short-term LP engine 220 generates the spectral envelope of the signal.

[0063] Other linear prediction-based codec systems can also be used to encode and decode voice signals, including Enhanced Voice Service (EVS), Adaptive Multi-Rate (AMR) speech codec system, Mixed Excitation Linear Prediction (MELP) speech codec system, Linear Prediction Codec-10 (LPC-10), etc.

[0064] There is a need for a speech codec for certain applications and / or devices (e.g., Internet of Things (IoT) applications and devices) that provides higher quality speech signal encoding and decoding at low bit rate, low complexity, and low memory requirements. There is currently no known speech codec that meets all requirements simultaneously. Existing linear prediction based codecs cannot meet such requirements. For example, ACELP based codec systems provide high quality but do not provide low bit rate or low complexity / low memory. Other linear prediction codec systems provide low bit rate and low complexity / low memory but do not provide high quality.

[0065] For CELP-based codec systems, the most bit-hungry stage is the FCB. The entries in the FCB are based on the encoding and decoding of the residual signal remaining after performing long-term and short-term linear prediction modeling. For example, algebraic pulse coding is performed on the residual signal remaining after performing short-term and long-term linear prediction without attempting to model the residual signal. The residual signal is a high-entropy signal, and strong, model-free quantization of high-entropy signals leads to a very bit-hungry FCB stage.

[0066] This article describes systems, devices, methods (also referred to as processes), and computer-readable media for replacing the FCB of a conventional speech decoder with an artificial intelligence model such as a neural network model. Other functional blocks of the speech decoder (e.g., a long-term linear prediction filter or engine and a short-term linear prediction filter or engine) can be based on conventional methods (neural network modeling of long-term and short-term linear prediction filters will not be performed). For example, the FCB of a conventional speech decoder can be replaced with an artificial intelligence model (e.g., a neural network model), while other components of the conventional speech decoder (e.g., a short-term linear prediction engine and a long-term linear prediction engine) are not replaced by a neural network. The advantages provided by the speech codec system described herein over existing speech codecs make such a system very suitable for various types of applications, such as mobile (smart phones, etc.), retail (point of service (POS), etc.), security, smart cities / buildings, wearable devices, industrial IoT, general IoT space applications, etc.

[0067] Figure 33 is a diagram showing an example of a speech codec system 300 using a neural network model. The speech codec system 300 includes a speech encoder 302 and a speech decoder 304. The speech encoder 302 may be similar to and may perform some or all of the functions of the speech encoder 202 described above. For example, the speech encoder 302 may include a short-term LP engine, an LTP engine, and an FCB (not shown). The speech encoder 302 may extract (and in some cases quantize) a feature set (referred to as a feature set) from the speech signal 301, and may send the extracted (and in some cases quantized) feature set to the speech decoder 304. The features calculated by the speech encoder 302 may depend on the specific encoder implementation used. Various illustrative examples of feature sets are provided below according to different encoder implementations, which feature sets may be extracted (and in some cases quantized) by the speech encoder 302 and sent to the speech decoder 304. However, it will be understood by a person of ordinary skill that other feature sets may be extracted by the speech encoder 302. For example, the speech encoder 302 may extract any feature set, may quantize the feature set, and may send the feature set to the speech decoder 304.

[0068] As described above, various combinations of features may be extracted as a feature set by the speech encoder 302. For example, the feature set may include one or any combination of the following features: linear prediction (LP) coefficients; line spectral pairs (LSPs); line spectral frequencies (LSFs); pitch lags with integer or fractional precision; pitch gains; pitch correlations; Mel-scale frequency cepstral coefficients (also known as Mel cepstrum) of the speech signal; Bark-scale frequency cepstral coefficients (also known as bark cepstrum) of the speech signal; Mel-scale frequency cepstral coefficients of the LTP residual; Bark-scale frequency cepstral coefficients of the LTP residual; a spectrum of the speech signal (e.g., a discrete Fourier transform (DFT) or other spectrum); and / or a spectrum of the LTP residual (e.g., a DFT or other spectrum).

[0069] For any one or more of the other features listed above, the speech encoder 302 may use any estimation and / or quantization method, such as an engine or algorithm from any suitable speech codec (e.g., EVS, AMR, or other speech codec), or a neural network-based estimation and / or quantization scheme (e.g., convolutional or fully connected (dense) or recursive autoencoder, or other neural network-based estimation and / or quantization scheme). The speech encoder 302 may also use any frame size, frame overlap, and / or update rate for each feature. For purposes of illustration, examples of estimation and quantization methods for each example feature are provided below, where other examples of estimation and quantization methods may be used by the speech encoder 302.

[0070] As described above, one example of features that can be extracted from the speech signal by the speech encoder 302 includes LP coefficients and / or LSF. Various estimation techniques can be used to calculate the LP coefficients and / or LSF. For example, as described above with respect to Figure 2 As described, the speech encoder 302 can use the Levinson-Durbin algorithm to estimate the LP coefficients (and / or LSF) from the speech signal. In some examples, the autocovariance method for LP estimation can be used to estimate the LP coefficients and / or LSF. In some cases, the LP coefficients can be determined, and the LP to LSF conversion algorithm can be performed to obtain the LSF. Any other LP and / or LSF estimation engine or algorithm can be used, such as an LP and / or LSF estimation engine or algorithm from an existing codec (e.g., EVS, AMR or other speech codec).

[0071] Various quantization techniques can be used to quantize the LP coefficients and / or LSF. For example, the speech encoder 302 can use a single-stage vector quantization (SSVQ) technique, a multi-stage vector quantization (MSVQ) or other vector quantization techniques to quantize the LP coefficients and / or LSF. In some cases, predictive or adaptive SSVQ or MSVQ (or other vector quantization techniques) can be used to quantize the LP coefficients and / or LSF. In another example, the speech encoder 302 can use an autoencoder or other neural network-based techniques to quantize the LP coefficients and / or LSF. Any other LP and / or LSF quantization engine or algorithm can be used, such as an LP and / or LSF quantization engine or algorithm from an existing codec (e.g., EVS, AMR or other speech codec).

[0072] Another example of features that can be extracted from the voice signal by the speech encoder 302 includes pitch lag (integer and / or fraction), pitch gain and / or pitch correlation. Various estimation techniques can be used to calculate pitch delay, pitch gain and / or pitch correlation. For example, the speech encoder 302 can use any pitch lag, gain, correlation estimation engine or algorithm (e.g., pitch lag estimation based on autocorrelation) to estimate pitch lag, pitch gain and / or pitch correlation (or any combination thereof) from the voice signal. For example, the speech encoder 302 can use a pitch delay, gain and / or correlation estimation engine (or algorithm) from any suitable voice codec (e.g., EVS, AMR or other voice codec). Various quantization techniques can be used to quantize pitch lag, pitch gain and / or pitch correlation. For example, the speech encoder 302 can use any pitch delay, gain, correlation quantization engine or algorithm from any suitable voice codec (e.g., EVS, AMR or other voice codec) to quantize pitch delay, pitch gain and / or pitch correlation (or any combination thereof) from the voice signal. In some cases, speech encoder 302 may use an autoencoder or other neural network based techniques to quantify pitch lag, pitch gain, and / or pitch correlation features.

[0073] Another example of features that can be extracted from the speech signal by the speech encoder 302 includes Mel cepstral coefficients and / or Bark cepstral coefficients of the speech signal, and / or Mel cepstral coefficients and / or Bark cepstral coefficients of the LTP residual. Various estimation techniques can be used to calculate the Mel cepstral coefficients and / or Bark cepstral coefficients. For example, the speech encoder 302 can use a Mel or Bark frequency cepstral technique including Mel or Bark frequency filter bank calculation, filter bank energy calculation, logarithmic application, and discrete cosine transform (DCT) or DCT truncation. Various quantization techniques can be used to quantize the Mel cepstral coefficients and / or Bark cepstral coefficients. For example, vector quantization (single stage or multi-stage) or predictive / adaptive vector quantization can be used. In some cases, the speech encoder 302 can use an autoencoder or other neural network-based techniques to quantize the Mel cepstral coefficients and / or Bark cepstral coefficients. Any other suitable cepstral quantization method can be used.

[0074] Another example of features that can be extracted from the speech signal by the speech encoder 302 includes the spectrum of the speech signal and / or the spectrum of the LTP residual. Various estimation techniques can be used to calculate the spectrum of the speech signal and / or the LTP residual. For example, a discrete Fourier transform (DFT), a fast Fourier transform (FFT) or other transforms of the speech signal can be determined. Quantization techniques that can be used to quantize the spectrum of the speech signal may include vector quantization (single stage or multi-stage) or predictive / adaptive vector quantization. In some cases, the speech encoder 302 can use an autoencoder or other neural network-based techniques to quantize the spectrum. Any other suitable spectrum quantization method can be used.

[0075] As described above, any one of the above features or any combination of the above features may be estimated, quantized, and sent to the speech decoder 304 by the speech encoder 302, depending on the specific encoder implementation used. In one illustrative example, the speech encoder 302 may estimate, quantize, and send LP coefficients, pitch lag with fractional precision, pitch gain, and pitch correlation. In another illustrative example, the speech encoder 302 may estimate, quantize, and send LP coefficients, pitch lag with fractional precision, pitch gain, pitch correlation, and Bark cepstrum of a speech signal. In another illustrative example, the speech encoder 302 may estimate, quantize, and send LP coefficients, pitch lag with fractional precision, pitch gain, pitch correlation, and spectrum (e.g., DFT, FFT, or other spectrum) of a speech signal. In another illustrative example, the speech encoder 302 may estimate, quantize, and send pitch lag with fractional precision, pitch gain, pitch correlation, and Bark cepstrum of a speech signal. In another illustrative example, the speech encoder 302 can estimate, quantize, and transmit the pitch lag, pitch gain, pitch correlation, and spectrum (e.g., DFT, FFT, or other spectrum) of the voice signal with fractional precision. In another illustrative example, the speech encoder 302 can estimate, quantize, and transmit the LP coefficients of the LTP residual, the pitch lag, pitch gain, pitch correlation, and Bark cepstrum with fractional precision. In another illustrative example, the speech encoder 302 can estimate, quantize, and transmit the LP coefficients of the LTP residual, the pitch lag, pitch gain, pitch correlation, and spectrum (e.g., DFT, FFT, or other spectrum) with fractional precision.

[0076] The speech decoder 304 includes a neural network model 334, an LTP engine 322, and a short-term LP engine 320. The neural network model 334 replaces the FCB of the conventional speech decoder. The LTP engine 322 and the short-term LP engine 320 can perform functions similar to those described above with respect to the LTP engine 222 and the short-term LP engine 220. Although Figure 3 The speech decoder 304 in and the examples described herein (e.g., in Figure 5A-Figure 8 ) is described as including both an LTP engine (e.g., LTP engine 322) and a short-term LP engine (e.g., short-term LP engine 320), but the speech decoder described herein may include only an LTP engine (without a short-term LP engine) or may include only a short-term LP engine (without an LTP engine). For example, in some embodiments, when only an LTP engine (without a short-term LP engine) is included in the speech decoder, data that is typically provided by the short-term LP engine can be provided as input to the speech decoder (e.g., from a speech encoder). In another example, in some embodiments, when only a short-term LP engine (without an LTP engine) is included in the speech decoder, data that is typically provided by the LTP engine can be provided as input to the speech decoder (e.g., from a speech encoder). In addition, although the examples described herein (e.g., in Figure 3 and Figure 5A-Figure 8 ) is described as including linear prediction filters (eg, short-term LP engine 320 and LTP engine 322), but the speech decoder and / or speech codec described herein may include long-term and short-term prediction filters in addition to linear prediction filters.

[0077] The neural network model 334 may include any neural network architecture that can be trained to model the residual data typically represented by the data in the FCB. The residual data generated by the neural network model 334 (e.g., Figure 5A The LTP residual shown in or Figure 5B The linear prediction (LP) residual shown in ) can be used to excite the LTP filter of the LTP engine 322 and / or the short-term prediction filter of the short-term LP engine 320. In one illustrative example, the neural network model 334 is a generative model (e.g., a probabilistic model in which the network does not directly output a sample value, but rather a probability distribution of the next sample value). In some embodiments, the output domain may include q-bit (e.g., 8-bit) mu-law encoded speech, which may reduce the number of possible output sample values ​​(e.g., to 256). In such an embodiment, the output of the neural network model 334 may be a probability distribution over 256 possible mu-law indices (8 bits = 2^8 = 256 possible values). The output distribution may be sampled and converted from mu-law to the linear domain to produce a speech sample.

[0078] In some examples, neural network model 334 may include two networks, including a frame rate network and a sampling rate network. Figure 44 is a diagram showing an illustrative example of a neural network model 434 including a frame rate network 442 and a sampling rate network 452. A person of ordinary skill will appreciate that the neural network model 334 may include any other suitable network architecture. In one illustrative example, the frame rate network 442 processes a 10-ms frame (e.g., 160 samples) and the sampling rate network 452 runs at 16kHz. Features 441 may be input to the frame rate network 442. The feature set included in the feature 441 may include any feature set received from the speech encoder 302 (in some cases, quantized features and in other cases, dequantized features). The features included in the feature set may depend on the encoder used. Any of the above illustrative example combinations of the above features may be used. In one of the illustrative examples provided above, the features calculated and quantized by the speech encoder 302 may include LP coefficients of the speech signal, pitch lag with fractional precision, pitch gain, pitch correlation, and Bark cepstrum. In such an example, the features 441 input to the frame rate network 442 may include 18-coefficient Bark scale cepstral coefficients of the speech signal, integer pitch periods (or lags) (e.g., between 16 and 256 samples), fractional pitch periods (or lags), and pitch correlations (e.g., between 0 and 1). The features 441 may be provided for at least the last two frames of the speech signal before the current speech frame, the current speech frame, and the next two speech frames after the current speech frame.

[0079] The fractional pitch period can be used to maximize the efficiency of the LTP filter of the LTP engine 322. For example, the fractional pitch period can be used to minimize the LTP residual energy and correlation. When the fractional pitch period is used for the LTP filter, it can also be provided as an input to the neural network of the neural network model 334 to provide the neural network with additional information about how to generate the LTP residual that is most suitable for the current target speech output. By using the fractional pitch period, the neural network of the neural network model 334 is able to predict a signal with minimal structure, which minimizes the network complexity.

[0080] For codec structures that can replace FCB and LTP operations with neural network models, fractional pitch periods will not be used because such codec structures do not have LTP cycles that need to be optimized outside of the neural network. Such codec structures can use only pitch periods as feature inputs to the neural network in place of FCB and LTP operations, and therefore do not have to be as precise. As described above, fractional pitch periods can be used to optimize the performance of the LTP filter of the LTP engine 322, removing the burden of LTP from the neural network, and the accurate pitch estimation provided by fractional pitch periods is beneficial.

[0081] like Figure 4As shown, the frame rate network 442 includes a convolution layer 444, a convolution layer 446, a fully connected layer 448, and a fully connected layer 450. For example, feature 441 is input to the convolution layer 444, and the output of the convolution layer 444 is input to the convolution layer 446. In some cases, the convolution layer 444 and the convolution layer 446 include filters of the same size. For example, the convolution layer 444 and the convolution layer 446 may include filters of size 3, thereby generating a receptive field of five frames (the first two frames and the last two frames as described above). The output of the convolution layer 446 is added to the feature data from the feature 441 and then input to the fully connected layer 448. The output from the fully connected layer 448 is input to the fully connected layer 450. The frame rate network outputs a conditional vector f that is then used by the sampling rate network 452. In an illustrative example, the conditional vector f is a 128-dimensional vector. The conditional vector f can remain constant over the duration of each frame.

[0082] The sampling rate network 452 includes a cascade layer 454, a gated recurrent unit (GRU) 456, a GRU 458, a combined fully connected layer 460, and a softmax layer 462. In some cases, a long short-term memory (LSTM) may be used instead of the GRU 456 and / or the GRU 458. The cascade layer 454 may cascade the conditional vector f, and the outputs from the LTP engine 422 and the short-term LP engine 420. The outputs from the LTP engine 422 and the short-term LP engine 420 may vary depending on the implementation. Figure 5A-Figure 8 Different examples of outputs from the LTP engine 422 and the short-term LP engine 420 are described. For example, as described below with respect to Figure 5A As described above, the input to the sampling rate network 452 may include the previous instantaneous LTP residual of the previous sample (e.g., generated by neural network model 334 or 434), previous instantaneous synthesized speech samples , LP prediction of the current sample (e.g., generated by the short-term LP engine 320 or 420), the LTP prediction of the current sample (n) (e.g., generated by LTP engine 322 or 422) and a condition vector f generated by frame rate network 442. In some cases, as described below with respect to Figure 6 As described above, the previous instantaneous LP residual of the previous sample can be (n-1) (e.g., based on combining the LTP residual from the neural network model with the LTP prediction from the LTP engine) is provided as an additional input to the sampling rate network 452. The LTP prediction and LTP residual from the LTP engine 322 or 422 are provided as inputs to provide feedback from the LTP engine 322 or 422, and the LP prediction is provided as input to provide feedback from the short-term LP engine 320 or 420. Using these inputs, the sampling rate network 452 can generate or predict the current sample (or as described below with respect to Fig. 9 The current LTP residual of the sample batch) (n).

[0083] In some examples, the input to the sampling rate network 452 can be mu-law encoded and embedded using an embedding layer (not shown). For example, the embedding can map each mu-law level (e.g., in an embedding matrix) to a vector, essentially learning a set of nonlinear functions to apply to the mu-law values. The embedding matrix can be sent to the GRU 456 and / or the GRU 458. For example, Figure 4 As shown, the embedding matrix can be input to GRU 456, and the output of GRU 456 can be input to GRU 458. In another example, the embedding matrix can be input to GRU 456 and GRU 458 respectively. The product of the embedding matrix and the corresponding sub-matrices of the non-circular weights of GRU 456 and GRU 458 can be calculated. The transformation can be applied to all gates (u, r, h) of GRU 456 and GRU 458 and all embedded inputs (e.g., , , , (n) and / or f; in some cases, one or more of these inputs may not be embedded, such as the conditional vector f and / or other inputs). Using the previous instantaneous synthesized speech sample As an example of embedding input, E can represent the embedding matrix, U (u,s) Can be expressed including application to The submatrix U of the columns where the samples are embedded (u) , and the new embedding matrix V can be derived (u,s) = U (u,s) E, the embedding matrix Acyclic terms that map directly to update gate computations.

[0084] The output from GRU 458 (such as Figure 4), or when the embedding matrices are input to GRU 456 and GRU 458, respectively, the outputs from GRU 456 and GRU 458 are provided to fully connected layer 460. In some examples, fully connected layer 460 may include two fully connected layers combined with element-wise weighted sums. Using a combined fully connected layer 460 may make it easier to calculate the output probability without significantly increasing the size of the previous layer. In an illustrative example, the combined fully connected layer 460 may be defined as dual_fc(x)=a 1 tanh(W 1 x)+a 2 tanh(W 2 x), where W 1 and W 2 is the weight matrix, and a 1 and a 2 is a weight vector. The output of the fully connected layer 460 is used together with the softmax activation of the softmax layer 462 to calculate the LTP residual (n) is the probability of each possible stimulus value.

[0085] like Figure 4 As shown, the output of the sampling rate network 452 is the current sample (or Fig. 9 The LTP residual of the sample batch) (n). As described below, the LTP residual (n) can be combined with the LTP prediction from the LTP engine 322 or 422 (n) combination to generate LP residual (n). LP residual (n) can be combined with the output of the short-term LP engine 320 or 420 to generate the current instantaneous synthesized speech sample (n). Figure 3 The illustrated reconstructed speech signal 305 may include one or more instantaneous synthesized speech samples.

[0086] The neural network models 334 and 434 may be trained using any suitable neural network training technique. An illustrative example will now be described with respect to the neural network model 434. However, one of ordinary skill will appreciate that other training techniques may be used to train the neural network models 334 and 434. Corresponding input-output pairs may be provided to the neural network model 334 for training. For example, for each instant n, the input to the frame rate network 442 may include the 18-coefficient Bark cepstral coefficients, the pitch coefficient and fractional lag (or period), and the pitch correlation of the past two speech frames before the current frame, the current speech frame, and the two subsequent frames after the current frame. In such an example, the input to the sampling rate network 452 may include the previous instantaneous LTP residual , the previous instantaneous speech sample 、Current LP Forecast and the current LTP prediction (n). The target output (or label or true value) of the neural network model 434 may include the current LTP residual (n).

[0087] Back propagation can be performed to train the neural network model 434 using the input and target output. Back propagation can include a forward pass, a loss function, a backward pass, and a parameter update to update one or more parameters (e.g., weights, biases, or other parameters). The forward pass, the loss function, the backward pass, and the parameter update are performed in one training iteration. For each input set, the process can be repeated for a certain number of iterations until the neural network model 434 is trained well enough so that the weights (and / or other parameters) of each layer are accurately adjusted.

[0088] Although the speech codec system 300 is shown to include certain components, one of ordinary skill will appreciate that the speech codec system 300 may include more than Figure 3 For example, in some instances, the speech codec system 300 may also include more or fewer components than those shown. Figure 3 One or more other memory components (e.g., one or more caches, buffers, RAM, ROM, etc.), storage components, display components, processing components, circuits, controllers, sensors, interfaces, ASICs (application specific integrated circuits), etc. not shown in the figure. In addition, in some cases, the speech codec system 300 may be part of a computing system that may include Figure 3 One or more components not shown, such as one or more wireless transceivers, one or more input devices (e.g., a touch screen, a keyboard, a mouse, an input sensor, etc.), one or more output devices (e.g., a display, a speaker, a projector, etc.), one or more storage devices, one or more other processing devices, etc.

[0089] The speech codec system 300 may be part of a computing device or may be implemented by a computing device. In some embodiments, the computing device may include an electronic device, such as a camera (e.g., a digital camera, a camera phone, a video phone, a tablet device with a built-in camera, or other suitable capture device), a mobile or fixed telephone handset (e.g., a smart phone, a cellular phone, etc.), a desktop computer, a laptop or notebook computer, a tablet computer, a head-mounted display (HMD) or a virtual reality headset, a head-up display (HUD), a vehicle (e.g., an autonomous vehicle or a human-driven vehicle), a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device, or any other suitable electronic device. In some cases, the computing device (or multiple devices) may include one or more wireless transceivers for wireless communication.

[0090] In some examples, the speech encoder 302 can be part of a first computing device or computing system (e.g., a server device or system, a vehicle, or other computing device or system), and the speech decoder 304 can be part of a second computing device or computing system (e.g., a mobile handset, a desktop computer, or other computing device or system). A computing system including the speech encoder 302 and / or a computing system including the speech decoder 304 can include the following regarding Fig.11 The computing system 1100 described herein. The computing system (or device) may communicate over any available communication network using any available communication protocol. For example, the computing system may include a user communication interface that may perform or facilitate the reception and / or transmission of wired or wireless communications using a wired and / or wireless transceiver (or a separate receiver and transmitter). Fig.11 Various illustrative examples of input-output devices and communication protocols are provided.

[0091] Figure 5A 5 is a block diagram showing an example of a speech codec system 500A including a speech encoder 502 and a speech decoder 504, the speech decoder 504 using a neural network model to generate an LTP residual, which is typically provided using an FCB. The speech codec system 500A is an example of the speech codec system 300 and can perform the above-mentioned Figure 3 and / or Figure 4 Some or all of the functionality described. Figure 5A (and Figure 5B-Figure 8 ) are shown with hat symbols (e.g., residual The terms in (n) are quantized values, while the terms shown without a hat symbol are unquantized values. Figure 5A (and in Figure 5b- Fig. 9 ), symbol z -Xrepresents an X-sample delayed element (e.g., z -1 represents a 1 sample delay). The backward triangle is an amplifier with a gain ( p ) is written inside the triangle. It has the transfer function The filter of (z) is represented by a rectangle with the transfer function ( (z)).

[0092] like Figure 5A As shown, the speech encoder 502 receives the speech signal and can be derived from the voice signal Extract features. Features 541 can be sent to the speech decoder 504. As previously mentioned, different feature sets can be determined, quantized and sent to the speech decoder 504 by the speech encoder 502. For example, the speech encoder 502 can extract any feature set, quantize the feature set, and send the feature set to the speech decoder 504. In an illustrative example using one of the above-mentioned encoder embodiments, the speech encoder 502 can estimate, quantize and send the LP coefficients of the voice signal, the pitch lag (or period) with fractional accuracy, the pitch gain, the pitch correlation and the Bark cepstral coefficients. However, it will be appreciated by those of ordinary skill that other feature sets can be extracted by the speech encoder 502.

[0093] In some examples, for LP coefficient estimation, the speech encoder 502 may use the Levinson-Durbin algorithm. In one illustrative example, for integer pitch delay estimation, the speech encoder 502 may use the EVS codec open-loop (OL) pitch estimation algorithm. For example, the OL pitch analysis calculates three estimates of the pitch lag for each frame, which can be performed by limiting the closed-loop pitch search to a small number of lags around the open-loop estimated lag to smooth the pitch evolution contour and simplify the pitch analysis. The OL pitch estimation is based on a perceptually weighted pre-emphasized input signal. The open-loop pitch analysis is performed on a twice-decimated signal (e.g., sampled at 6.4 kHz), which can reduce the computational complexity of the search process. The decimated signal can be obtained by filtering the signal with a fifth-order finite impulse response (FIR) filter having specific coefficients (e.g., coefficients {0.13, 0.23, 0.28, 0.23, 0.13}) and then downsampling the output by a factor (e.g., a factor of 2). The OL pitch analysis may be performed three times per frame to find three estimates of the pitch lag, including two in the current frame and one in a subsequent frame. The first two estimates are based on certain segments of the current frame (e.g., 10 millisecond (ms) segments). The third estimate corresponds to the subsequent frame, and the length of the segment may be a certain duration (e.g., 8.75 ms). In some examples, the OL pitch estimate may be further refined using an autocorrelation-based method, such as the autocorrelation technique for open-loop pitch analysis described in the EVS codec standard (e.g., 3GPP TS 26.445 Version 12.0.0 Version 12, which is incorporated herein by reference in its entirety and for all purposes).

[0094] In some examples, for fractional pitch lag estimation, speech encoder 502 may use the EVS codec fractional pitch estimation method. For example, by [0] and d [1] The OL pitch is further refined by maximizing the normalized correlation function with fractional resolution around (in the 12.8-kHz sampling domain). The fractional open-loop pitch lag can be calculated four times per frame (e.g., for each subframe of 64 samples). The maximum normalized correlation corresponding to the best fractional open-loop pitch lag can then be used for classification of voiced codec (VC) frames (e.g., frames with high periodicity and smooth pitch contours). The fractional open-loop pitch search can be performed by first maximizing the interval [d [i] –7…d [i] +6] with integer lags in the perceptually weighted speech s h The autocorrelation function C is used to perform the search in the first and second subframes, where d [i] =d [0] , for the third and fourth subframes, d [i] =d [1]In addition to the use of perceptually weighted speech at a 12.8 kHz sampling rate, an autocorrelation function similar to that used above for integer pitch lag estimation may be used. For example, the autocorrelation technique described in the EVS codec standard for fractional open-loop pitch estimation may be used.

[0095] In another illustrative example, for integer and / or fractional pitch lag estimation, speech encoder 502 may use an enhanced variable rate codec (EVRC) pitch estimation algorithm. For example, the input to the delay estimate and long-term prediction gain calculation may include a short-term residual vector (denoted as { }), and the output of the delay estimate and long-term prediction gain calculation may include a pitch delay estimate (expressed as and the long-term prediction gain (expressed as )). As part of initialization, the values ​​of some or all local state variables and buffers may be set to zero at startup. The pitch delay may be computed using the techniques described below. The pitch delay is the delay that maximizes the autocorrelation function (or other appropriate function) of the short-term prediction residual signal, subject to certain constraints. The computation of the pitch delay may be performed independently over two estimation windows. The first estimation window may include the entire current frame, and the second estimation window may include the second half of the current frame and the look-ahead. Rules may then be applied to combine the delay estimates and gains of the two estimation windows.

[0096] For each of the two estimation windows for each frame, a constrained search for the best delay in each window may be performed. In some cases, a non-exhaustive open-loop delay search may be performed, as described in the 3rd Generation Partnership Project 2 (3GPP2) Enhanced Variable Rate Codec (EVRC), Voice Services Codec standard (e.g., 3GPP C.S0014-D, Version 3.0, Section 4.6.3.1, which is incorporated herein by reference in its entirety and for all purposes). For example, the residual signal , can be filtered and decimated four times (or other suitable values) to produce the decimated residual signal In some cases, the decimated residual signal can be generated by applying :

[0097]

[0098] as well as:

[0099]

[0100] as well as:

[0101]

[0102] in is the decimation filter memory, is the decimation residual buffer, and For the first estimation window, it is 160 (or other appropriate values), and for the second estimation window, it is 240 (or other appropriate values). The residual signal after decimation The autocorrelation function of can be generated using the following method:

[0103]

[0104] Then we can find the value corresponding to the maximum positive correlation Delay .

[0105] Other functions may also be performed, such as refinement of an open-loop delay estimate determined using the above-described non-exhaustive open-loop delay search (e.g., for Service Option (SO) 3, SO 68, SO 70, and / or SO 73 speech codecs), long-term prediction gain calculation, smoothed delay estimate and LTP gain (e.g., for SO 3, SO 68, SO 70, and / or SO 73 speech codecs), and / or composite delay and gain calculation (e.g., for SO 3, SO 68, SO70, and / or SO 73 speech codecs), such as those described in the 3GPP2 Enhanced Variable Rate Codec (EVRC), Speech Services Codec standard (e.g., 3GPP C.S0014-D, Version 3.0, Sections 4.6.3 to 4.6.3.8, which is incorporated herein by reference in its entirety and for all purposes).

[0106] In some examples, for Bark cepstrum calculation, the speech encoder 502 can use any suitable Bark cepstrum calculation algorithm. For example, the Bark cepstrum can be calculated from a band of multiple Bark intervals (e.g., a band of 18 Bark intervals). In some cases, the Bark cepstrum is calculated once every predetermined time period (e.g., every 10 ms). In some embodiments, quantization can be performed using a cyclic or non-cyclic convolutional autoencoder (neural network). In some cases, four cepstrum vectors can be packed into each group. In some embodiments, a prediction scheme can be performed to minimize redundancy within a group while limiting dependencies across groups, which can reduce the impact of packet loss. In one example, group k may include a cepstrum vector c 4k to c 4k+3 .c 4k+3 The first component (C0) of can be independently coded and decoded with a unified scalar quantizer (e.g., a 7-bit scalar quantizer with 0.83 dB resolution). Then, c can be quantized using a codebook (e.g., a 3-stage codebook with 17 dimensions and 10 bits per stage). 4k+3The remaining components of 4k+1 (encoded independently in the previous packet) and c 4k+3 Vector c 4k+1 In some cases, if the prediction is the average value (c 4k+1 +c 4k+3 / 2), a single bit can be used for signaling if the prediction is c 4k+1 or c 4k+3 , two bits can be used for signaling. Then, for c 4k+1 The total of 13 bits can be used to encode the 18-dimensional prediction residual with the 11-bit + symbol codebook of the average predictor, or if not, with the 10-bit + symbol codebook. In some cases, there is not enough bit rate to adequately encode c 4k and c 4k+2 , in which case predictions from neighboring vectors can be used. For example, the vector c 4k From the adjacent c 4k-1 and c 4k+1 Prediction, and c 4k+2 From c 4k+1 and c 4k+3 predict.

[0107] Although various illustrative examples of techniques for determining various characteristics are provided above, any other suitable technique may be used to determine the characteristics.

[0108] The speech encoder 502 may send the features 541 (eg, after the features are quantized) to the speech decoder 504. The speech decoder 504 includes a neural network model 534, an LTP engine 522, and a short-term LP engine 520, which may be similar to the above description of Figure 3 The neural network model 334 (and in some cases the neural network model 434), the LTP engine 322, and the short-term LP engine 320 described above. In some cases, the neural network model 534 includes a frame rate network and a sampling rate network, such as the above-mentioned Figure 4 Those described.

[0109] The speech decoder 504 may also include a pitch gain estimation engine 564, a pitch estimation engine 566, and an LP coefficient estimation engine 568. For example, the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 include algorithms for implicitly estimating those features from other encoder features when the speech encoder 502 does not explicitly estimate and send one or more of those features. If the encoder implementation used does not explicitly send pitch gain, pitch estimation, and LP coefficients, the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 on the decoder side may be needed, in which case these features can be implicitly estimated from other encoder features. For the encoder implementations described above that extract, quantize, and send the LP coefficients of the speech signal, the pitch lag (or period) with fractional precision, the pitch gain, the pitch correlation, and the Bark cepstral coefficients, the LP and pitch parameters are explicitly sent, in which case the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 are not needed in these implementations.

[0110] Based on other encoder implementations that may not explicitly send LP coefficients or pitch parameters (e.g., gain, lag, period, etc.), the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 may be used to estimate those features. For example, for the encoder implementations described above that extract, quantize, and send the pitch lag, pitch gain, pitch correlation, and Bark cepstrum of the voice signal with fractional accuracy, the speech encoder does not explicitly send LP features. In this case, the LP coefficient estimation engine 568 can be used by the speech decoder 504 to estimate the LP coefficients from the Bark cepstrum of the voice signal that the encoder does send. In an illustrative example, the LP coefficients can be estimated from the Bark frequency cepstrum of the voice signal by first converting the Bark frequency cepstrum (e.g., 18-band Bark frequency cepstrum) to a linear frequency power spectral density (PSD). The PSD can then be converted to an autocorrelation using an inverse FFT. Based on the autocorrelation, the Levinson-Durbin algorithm can be used to calculate the LP coefficients. In the event that the pitch is not explicitly sent by the speech encoder 502, the speech decoder 504 can use the pitch estimation engine 566 to derive the pitch from the speech spectrum.

[0111] The neural network model 534 effectively decodes the input features into LTP residuals To generate the LTP residual .like Figure 5A As shown, the input of the neural network model can include the LTP prediction of the current sample n (output from LTP engine 522) and LP prediction (output from the short-term LP engine 520), the LTP residual of the previous sample n-1 and previous samples of the synthesized speech signal The features 541 sent by the speech encoder 502 and / or determined by one or more of the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 may also be provided to the neural network model 534. Based on this input, the neural network model 534 may generate the LTP residual for the current sample n. , the residual in the original signal and synthesize (reconstruct) the signal Minimum error is produced between .

[0112] LTP residual output from neural network model 534 can be used as input to (and can be used to stimulate) the LTP engine 522, such as Figure 5A In some examples, the LTP residual Can be used as input to the short-term LP engine 520. Figure 5A As shown, the LTP residual The pitch parameters (e.g., pitch period T and pitch gain T) sent by the speech encoder 502 and / or derived by one or more of the pitch gain estimation engine 564, the pitch estimation engine 566, and the LP coefficient estimation engine 568 are ) are input to the LTP engine 522. The LTP filter of the LTP engine 522 can model the pitch of the signal (where pitch is the key to determine the synthesized LP residual The interval or periodicity of the pulses in the sample n is determined by the LP residual signal, and thus the synthetic LP residual signal of the current sample n with the correct pulse interval is generated. The LTP prediction of the current signal sample n is predicted from the previous sample separated by one or more pitch periods. , and the tone gain can be applied to the LP residual signal of the previous sample (e.g., the LP residual of the previous sample n-1 or any other previous sample n-2, n-3, etc.) ). For example, the LTP prediction for the current sample n is can be predicted as the pitch gain Multiply by the LP residual of the previous sample nT , where T is the pitch period, indicates to the LTP filter how far back to look and copy the pulse. In one illustrative example, the LTP engine 522 may determine the LTP prediction using the following formula: The LTP prediction predicted by the LTP filter and the LTP residual from the neural network model 534 Combine (as indicated by the “+” sign) to generate the synthetic LP residual for the current sample n .

[0113] The synthesized LP residual output from the LTP engine 522 The short-term LP engine 520 may include a short-term LP filter. The LP coefficients sent by the speech encoder 502 or determined by the LP coefficient estimation engine 568 may also be input to the short-term LP engine 520. The short-term LP filter of the short-term LP engine 520 may be used to model the spectral shape (or spectral peak or envelope) of the speech signal. For example, at the speech encoder 502, the signal may be processed by an autoregressive (AR) synthesizer. Filtering is performed to obtain an AR signal. As described above, a linear predictor can be used to transform an AR signal (where the predicted AR signal is the LP prediction ) is predicted as a linear combination of the previous m samples as follows:

[0114]

[0115] in item( , … ) are estimates of the AR parameters (also called LP coefficients). The LP residual (also called prediction error) is the sum of the original AR signal and the predicted AR signal =The difference between the actual sample and the predicted sample (and therefore the difference between the actual sample and the predicted sample). At the speech encoder 502, the linear prediction codec is used to find the best LP coefficients to minimize the quadratic error function and thus minimize the error. The short-term linear prediction process removes short-term correlations from the speech signal. The LP coefficients are an effective way to represent the short-term spectrum of the speech signal. At the speech decoder 504, the short-term LP filter uses the calculated or received LP coefficients and the transfer function (Z) Determine the LP prediction for the current sample n .

[0116] The final reconstructed (or predicted) signal The synthesized LP residual output from the LTP engine 522 may be determined as and the LP forecast determined by the short-term LP engine 520 The sum (for the current sample n).

[0117] Figure 66 is a block diagram showing another example of a speech codec system 600 including a speech encoder 602 and a speech decoder 604, which uses a neural network model to generate LTP residuals (usually provided using FCB). The speech codec system 600 is similar to the speech codec system 500A and can perform similar functions as the speech codec system 500A, and the common components and functions are not repeated here.

[0118] In addition to the above Figure 5A In addition to the functions described, the previous instantaneous LP residual is provided As an additional input to the neural network model. The previous instantaneous LP residual Provides another parameter that the neural network model can use to provide an estimate of the LTP residual more precise estimate.

[0119] Figure 7 is a block diagram showing another example of a speech codec system 700. Similar to the speech codec system 500A ( Figure 5A ) and 600 ( Figure 6 ), the speech codec system 700 includes a speech encoder 702 and a speech decoder 704, which uses a neural network model to generate LTP residuals. The speech codec system 700 is similar to the speech codec system 500A and can perform similar functions as the speech codec system 500A, and the common components and functions are not repeated here.

[0120] In addition to the above Figure 5A In addition to the functions described, the LTP predictions predicted by the LTP engine and LP forecasts predicted by the short-term LP engine are combined (as indicated by the "+" symbol 769). The summed LP and LTP predictions are input to Figure 7 The neural network model in the architecture shown in Figure 1 instead provides two separate LP and LTP prediction inputs to Figure 5A and Figure 6 The neural network model in the architecture shown. In some cases, a combination of LP and LTP prediction inputs can be used Figure 5A The example shown (where the LP residual of the previous sample is not provided as an additional input to the neural network model) and can also be used Figure 6 The example shown (where the LP residual of the previous sample provided as additional input to the neural network model), such as Figure 7 As shown by the dashed line 770.

[0121] Above about Figure 5A and Figure 6The described example speech codec system architecture operates sample-by-sample by generating a synthesized speech sample at each iteration. In some embodiments, the speech codec system can be designed with a neural network model (e.g., a generative model) that performs batch generation, which can generate an entire speech frame at once, rather than generating sample-by-sample output. The batch generation alternative implementation would allow the addition of LTP residuals (e.g., ) and LTP predictions (e.g., ) occur in parallel throughout the frame.

[0122] Figure 8 8 is a diagram showing an example of a speech decoder 804 in a speech codec system 800 including a batch generation neural network model 834. As shown in the figure, the input of the neural network model 834 includes LTP residuals , reconstruct the signal LP prediction and the LTP prediction of all samples in the previous frame Features 841 of the current frame are also provided as input to the neural network model 834. Features 841 may include any of the above-described feature sets. Some or all of features 841 may be sent to the speech decoder 804 by a speech codec (not shown) and / or may be determined by the speech decoder 804.

[0123] Using the provided input, the neural network model 834 generates the LTP residual for the entire current speech frame . The LTP residual Provided as input to the LTP engine 822. As described above, the LTP engine 822 can use the LP residual of the previous frame , and tone gain (For example, by calculating ) to generate the LTP prediction for the entire frame The LTP engine 822 can combine the LTP predictions for the entire frame and LTP residual , to generate the synthetic LP residual of the current frame .

[0124] The synthesized LP residual output from the LTP engine 822 Provided as input to the short-term LP engine 820. Using the LP coefficients sent by the speech encoder or determined by the speech decoder 804, the short-term LP engine 820 can use the transfer function (z) Determine the LP prediction of the frame The final reconstructed signal of the entire frame can be Determined as the synthesized LP residual output from the LTP engine 822 and the LP forecast determined by the short-term LP engine 820 sum.

[0125] The above about Figure 5A-Figure 8 The example speech codec system architecture described is provided as an example for illustration purposes. However, other coding system architectures may be used in which the speech decoder includes a neural network model that generates the LTP residual. For example, in some embodiments, the speech decoder 505 may have Figure 5B The configuration shown in the speech codec system 500B, wherein Figure 5A Compared to the configuration of the speech decoder 504 shown in , the short-term LP engine (e.g., the short-term LP engine 520, etc.) and the LTP engine (e.g., the LTP engine 522, etc.) are swapped. Figure 5B In the example shown, the output from the neural network model 534 is a residual signal comprising a linear prediction (LP) residual of a short-term LP filter configured to excite the short-term LP engine 521. In such an embodiment, the output of the short-term LP engine 521 is provided to the LTP engine 523. For example, the short-term LP filter of the short-term LP engine 521 may be based on the LP prediction of the sample and the LP residual generated using the neural network model 534 To determine the LTP residual of the samples of the speech signal The LTP filter of the LTP engine 523 may be based on the LTP residual of the sample determined by the short-term LP filter of the short-term LP engine 521. and sample-based LTP prediction To determine the samples of the reconstructed speech signal .like Figure 5B As shown, LTP prediction The gain can be used by the LTP filter and at least one previous sample of the reconstructed speech signal (eg, the previous sample nT ) to determine. In some examples, the neural network model (e.g., neural network model 534) can provide a residual signal to a short-term linear prediction engine (e.g., short-term LP engine 520 and / or short-term LP engine 521) and an LTP engine (e.g., LTP engine 522 and / or LTP engine 523). In some examples, only one of the LTP engine and the short-term linear prediction engine can be used to determine at least one sample of the reconstructed audio signal.

[0126] In some examples, the neural network model can be of other types and architectures and can accept other input sets that may be different from those described above. A key part of any codec system architecture according to the examples provided herein is that there is an LTP synthesis loop outside the neural network model that generates the LTP residual, which is then used by the LTP synthesis loop, which means that the neural network model only needs to predict the LTP residual, and does not need to directly predict the LP residual or speech signal. The LP residual and speech signal have structures that are difficult for neural networks to model.

[0127] Using the above neural network model to generate the LTP residual is advantageous for bit rate reduction because it eliminates the FCB stage of the conventional CELP codec, which is the most bit-consuming stage of the CELP codec. The above neural network-based speech codec system models the pitch component outside the neural network model, in which case the encoder estimates and sends the pitch parameters (e.g., using the LTP engine at the speech codec). In some examples, the quantization of the pitch parameters consumes a large number of bits in conventional CELP-based codecs.

[0128] In some embodiments, pitch parameter estimation and quantization can be performed on the encoder side in a way that saves a lot of bits. For example, the relaxed code excited linear prediction (RCELP) codec technique can be used for pitch parameter estimation together with the LTP residual generation neural network model, which can allow maximum bit savings (by removing the FCB stage that needs a lot of bits and the pitch parameter quantization stage that needs a lot of bits). RCELP does not try to match the original voice signal completely. RCELP does not try to match the original residual signal, but matches a time-warped version of the original residual that conforms to the simplified pitch contour. The pitch contour is obtained by estimating the pitch delay once in each frame and linearly interpolating the pitch frame by frame. Simplified pitch representation can significantly save pitch quantization, for example by eliminating the need for fractional pitch and / or subframe pitch lag and gain updates.

[0129] The neural network model for generating LTP residual described above acts as a model of the residual signal remaining after performing LTP and short-term LP on the signal, and does not perform the brute force, model-free quantizer performed for FCB of typical CELP-based speech codecs. The neural network model requires fewer (possibly 0) bits for residual encoding and decoding to address the low bit rate requirements of various target applications (e.g., IoT, etc.).

[0130] Other advantages are provided over conventional codecs and over other novel machine learning (ML) based codecs using the LTP residual generation neural network based systems described herein. In one example, the advantages of the techniques and systems described herein over conventional codecs include better voice quality at lower bit rates. This advantage is based in part on eliminating the bit-hungry FCB stage of conventional codecs. In addition, the neural network model has been shown to be powerful and robust, allowing for tighter quantization of transmitted features while maintaining quality. Using a neural network model instead of an FCB can compensate for higher quantization errors in LP and pitch parameters by appropriately adjusting the output of the neural network model (e.g., as described above, used in a stage after the neural network model).

[0131] Advantages include lower computational complexity compared to other ML-based codecs. For example, by replacing only one stage (FCB) of a conventional codec with a neural network model, the neural network only needs to model one component (FCB) of the speech signal, but not the pitch or spectral shape components (which are processed by the LTP and short-term LP stages, respectively). By modeling only one component of the speech signal, the size and complexity of the neural network can be kept low relative to ML-based codecs that may model multiple stages of the CELP codec pipeline. Furthermore, in ML-based codecs that may attempt to replace the entire conventional decoder with a neural network model, the neural network model must model all three speech components (FCB, pitch, and spectral shape), resulting in a high complexity model (e.g., greater than 15 Giga Floating Point Operations per Second (GFLOPs)). In an ML-based codec that might attempt to replace two stages of a conventional codec (e.g., FCB and LTP) with a neural network model, the neural network model must model both speech components (FCB and pitch when FCB and LTP are replaced by a neural network model), resulting in a model complexity that is lower than an ML-based system that replaces the entire decoder, but still larger than the model required for FCB modeling alone.

[0132] The techniques and systems described herein use a neural network model to model the FCB component, while other components of a traditional speech decoder (e.g., a short-term linear prediction engine and a long-term linear prediction engine) are not replaced by the neural network, thereby allowing the complexity of the neural network model to be reduced to the required minimum, allowing the speech codec to achieve quality targets at the required complexity point.

[0133] In addition, modeling the pitch and spectral shape components of the speech signal requires filtering operations. The LTP and short-term LP stages of conventional codecs are linear time-varying filters. Filtering operations require multiplication. Neural networks cannot effectively model time-varying filtering operations because they cannot effectively model multiplication of inputs. Because the filters are time-varying, the filters cannot be modeled with constant neural network weights. Instead, the filter coefficients must be provided to the neural network as additional inputs because they must vary. The neural network calculates a linear combination of the inputs and applies nonlinearity to the results. Therefore, multiple layers / neurons are required to model the input-by-input multiplications. Therefore, it can be more efficient to implement filtering operations using algorithms outside the neural network model (e.g., digital signal processing (DSP) algorithms, such as LTP and short-term LP filtering algorithms) without attempting to include such filtering operations within the neural network model. Therefore, the systems and techniques provided herein have a computational complexity advantage over ML-based codecs that may attempt to replace the entire conventional decoder operation (FCB, LTP, and short-term LP filtering) or replace two stages of conventional decoder operations (e.g., FCB and LTP) because both the LTP and short-term LP filtering operations are performed outside the LTP residual generation neural network model.

[0134] For a codec structure that can replace the FCB and LTP operations with a neural network model, the neural network must model the LTP filter of the traditional codec. The LTP filter output is the result of the multiplication: , where g is the pitch gain, r is the LP residual, and T is the pitch period, as described above. Modeling the LTP filter operation with a neural network is inefficient for at least two reasons. For example, modeling the LTP filter operation with a neural network requires remembering past samples: ), where T can have more than 100 samples. Even state-of-the-art neural network architectures (e.g., recurrent neural network (RNN) architectures such as gated recurrent units (GRU) or long short-term memory (LSTM)) cannot preserve distant past information in their internal states. Furthermore, modeling the LTP filter operation requires multiplications, which are inefficient to represent with a neural network even if (nT) is explicitly provided as input to the neural network and the neural network does not need to rely on memory. The LTP residual generation neural network model only models the LTP residual, so there is no need to remember past samples and no multiplication operations are required.

[0135] The systems and techniques described herein also provide a convenient framework for integrating other speech codec techniques, such as RCELP (e.g., allowing the LTP module to perform with maximum bit efficiency), selective LTP low-pass filtering at the speech codec (e.g., determined based on the lowest adaptive codebook (ACB) residual energy), and / or other speech codec techniques.

[0136] Fig. 9 is a flow chart showing an example of a process 900 for reconstructing one or more audio signals using the techniques described herein. The process 900 may be performed by an audio decoder, such as Figure 3 The speech decoder 304, Figure 5A The speech decoder 504 or Figure 5B The speech decoder 505 or any other speech decoder example provided herein is implemented. The speech decoder described herein can also be used to decode other types of audio signals besides speech, such as music signals.

[0137] At block 902, process 900 includes generating, using a neural network, a residual signal of at least one sample of an audio signal based on one or more inputs to the neural network. The audio signal may include a speech signal, a music signal, and / or other types of audio signals. The residual signal is configured to excite at least one of a long-term prediction filter and a short-term prediction filter. The long-term prediction filter may include a long-term linear prediction (LTP) filter (e.g., as part of the LTP engine 322, the LTP engine 522, or other LTP engines described herein). The residual signal may include a long-term linear prediction (LTP) residual configured to excite the LTP filter (e.g., the LTP residual ). The short-term prediction filter may include a short-term linear prediction (LP) filter (e.g., as part of the short-term LP engine 320, the short-term LP engine 520, or other short-term LP engines described herein). In some embodiments, at least one sample of the audio signal includes a single sample (e.g., as may be processed by the speech codec system 500A, 500B, 600, or 700). In some embodiments, at least one sample of the audio signal includes a plurality of samples. For example, the plurality of samples are included in a frame of the audio signal (e.g., as may be processed by the speech codec system 800).

[0138] At block 904, process 900 includes determining at least one sample of a reconstructed audio signal using at least one of a long-term prediction filter and a short-term prediction filter. The reconstructed audio signal may include a reconstructed speech signal, a reconstructed music signal, and / or other types of reconstructed audio signals. The at least one sample of the reconstructed audio signal is determined based on a residual signal generated for the at least one sample of the audio signal using a neural network.

[0139] In some examples, process 900 may include determining a linear prediction (LP) residual of at least one sample of the audio signal using an LTP filter. The LP residual may include the LP residual described above. . may be based on an LTP residual generated using a neural network (for at least one sample of the audio signal) (e.g., (n)), using an LTP prediction of at least one sample of the audio signal (e.g., the LTP prediction described above) (n)) to determine the LP residual of at least one sample of the audio signal. As described above, the gain and the LP residual of at least one previous sample of the audio signal (eg, ,For example, ) to determine the LTP prediction (n). For example, a gain (eg, a pitch gain) may be applied to the LP residual of at least one previous sample of the audio signal to generate an LTP prediction (eg, based on calculating ).

[0140] In some examples, process 900 may include determining at least one sample of the reconstructed audio signal using a short-term LP filter. ) and an LP residual determined for at least one sample of the audio signal (eg, the LP residual described above ) to determine at least one sample of the reconstructed audio signal.

[0141] In some examples, one or more inputs to the neural network include an LTP prediction for at least one sample of the audio signal (e.g., ), an LP prediction for at least one sample of the audio signal (e.g., ), a previous LTP residual determined by the neural network for at least one previous sample of the audio signal (e.g., (n-1)), reconstructing at least one previous sample of the audio signal (e.g., (n-1)), or any one or more thereof in any combination. Other inputs may include any of the features described above, such as feature 441, feature 541, or other features described herein.

[0142] In some examples, the LTP residual generated for at least one sample of the audio signal using a neural network is combined with the LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal. Figure 5A As mentioned above, the LTP residual (n) can be predicted with LTP Combine to generate LP residual In some examples, the LP residual determined for at least one sample of the audio signal is combined with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal. Figure 5A The LP residual determined for at least one sample of the audio signal with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal (n).

[0143] In some examples, the residual signal generated by the neural network includes an LP residual configured to excite a short-term LP filter. Figure 5B As described, process 900 may include determining an LTP residual of at least one sample of the audio signal using a short-term LP filter. As described above, the LTP residual of at least one sample may be determined based on an LP prediction of at least one sample of the audio signal and an LP residual generated using a neural network.

[0144] In some examples, process 900 may include determining at least one sample of the reconstructed audio signal using an LTP filter. The LTP residual of at least one sample of the audio signal may be determined based on the LTP residual (e.g., (n)) and at least one previous sample (e.g., previous sample nT) of the reconstructed audio signal using the gain and ) determines the LTP prediction of at least one sample (e.g., ) to determine at least one sample of the reconstructed audio signal.

[0145] about Fig. 9 The above examples described may be used alone or in any combination.

[0146] Fig.10 1 is a flow chart illustrating an example of a process 1000 for reconstructing one or more audio signals using the techniques described herein. The process 1000 may be performed by a speech decoder, such as Figure 3 The speech decoder 304, Figure 5A The speech decoder 504 or Figure 5B 's speech decoder 505, or any other speech decoder example provided herein.

[0147] In box 1002, process 1000 includes using a neural network to generate a long-term linear prediction (LTP) residual of at least one sample of an audio signal based on one or more inputs. The audio signal may include a voice signal, a music signal, and / or other types of audio signals. The LTP residual is configured to excite an LTP filter. The LTP filter may be part of an LTP engine 322, an LTP engine 522, or other LTP engines described herein. In some embodiments, at least one sample of an audio signal includes a single sample (e.g., which may be processed by a speech codec system 500A, 500B, 600, or 700). In some embodiments, at least one sample of an audio signal includes a plurality of samples. For example, a plurality of samples are included in a frame of an audio signal (e.g., which may be processed by a speech codec system 800).

[0148] At block 1004, process 1000 includes determining a linear prediction (LP) residual of at least one sample of the audio signal using the LTP filter. The LP residual of at least one sample of the audio signal may be determined based on the LTP residual generated for the at least one sample using the neural network and an LTP prediction of the at least one sample determined using the gain and the LP residual of at least one previous sample of the audio signal. The LTP residual may include the LTP residual (n), and the LP residual can include the above LP residual As mentioned above, the gain and the LP residual of at least one previous sample of the audio signal (e.g., ,For example, ) to determine the LTP prediction (n). For example, a gain (eg, a pitch gain) may be applied to the LP residual of at least one previous sample of the audio signal to generate an LTP prediction (eg, based on calculating ).

[0149] At block 1006, process 1000 includes determining at least one sample of a reconstructed audio signal using a short-term linear prediction (LP) filter. The reconstructed audio signal may include a reconstructed speech signal, a reconstructed music signal, and / or other types of reconstructed audio signals. The reconstructed audio signal may be based on an LP prediction (e.g., the LP prediction described above). ) and an LP residual determined for at least one sample of the audio signal (eg, the LP residual described above ) to determine at least one sample of the reconstructed audio signal.

[0150] In some examples, one or more inputs to the neural network include an LTP prediction for at least one sample of the audio signal (e.g., ), an LP prediction for at least one sample of the audio signal (e.g., ), a previous LTP residual determined by the neural network for at least one previous sample of the audio signal (e.g., (n-1)), reconstructing at least one previous sample of the audio signal (e.g., (n-1)), or any one or more thereof in any combination. Other inputs may include any of the features described above, such as feature 441, feature 541, or other features described herein.

[0151] In some examples, the LTP residual generated for at least one sample of the audio signal using a neural network is combined with the LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal. Figure 5A As mentioned above, the LTP residual (n) can be predicted with LTP Combine to generate LP residual In some examples, the LP residual determined for at least one sample of the audio signal is combined with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal. Figure 5A The LP residual determined for at least one sample of the audio signal with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal (n).

[0152] about Fig.10 The above examples described may be used alone or in any combination.

[0153] In some examples, processes 900 and 1000 may be performed by a computing device or apparatus, such as a device including the speech codec system 300 or other speech codec systems described herein (e.g., a device having the computing system 1100). In some cases, the computing device or apparatus may include one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components configured to perform the steps of processes 900 and 1000. In some examples, the computing device may include a mobile device, a desktop computer, a server computer, and / or a server system, or other types of computing devices. The components of the computing device (e.g., one or more input devices, one or more output devices, one or more processors, one or more microprocessors, one or more microcomputers, and / or other components) may be implemented in a circuit. For example, the component may include and / or may be implemented using an electronic circuit or other electronic hardware, which may include one or more programmable electronic circuits (e.g., a microprocessor, a graphics processing unit (GPU), a digital signal processor (DSP), a central processing unit (CPU), and / or other suitable electronic circuits), and / or may include and / or be implemented using computer software, firmware, or any combination thereof to perform the various operations described herein. The computing device may also include a display and / or speakers (as examples of output devices), a network interface configured to communicate and / or receive data, one or more receivers, transmitters, and / or transceivers configured to communicate voice data (as examples of input devices and / or output devices). In some examples, the network interface, transceiver, and / or transmitter may be configured to transmit Internet Protocol (IP)-based data or other network data.

[0154] Processes 900 and 1000 are illustrated as a logical flow diagram, the operations of which represent a sequence of operations that can be implemented in hardware, computer instructions, or a combination thereof. In the context of computer instructions, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the operations. Typically, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform specific functions or implement specific data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined in any order and / or in parallel to implement the process.

[0155] In addition, processes 900 and 1000 may be executed under the control of one or more computer systems configured with executable instructions, and may be implemented as code (e.g., executable instructions, one or more computer programs, or one or more applications) that is jointly executed on one or more processors through hardware or a combination thereof. As described above, the code may be stored on a computer-readable or machine-readable storage medium, for example, in the form of a computer program including a plurality of instructions executable by one or more processors. The computer-readable or machine-readable storage medium may be non-transitory.

[0156] Fig.11 1 shows an example of a computing system 1100 that can implement the various techniques described herein. For example, the computing system 1100 can implement Figure 3 The speech codec system 300 shown or any other speech codec system described herein. The components of the computing system 1100 communicate with each other using connection 1105. The connection 1105 can be a physical connection via a bus, or a direct connection to the processor 1110, such as in a chipset architecture. The connection 1105 can also be a virtual connection, a network connection, or a logical connection.

[0157] In some embodiments, computing system 1100 is a distributed system, where the functionality described in the present disclosure may be distributed across a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality described for that component. In some embodiments, a component may be a physical device or a virtual device.

[0158] The example system 1100 includes at least one processing unit (CPU or processor) 1110 and a connection 1105 that couples various system components including system memory 1115 (e.g., read-only memory (ROM) 1120 and random access memory (RAM) 1125) to the processor 1110. The computing system 1100 may include a cache of high-speed memory 1112 that is directly connected to, close to, or integrated as part of the processor 1110. In some cases, the computing system 1100 may copy data from memory 1115 and / or storage device 1130 to the cache 1112 for quick access by the processor 1110. In this way, the cache may provide a performance enhancement that avoids processor 1110 delays while waiting for data. These modules and other modules may control or be configured to control the processor 1110 to perform various actions. Other computing device memory 1115 may also be used. The memory 1115 may include multiple different types of memory with different performance characteristics.

[0159] Processor 1110 may include any general purpose processor and hardware services or software services, such as service 1 1132, service 2 1134, and service 3 1136 stored in storage device 1130, which are configured to control processor 1110 as well as a dedicated processor where software instructions are incorporated into the actual processor design. Processor 1110 may essentially be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0160] To enable user interaction, the computing system 1100 includes an input device 1145, which can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 1100 can also include an output device 1135, which can be one or more of a variety of output mechanisms known to those skilled in the art, such as a display, a projector, a television, a speaker device, etc. In some examples, a multimodal system can enable a user to provide multiple types of input / output to communicate with the computing system 1100. The computing system 1100 can include a communication interface 1140, which can generally control and manage user input and system output. The communication interface may perform or facilitate reception and / or transmission of wired or wireless communications via a wired and / or wireless transceiver, including those using an audio jack / plug, a microphone jack / plug, a universal serial bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, a fiber optic port / plug, a proprietary wired port / plug, Bluetooth® wireless signal transmission, Bluetooth® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), world interoperability for microwave access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, 3G / 4G / 5G / LTE cellular data network wireless signal transmission, ad-hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 1140 may also include one or more global navigation satellite system (GNSS) receivers or transceivers for determining the location of the computing system 1100 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russian-based Global Navigation Satellite System (GLONASS), the China-based BeiDou Navigation Satellite System (BDS), and the European-based Galileo GNSS. There is no restriction on operation on any particular hardware configuration, so the basic features herein may be easily replaced with improved hardware or firmware configurations as they are developed.

[0161] The storage device 1130 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store computer-accessible data, such as a magnetic cartridge, a flash memory card, a solid-state storage device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / strip, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disc read-only memory (CD-ROM) disc, a rewritable compact disc (CD) disc, a digital video disc (DVD) disc, a Blu-ray disc, a holographic disc, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick® card, a smart card chip, a Europay MasterCard, and a Visa (Europay) card. V) chip, subscriber identity module (SIM) card, mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, random access memory (RAM), static RAM (SRAM), dynamic RAM (DRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash EPROM (FLASHEPROM), cache memory (L1 / L2 / L3 / L4 / L5 / L#), resistance random access memory (RRAM / ReRAM), phase change memory (PCM), spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0162] Storage device 1130 may include software services (e.g., service 1 1132, service 2 1134, and service 3 1136, and / or other services), servers, services, etc., which enable the system to perform functions when the code defining such software is executed by processor 1110. In some embodiments, hardware services that perform specific functions may include software components stored in a computer-readable medium that are combined with necessary hardware components such as processor 1110, connection 1105, output device 1135, etc. to perform the functions.

[0163] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media that can store data, and does not include carrier waves and / or transient electronic signals that are propagated wirelessly or through wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memory, or storage devices. Computer-readable media may have stored thereon code and / or machine-executable instructions that may represent any combination of procedures, functions, subroutines, programs, routines, subroutines, modules, software packages, classes, or instructions, data structures, or program statements. A code segment may be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. may be passed, forwarded, or sent by any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0164] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and signals themselves.

[0165] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those of ordinary skill in the art that embodiments may be practiced without these specific details. For the sake of clarity of explanation, in some cases, the present technology may be presented as including separate functional blocks, which include functional blocks including devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. In addition to the components shown in the figures and / or described herein, other components may also be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary details to avoid obscuring the embodiments.

[0166] Various embodiments may be described above as processes or methods, which are depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many operations may be performed in parallel or concurrently. In addition, the order of operations may be rearranged. A process terminates when its operations are completed, but there may be other steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to a calling function or a main function.

[0167] The processes and methods according to the above examples can be implemented using computer executable instructions stored in or otherwise obtained from a computer readable medium. These instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Some of the computer resources used can be accessed through a network. Computer executable instructions can be, for example, binary files, intermediate format instructions (e.g., assembly language, firmware, source code, etc.). Examples of computer readable media that can be used to store instructions, information used, and / or information created during the method according to the examples include disks or optical disks, flash memory, USB devices provided with non-volatile memory, network storage devices, etc.

[0168] Devices according to these disclosed implementation processes and methods may include hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) that perform necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical form factor examples include laptop computers, smart phones, mobile phones, tablet computer devices or other small personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or plug-in cards. By way of further example, such functions may also be implemented on circuit boards between different chips, or on different processes performed in a single device.

[0169] Instructions, media for transmitting such instructions, computing resources for executing the instructions, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0170] In the foregoing description, various aspects of the present application are described with reference to the specific embodiments of the present application, but those skilled in the art will recognize that the present application is not limited thereto. Therefore, although the illustrative embodiments of the present application have been described in detail herein, it should be understood that the concepts of the present disclosure may be embodied and applied differently in other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. The various features and aspects of the above-mentioned application may be used individually or in combination. In addition, without departing from the broader spirit and scope of this specification, embodiments may be used in any number of environments and applications outside of those described herein. Therefore, the description and the accompanying drawings are considered to be illustrative rather than restrictive. For the purpose of illustration, the methods are described in a particular order. It should be understood that in alternative embodiments, the methods may be performed in an order different from the described order.

[0171] One of ordinary skill will appreciate that the less than ("<") and greater than (">") symbols or terms used herein may be replaced by less than or equal to (" ”) and greater than or equal to (“ ”) symbol without departing from the scope of this specification.

[0172] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (such as a microprocessor, or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0173] The phrase "coupled to" refers to any component that is physically connected directly or indirectly to another component, and / or that communicates with another component (eg, connected to other components via a wired or wireless connection, and / or other suitable communication interfaces).

[0174] Claim language or other language reciting “at least one of” a set and / or “one or more” in a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, claim language reciting “at least one of A and B” means A, B, or A and B. In another example, claim language reciting “at least one of A, B, and C” means A, B, C, or A and B, or A and C, or B and C, or A and B and C. The language “at least one of” a set and / or “one or more” in a set does not limit the set to the items listed in the set. For example, claim language reciting “at least one of A and B” may mean A, B, or A and B, and may additionally include items not listed in the A and B set.

[0175] The various illustrative logic blocks, modules, circuits and algorithm steps described in conjunction with the examples disclosed herein can be implemented as electronic hardware, computer software, firmware or a combination thereof. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits and steps have been generally described above with respect to their functionality. Whether these functions are implemented as hardware or software depends on the specific application and design constraints imposed on the entire system. The technician can implement the described functions in different ways for each specific application, but this implementation decision should not be interpreted as causing deviations from the scope of the application.

[0176] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. These techniques may be implemented in any of a variety of devices (e.g., general-purpose computers, wireless communication device handsets, or integrated circuit devices with multiple uses, including applications in wireless communication device handsets and other devices). Any features described as modules or components may be implemented together in an integrated logic device, or individually as discrete but interoperable logic devices. If implemented in software, these techniques may be implemented at least in part by a computer-readable data storage medium, which includes a program code including instructions for executing one or more of the above methods, algorithms, and / or operations when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, these techniques may be implemented at least in part by a computer-readable communication medium that carries or transmits program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer (e.g., a propagated signal or wave).

[0177] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a DSP and a microprocessor, a combination of multiple microprocessors, a combination of one or more microprocessors with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the aforementioned structures, any combination of the aforementioned structures, or any other structure or device suitable for implementing the technology described herein.

[0178] Illustrative examples of the present disclosure include:

[0179] Example 1: An apparatus for reconstructing one or more audio signals, comprising: a memory configured to store audio data; and one or more processors implemented in a circuit and coupled to the memory, the one or more processors configured to: generate a residual signal for at least one sample of the audio signal using a neural network based on one or more inputs to the neural network, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter; and determine at least one sample of the reconstructed audio signal using at least one of the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0180] Example 2: The apparatus of Example 1, wherein the long term prediction filter is a long term linear prediction (LTP) filter.

[0181] Example 3: The apparatus of any of Examples 1 or 2, wherein the residual signal comprises a long term linear prediction (LTP) residual configured to excite an LTP filter.

[0182] Example 4: An apparatus according to Example 3, wherein the one or more processors are configured to: determine a linear prediction (LP) residual of at least one sample of an audio signal using an LTP filter, the LP residual of at least one sample of the audio signal being determined based on an LTP residual generated for at least one sample of the audio signal using a neural network and an LTP prediction of at least one sample of the audio signal determined using a gain and an LP residual of at least one previous sample of the audio signal.

[0183] Example 5: An apparatus according to Example 4, wherein the gain is applied to the LP residual of at least one previous sample of the audio signal.

[0184] Example 6: The apparatus of any one of Examples 4 to 5, wherein the short-term prediction filter is a short-term linear prediction (LP) filter.

[0185] Example 7: An apparatus according to Example 6, wherein the one or more processors are configured to: use a short-term LP filter to determine at least one sample of a reconstructed audio signal, wherein the at least one sample of the reconstructed audio signal is determined based on an LP prediction and an LP residual determined for at least one sample of the audio signal.

[0186] Example 8: An apparatus according to any of Examples 1 to 7, wherein one or more inputs to the neural network include at least one of an LTP prediction for at least one sample of the audio signal, an LP prediction for at least one sample of the audio signal, a previous LTP residual determined by the neural network for at least one previous sample of the audio signal, and at least one previous sample of the reconstructed audio signal.

[0187] Example 9: An apparatus according to any one of Examples 4 to 8, wherein the LTP residual generated for at least one sample of the audio signal using a neural network is combined with the LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal.

[0188] Example 10: An apparatus according to any of Examples 4 to 9, wherein the LP residual determined for at least one sample of the audio signal is combined with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal.

[0189] Example 11: The apparatus of any one of Examples 1 to 10, wherein the short-term prediction filter is a short-term linear prediction (LP) filter.

[0190] Example 12: The apparatus of any of Examples 1 to 11, wherein the residual signal comprises a linear prediction (LP) residual configured to excite a short-term LP filter.

[0191] Example 13: An apparatus according to any of Examples 11 to 12, wherein one or more processors are configured to: determine a long-term linear prediction (LTP) residual of at least one sample of an audio signal using the short-term LP filter, the LTP residual of at least one sample being determined based on an LP prediction of at least one sample of the audio signal and an LP residual generated using a neural network.

[0192] Example 14: The apparatus according to any one of Examples 11 to 13, wherein the long term prediction filter is a long term linear prediction (LTP) filter.

[0193] Example 15: An apparatus according to Example 14, wherein one or more processors are configured to: determine at least one sample of a reconstructed audio signal using an LTP filter, wherein the at least one sample of the reconstructed audio signal is determined based on an LTP residual of at least one sample of the audio signal and an LTP prediction of at least one sample of the audio signal determined using a gain and at least one previous sample of the reconstructed audio signal.

[0194] Example 16: An apparatus according to any of Examples 1 to 15, wherein at least one sample of the audio signal comprises a single sample.

[0195] Example 17: An apparatus according to any one of Examples 1 to 16, wherein at least one sample of the audio signal comprises a plurality of samples.

[0196] Example 18: An apparatus according to any one of Examples 1 to 17, wherein the plurality of samples are included in a frame of the audio signal.

[0197] Example 19: An apparatus according to any one of Examples 1 to 18, wherein the audio signal comprises a speech signal, and wherein reconstructing the audio signal comprises reconstructing the speech signal.

[0198] Example 20: A method of reconstructing one or more audio signals, comprising: using a neural network, based on one or more inputs to the neural network, generating a residual signal for at least one sample of the audio signal, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter; and using at least one of the long-term prediction filter and the short-term prediction filter to determine at least one sample of the reconstructed audio signal, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0199] Example 21: The method of Example 20, wherein the long term prediction filter is a long term linear prediction (LTP) filter.

[0200] Example 22: The method according to any of Examples 20 or 21, wherein the residual signal comprises a long term linear prediction (LTP) residual configured to excite an LTP filter.

[0201] Example 23: A method according to Example 22, wherein one or more processors are configured to: determine a linear prediction (LP) residual of at least one sample of an audio signal using an LTP filter, the LP residual of at least one sample of the audio signal being determined based on an LTP residual generated for at least one sample of the audio signal using a neural network and an LTP prediction of at least one sample of the audio signal determined using a gain and an LP residual of at least one previous sample of the audio signal.

[0202] Example 24: The method of Example 23, wherein the gain is applied to the LP residual of at least one previous sample of the audio signal.

[0203] Example 25: The method according to any one of Examples 23 to 24, wherein the short-term prediction filter is a short-term linear prediction (LP) filter.

[0204] Example 26: The method according to Example 25, further comprising determining at least one sample of the reconstructed audio signal using a short-term LP filter, the at least one sample of the reconstructed audio signal being determined based on the LP prediction and the LP residual determined for the at least one sample of the audio signal.

[0205] Example 27: A method according to any of Examples 20 to 26, wherein one or more inputs to the neural network include at least one of an LTP prediction for at least one sample of the audio signal, an LP prediction for at least one sample of the audio signal, a previous LTP residual determined by the neural network for at least one previous sample of the audio signal, and at least one previous sample of the reconstructed audio signal.

[0206] Example 28: A method according to any one of Examples 23 to 27, wherein an LTP residual generated for at least one sample of the audio signal using a neural network is combined with an LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal.

[0207] Example 29: The method of any of Examples 23 to 28, wherein the LP residual determined for at least one sample of the audio signal is combined with the LP prediction of at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal.

[0208] Example 30: The method according to any one of Examples 20 to 29, wherein the short-term prediction filter is a short-term linear prediction (LP) filter.

[0209] Example 31: The method of any of Examples 20 to 30, wherein the residual signal comprises a linear prediction (LP) residual configured to excite a short-term LP filter.

[0210] Example 32: The method according to any one of Examples 30 to 31 further includes determining a long-term linear prediction (LTP) residual of at least one sample of an audio signal using a short-term LP filter, wherein the LTP residual of at least one sample is determined based on an LP prediction of at least one sample of the audio signal and an LP residual generated using a neural network.

[0211] Example 33: The method according to any one of Examples 30 to 32, wherein the long term prediction filter is a long term linear prediction (LTP) filter.

[0212] Example 34: The method according to Example 33 also includes determining at least one sample of the reconstructed audio signal using an LTP filter, wherein the at least one sample of the reconstructed audio signal is determined based on the LTP residual of at least one sample of the audio signal and the LTP prediction of at least one sample of the audio signal determined using a gain and at least one previous sample of the reconstructed audio signal.

[0213] Example 35: A method according to any one of Examples 20 to 34, wherein at least one sample of the audio signal comprises a single sample.

[0214] Example 36: A method according to any one of Examples 20 to 35, wherein at least one sample of the audio signal comprises a plurality of samples.

[0215] Example 37: A method according to any one of Examples 20 to 36, wherein the plurality of samples are included in a frame of the audio signal.

[0216] Example 38: The method of any one of Examples 20 to 37, wherein the audio signal comprises a speech signal, and wherein reconstructing the audio signal comprises reconstructing the speech signal.

[0217] In some embodiments, the above examples 1-19 and / or 20-38 can be implemented using a computer-readable storage medium storing instructions that, when executed, cause one or more processors of a device to perform some or all of the various operations. For example, a computer-readable storage medium storing instructions can be provided that, when executed, causes one or more processors of a device for decoding video data to: generate a residual signal for at least one sample of an audio signal using a neural network based on one or more inputs to the neural network, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter; and determine at least one sample of a reconstructed audio signal using at least one of the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

[0218] In some embodiments, the above examples 1-19 and / or 20-38 can be implemented using an apparatus including one or more components for performing some or all of the various operations. For example, an apparatus for reconstructing one or more audio signals can be provided, including: a component for generating a residual signal of at least one sample of the audio signal using a neural network based on one or more inputs to the neural network, the residual signal being configured to excite at least one of a long-term prediction filter and a short-term prediction filter; and a component for determining at least one sample of the reconstructed audio signal using at least one of the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the residual signal generated for at least one sample of the audio signal using the neural network.

Claims

1. An apparatus for reconstructing one or more audio signals, comprising: a memory configured to store audio data; as well as one or more processors implemented in circuitry and coupled to the memory, the one or more processors being configured to: generating (902) a first residual signal for at least one sample of the audio signal using a neural network, the residual signal being configured to excite at least a first filter; determining, using the first filter and based on the first residual signal generated for at least one sample of the audio signal using the neural network, a second residual signal for input to a second filter, wherein the first filter is a long-term prediction filter and the second filter is a short-term prediction filter, or wherein the first filter is a short-term prediction filter and the second filter is a long-term prediction filter; and At least one sample of a reconstructed audio signal is determined (904) using the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the second residual signal.

2. The apparatus of claim 1, wherein the neural network comprises a frame rate network (442) and a sampling rate network (452).

3. An apparatus according to claim 2, wherein features (441) are input to the frame rate network, and wherein the first residual signal is output by the sampling rate network. 4 . The apparatus of claim 3 , wherein the features are input to the frame rate network for one or more frames of the audio signal, each frame comprising a plurality of samples of the audio signal.

5. The device according to claim 4, wherein: The features are input to the frame rate network for at least two frames preceding a current frame of the audio signal, the current frame of the audio signal, and two future frames of the audio signal.

6. An apparatus according to any one of claims 3 to 5, wherein a fractional pitch period is input to the long-term prediction filter, and wherein the features input to the frame rate network include the fractional pitch period.

7. The apparatus of claim 1, wherein the neural network is a probabilistic model, wherein the neural network outputs a probability distribution.

8. The device of claim 1, wherein the long-term prediction filter is a long-term linear prediction (LTP) filter, and wherein the first residual signal comprises a long-term linear prediction (LTP) residual configured to excite the LTP filter.

9. The device according to claim 8, wherein: The one or more processors are configured to: A linear prediction LP residual of at least one sample of the audio signal is determined as the second residual signal using the LTP filter, the LP residual of at least one sample of the audio signal being determined based on the LTP residual generated for at least one sample of the audio signal using the neural network and an LTP prediction of the at least one sample of the audio signal determined using a gain and the LP residual of at least one previous sample of the audio signal, wherein the gain is applied to the LP residual of at least one previous sample of the audio signal.

10. The apparatus of claim 9, wherein the short-term prediction filter is a short-term linear prediction (LP) filter, and wherein the one or more processors are configured to: At least one sample of the reconstructed audio signal is determined using the short-term LP filter, the at least one sample of the reconstructed audio signal being determined based on an LP prediction and the LP residual determined for at least one sample of the audio signal.

11. The apparatus of claim 10, wherein the one or more inputs to the neural network include at least one of an LTP prediction for at least one sample of the audio signal, an LP prediction for at least one sample of the audio signal, a previous LTP residual determined by the neural network for at least one previous sample of the audio signal, and at least one previous sample of the reconstructed audio signal; and / or wherein an LTP residual generated for at least one sample of the audio signal using the neural network is combined with an LTP prediction of at least one sample of the audio signal to generate an LP residual for at least one sample of the audio signal; and / or The LP residual determined for at least one sample of the audio signal is combined with the LP prediction for at least one sample of the audio signal to determine at least one sample of the reconstructed audio signal.

12. The apparatus of claim 1 , wherein the short-term prediction filter is a short-term linear prediction (LP) filter, and wherein the first residual signal comprises a linear prediction (LP) residual configured to excite the short-term (LP) filter, wherein the one or more processors are configured to: determining a long term linear prediction (LTP) residual of at least one sample of the audio signal using the short term LP filter, the LTP residual of the at least one sample being determined based on an LP prediction for at least one sample of the audio signal and an LP residual generated using the neural network, wherein the long term prediction filter is a long term linear prediction (LTP) filter; and At least one sample of the reconstructed audio signal is determined using the LTP filter, the at least one sample of the reconstructed audio signal being determined based on an LTP residual for at least one sample of the audio signal and an LTP prediction of at least one sample of the audio signal determined using a gain and at least one previous sample of the reconstructed audio signal.

13. An apparatus according to any preceding claim, wherein the at least one sample of the audio signal comprises a single sample; or Wherein the at least one sample of the audio signal comprises a plurality of samples, wherein the plurality of samples are comprised in a frame of the audio signal.

14. An apparatus according to any preceding claim, wherein the audio signal comprises a speech signal, and wherein the reconstructed audio signal comprises a reconstructed speech signal.

15. A method for reconstructing one or more audio signals, the method comprising: generating (902) a first residual signal for at least one sample of the audio signal using a neural network, the residual signal being configured to excite at least a first filter; determining, using the first filter and based on the first residual signal generated for at least one sample of the audio signal using the neural network, a second residual signal for input to a second filter, wherein the first filter is a long-term prediction filter and the second filter is a short-term prediction filter, or wherein the first filter is a short-term prediction filter and the second filter is a long-term prediction filter; and At least one sample of a reconstructed audio signal is determined (904) using the long-term prediction filter and the short-term prediction filter, the at least one sample of the reconstructed audio signal being determined based on the second residual signal.

Images