Audio sample reconstruction using neural networks and multiple subband networks - Patents.com

Abstract

The device includes a neural network, a first sub-band neural network, a second sub-band neural network, and a reconstructor. The neural network processes a neural network input to generate a neural network output. The neural network input includes at least one previous audio sample. The first sub-band neural network processes the first sub-band network input to generate a first sub-band audio sample. The first sub-band network input includes at least the neural network output. The second sub-band neural network processes the second sub-band network input to generate a second sub-band audio sample. The second sub-band network input includes at least the neural network output. The reconstructor generates a reconstructed audio sample based on the first sub-band audio sample and the second sub-band audio sample. The at least one previous audio sample includes a previous sub-band audio sample, a previous reconstructed audio sample, or both.

Description

[Technical field]

[0001] (CROSS REFERENCE TO RELATED APPLICATIONS) This application claims the benefit of priority to commonly owned Greek Patent Application No. 20220100343, filed April 26, 2022, the entire contents of which are expressly incorporated herein by reference.

[0002] The present disclosure relates generally to audio sample reconstruction using neural networks and multiple sub-band networks.

[0003] Description of Related Art Advances in technology have resulted in smaller and more powerful computing devices. For example, there are now a variety of portable personal computing devices, including wireless telephones, such as mobile phones and smart phones, tablet and laptop computers, that are small, lightweight, and easily carried by users. These devices may communicate voice and data packets over wireless networks. In addition, many such devices incorporate additional functionality, such as digital still cameras, digital video cameras, digital recorders, and audio file players. Such devices may also process executable instructions, including software applications, such as web browser applications, that may be used to access the Internet. Thus, these devices may contain significant computing power.

[0004] Such computing devices may include the capability to generate sample data, such as reconstructed audio samples. For example, the device may receive encoded audio data that is decoded and processed to generate reconstructed audio samples. The process of generating reconstructed audio samples using a single neural network tends to have high computational complexity, which may result in slower processing and higher memory usage. Summary of the Invention

[0005] According to one implementation of the present disclosure, a device includes a neural network, a first sub-band neural network, a second sub-band neural network, and a reconstructor. The neural network is configured to process one or more neural network inputs to generate a neural network output. The one or more neural network inputs include at least one previous audio sample. The first sub-band neural network is configured to process the one or more first sub-band network inputs to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal. The one or more first sub-band network inputs include at least a neural network output. The first reconstructed sub-band audio signal corresponds to a first audio sub-band. The second sub-band neural network is configured to process one or more second sub-band network inputs to generate at least one second sub-band audio sample of a second reconstructed sub-band audio signal. The one or more second sub-band network inputs include at least a neural network output. The second reconstructed subband audio signal corresponds to a second audio subband different from the first audio subband. The reconstructor is configured to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample. The at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0006] According to another implementation of the present disclosure, the method includes processing one or more neural network inputs using a neural network to generate a neural network output. The one or more neural network inputs include at least one previous audio sample. The method also includes processing the one or more first sub-band network inputs using a first sub-band neural network to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal. The one or more first sub-band network inputs include at least a neural network output. The first reconstructed sub-band audio signal corresponds to a first audio sub-band. The method further includes processing the one or more second sub-band network inputs using a second sub-band neural network to generate at least one second sub-band audio sample of a second reconstructed sub-band audio signal. The one or more second sub-band network inputs include at least a neural network output. The second reconstructed sub-band audio signal corresponds to a second audio sub-band different from the first audio sub-band. The method also includes using a reconstructor to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, where the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0007] According to another implementation of the present disclosure, a non-transitory computer-readable medium includes instructions, which, when executed by one or more processors, cause the one or more processors to process one or more neural network inputs using a neural network to generate a neural network output. The one or more neural network inputs include at least one previous audio sample. The instructions, when executed by the one or more processors, also cause the one or more processors to process one or more first sub-band network inputs using a first sub-band neural network to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal. The one or more first sub-band network inputs include at least the neural network output. The first reconstructed sub-band audio signal corresponds to a first audio sub-band. The instructions, when executed by the one or more processors, also cause the one or more processors to process the one or more second sub-band network inputs using a second sub-band neural network to generate at least one second sub-band audio sample of a second reconstructed sub-band audio signal. The one or more second sub-band network inputs include at least a neural network output. The second reconstructed sub-band audio signal corresponds to a second audio sub-band different from the first audio sub-band. The instructions, when executed by the one or more processors, also cause the one or more processors to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first sub-band audio sample and the at least one second sub-band audio sample.The at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0008] According to another implementation of the present disclosure, the apparatus includes means for processing one or more neural network inputs using a neural network to generate a neural network output. The one or more neural network inputs include at least one previous audio sample. The apparatus also includes means for processing one or more first sub-band network inputs using a first sub-band neural network to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal. The one or more first sub-band network inputs include at least a neural network output. The first reconstructed sub-band audio signal corresponds to a first audio sub-band. The apparatus further includes means for processing one or more second sub-band network inputs using a second sub-band neural network to generate at least one second sub-band audio sample of a second reconstructed sub-band audio signal. The one or more second sub-band network inputs include at least a neural network output. The second reconstructed sub-band audio signal corresponds to a second audio sub-band different from the first audio sub-band. The apparatus also includes means for generating at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, where the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0009] Other aspects, advantages, and features of the present disclosure will become apparent after consideration of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims. [Brief description of the drawings]

[0010] [Figure 1] FIG. 1 is a block diagram of a particular illustrative aspect of a system operable to perform audio sample reconstruction using a sample generation network including a neural network and multiple sub-band networks, in accordance with certain examples of the present disclosure. [Diagram 2] 2 is a diagram of an example embodiment of a system operable to perform audio sample reconstruction using the sample generation network of FIG. 1, in accordance with some examples of the present disclosure. [Diagram 3] 2 is a diagram of an example implementation of the sample generation network of FIG. 1 in accordance with some examples of the present disclosure. [Figure 4] 2 is a diagram of another example implementation of the sample generation network of FIG. 1 in accordance with some examples of the present disclosure. [Diagram 5] 2 is a diagram of one implementation of a sub-band network of the sample generation network of FIG. 1 in accordance with some examples of the present disclosure. [Figure 6] 6 is a diagram of an example implementation of a linear prediction (LP) module of the sub-band network of FIG. 5 in accordance with some examples of the present disclosure. [Figure 7] A diagram of an illustrative example of audio subbands corresponding to reconstructed subband audio samples generated by any of the systems of Figures 1-2 in accordance with some examples of the present disclosure. [Figure 8] FIG. 3 is a diagram of additional illustrative examples of audio subbands corresponding to reconstructed subband audio samples generated by any of the systems of FIGS. 1-2 in accordance with some examples of the present disclosure. [Figure 9]FIG. 3 is a diagram of additional illustrative examples of audio subbands corresponding to reconstructed subband audio samples generated by any of the systems of FIGS. 1-2 in accordance with some examples of the present disclosure. [Figure 10] 1 illustrates an example of an integrated circuit operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 11] FIG. 1 illustrates a diagram of a mobile device operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 12] FIG. 1 is a diagram of a headset operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 13] FIG. 1 illustrates a diagram of a wearable electronic device operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 14] FIG. 1 illustrates a diagram of a voice-controlled speaker system operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 15] FIG. 1 is a diagram of a camera operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 16] FIG. 1 illustrates a diagram of a headset, such as a virtual reality headset, a mixed reality headset, or an augmented reality headset, operable to perform audio sample reconstruction using a neural network and multiple subband networks, in accordance with some examples of the present disclosure. [Figure 17]FIG. 1 illustrates a first example of a vehicle operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 18] FIG. 13 is a diagram of a second example of a vehicle operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. [Figure 19] 2 is a diagram of a specific implementation of a method of audio sample reconstruction using a neural network and multiple sub-band networks that may be implemented by the device of FIG. 1 in accordance with some examples of the present disclosure. [Figure 20] FIG. 1 is a block diagram of a particular illustrative example of a device operable to perform audio sample reconstruction using a neural network and multiple sub-band networks, in accordance with some examples of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Audio sample reconstruction using a single neural network tends to have high computational complexity. Systems and methods for audio sample reconstruction using neural networks and multiple sub-band networks are disclosed. For example, the neural network is configured to generate a neural network output based on a neural network input. The sub-band network generates a reconstructed sub-band audio sample based at least in part on the neural network output. For example, a first sub-band network generates a first reconstructed sub-band audio sample associated with a first audio sub-band. A second sub-band network generates a second reconstructed sub-band audio sample associated with a second audio sub-band and based on the first reconstructed sub-band audio sample. The reconstructor generates the reconstructed audio sample by combining the first reconstructed sub-band audio sample, the second reconstructed sub-band audio sample, one or more additional reconstructed sub-band audio samples, or a combination thereof.

[0012] Compared to a single neural network performing all processing on multiple audio subbands, having a neural network perform an initial stage of processing on multiple (e.g., all) audio subbands to generate neural network outputs to be processed by a multiple subband audio network reduces complexity. For example, to generate a reconstructed audio signal having a first sample rate (e.g., 16 kilohertz (kHz), each layer of a single neural network would operate 16,000 times per second to generate 16,000 reconstructed audio samples. A neural network performing an initial stage of processing to generate neural network outputs to two subband audio networks would operate 8,000 times per second to output neural network outputs (e.g., 16,000 samples / two subband networks=8,000 samples). The first subband network would operate 8,000 times per second to process the neural network outputs to generate 8,000 first reconstructed audio samples. The second subband network would operate 8,000 times per second to process the neural network outputs to generate 8,000 first reconstructed audio samples. The reconstructor operates 8000 times per second to process the neural network output to generate 8000 second reconstructed audio samples. The reconstructor outputs 16000 reconstructed audio samples per second (e.g., based on the 8000 first reconstructed audio samples + 8000 second reconstructed audio samples). Operating the neural network 8000 times per second reduces complexity compared to operating the neural network 16000 times per second. Separate sub-band networks, each processing the output of the previous sub-band network, take into account any dependencies between audio sub-bands. The reduced complexity may result in increased processing speed, reduced memory usage, or both, while multiple sub-band networks take into account dependencies between audio sub-bands.

[0013] Certain aspects of the present disclosure are described below with reference to the drawings. In this description, common features are designated by common reference numerals. Various terms used herein are used only for the purpose of describing particular implementations and are not intended to limit the implementations. For example, the singular forms "a," "an," and "the" are intended to include the plural unless the context clearly indicates otherwise. Furthermore, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG. 1 shows a device 102 including one or more processors ("processor(s)" 190 in FIG. 1), indicating that in some implementations, the device 102 includes a single processor 190 and in other implementations, the device 102 includes multiple processors 190. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular unless a description is given of an aspect relating to a plurality of that feature. In some figures, multiple instances of a particular type of feature are used. Although these features are physically and / or logically different, the same reference numbers are used for each, and the different instances are distinguished by the addition of a letter to the reference number.

[0014] As used herein, the terms "comprise", "comprises", and "comprising" may be used interchangeably with "include", "includes", or "including". Additionally, the term "wherein" may be used interchangeably with "where". As used herein, "exemplary" refers to an example, implementation, and / or aspect, and should not be construed as limiting, or as indicating a preferred or preferred implementation. As used herein, orthogonal terms (e.g., "first", "second", "third", etc.) used to modify an element, such as a structure, component, operation, etc., do not in themselves indicate any priority or order of the element with respect to another element, but merely distinguish the element from another element having the same name (apart from the use of orthogonal terms). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to a multiple (e.g., two or more) of a particular element.

[0015] As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," as well as (or alternatively) any combination thereof. Two devices (or components) may be directly or indirectly coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) via one or more other devices, components, wires, buses, networks (e.g., wired networks, wireless networks, or combinations thereof), and the like. Two devices (or components) that are electrically coupled may be included within the same device or within different devices, and may be connected via electronic circuits, one or more connectors, or inductive coupling, as illustrative and non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital or analog signals) directly or indirectly via one or more wires, buses, networks, and the like. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without any intervening components.

[0016] In this disclosure, terms such as "determining," "calculating," "estimating," "shifting," "adjusting," and the like may be used to describe how one or more operations are performed. It should be noted that such terms should not be construed as limiting, and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, "generating," "calculating," "estimating," "using," "selecting," "accessing," and "determining" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or signal) may refer to actively generating, estimating, calculating, or determining a parameter (or signal), or may refer to using, selecting, or accessing a parameter (or signal) that has already been generated, such as by another component or device.

[0017] 1, a particular exemplary embodiment of a system 100 configured to perform audio sample reconstruction is shown. The system 100 includes a neural network 170 and a sub-band network 162 (e.g., a sub-band neural network).

[0018] The device 102 includes one or more processors 190. The sample generation network 160 of the one or more processors 190 includes a synthesizer 154 coupled to a sub-band network 162 via a neural network 170. The sub-band network 162 is coupled to a reconstructor 166. In certain aspects, the sample generation network 160 is included within an audio synthesizer 150.

[0019] In some implementations, the system 100 corresponds to an audio coding system. For example, an audio decoder (e.g., a feedback recurrent autoencoder (FRAE) decoder 140) is coupled to an audio synthesizer 150. For illustration purposes, in a particular aspect, the FRAE decoder 140 is coupled to a sub-band network 162. The one or more processors 190 are coupled to one or more speakers 136. In some implementations, the one or more speakers 136 are external to the device 102. In other implementations, the one or more speakers 136 are incorporated within the device 102.

[0020] The FRAE decoder 140 is configured to generate feature data (FD) 171. For example, the feature data 171 includes linear predictive coefficients (LPCs) 141, pitch gain 173, pitch estimate 175, or a combination thereof. The LPCs 141, pitch gain 173, and pitch estimate 175 are provided as illustrative examples of types of feature data included in the feature data 171. In other examples, the feature data 171 may additionally or alternatively include various other types of feature data, such as a Bark cepstrum, a Bark spectrum, a Mel spectrum, a magnitude spectrum, or a combination thereof. One or more of the types of feature data may be in the linear domain or the log-amplitude domain. The synthesizer 154 is configured to process one or more neural network inputs 151 to generate an embedding 155, as further described with reference to FIG. 3. The neural network 170 is configured to process the embedding 155 to generate a neural network output 161. In some implementations, the neural network 170 includes an autoregressive (AR) generative neural network. For example, the neural network 170 is configured to process embedding based on previous outputs of the sub-band network 162, the reconstructor 166, or both to generate a neural network output 161 that is used by the sub-band network 162 to generate a subsequent output, as further described with reference to FIG. 3. In some aspects, the neural network 170 includes a convolutional neural network (CNN), WaveNet, PixelCNN, a transformer network with an encoder and decoder, Bidirectional Encoder Representations from Transformers (Bert), another type of AR generative neural network, another type of neural network, or a combination thereof.

[0021] The subband network 162 is configured to generate reconstructed subband audio samples 165 based on the neural network output 161, the feature data 171, or both, as will be further described with reference to Figure 4. The reconstructor 166 (e.g., a subband reconstruction filter bank) is configured to generate reconstructed audio samples 167 of a reconstructed audio signal 177 based on the reconstructed subband audio samples 165 generated during one or more iterations by the subband network 162.

[0022] In some implementations, the audio signal 105 is captured by one or more microphones, converted from an analog signal to a digital signal by an analog-to-digital converter, and compressed by an encoder for storage or transmission. In these implementations, the FRAE decoder 140 performs the inverse of the coding algorithm used by the encoder to decode the compressed signal to generate the feature data 171. In other implementations, the audio signal 105 (e.g., a compressed digital signal) is generated by an audio application of one or more processors 190, and the FRAE decoder 140 decodes the compressed digital signal to generate the feature data 171. The audio signal 105 may include a speech signal, a music signal, another type of audio signal, or a combination thereof.

[0023] The FRAE decoder 140 is provided as an illustrative example of an audio decoder. In some examples, the one or more processors 190 may include any type of audio decoder that generates the feature data 171 using a suitable audio coding algorithm, such as a linear predictive coding algorithm (e.g., Code-Excited Linear Prediction (CELP), algebraic CELP (ACELP), or other linear predictive techniques), or another audio coding algorithm.

[0024] The audio signal 105 may be divided into blocks of samples, each referred to as a frame. For example, the audio signal 105 may include a sequence of audio frames, including audio frame (AF) 103A, audio frame 103B, one or more additional audio frames, audio frame 103N, or a combination thereof. In some examples, each of the audio frames 103A-N represents audio corresponding to a play time of 10-20 milliseconds (ms), and each of the audio frames 103A-N includes approximately 160 audio samples.

[0025] In some examples, the reconstructed audio signal 177 corresponds to a reconstruction of the audio signal 105. For example, the reconstructed audio frame (RAF) 153A includes a representative reconstructed audio sample (RAS) 167 corresponding to a reconstruction (e.g., an estimate) representing the audio sample (AS) 107 of the audio frame 103A. The audio synthesizer 150 is configured to generate the reconstructed audio frame 153A based on the reconstructed audio sample 167, one or more additional reconstructed audio samples, or a combination thereof (e.g., approximately 160 reconstructed audio samples including the reconstructed audio sample 167). The reconstructed audio signal 177 includes the reconstructed audio frame 153A as a reconstruction or estimate of the audio frame 103A.

[0026] In some implementations, the device 102 corresponds to or is included within one of a variety of types of devices. In an illustrative example, the one or more processors 190 are incorporated within a headset device as further described with reference to FIG. 12. In other examples, the one or more processors 190 are incorporated within at least one of a mobile phone or tablet computing device as described with reference to FIG. 11, a wearable electronic device as described with reference to FIG. 13, a voice-controlled speaker system as described with reference to FIG. 14, a camera device as described with reference to FIG. 15, or a virtual reality headset, mixed reality headset, or augmented reality headset as described with reference to FIG. 16. In another illustrative example, the one or more processors 190 are incorporated within a vehicle as further described with reference to FIG. 17 and FIG. 18.

[0027] In operation, the FRAE decoder 140 generates feature data 171 representing the audio frame 103A. In some implementations, the FRAE decoder 140 generates at least a portion of the feature data 171 (e.g., one or more of the LPC 141, the pitch gain 173, or the pitch estimate 175) by decoding a corresponding encoded version of the portion of the feature data 171 (e.g., the LPC 141, the pitch gain 173, or the pitch estimate 175). In some implementations, at least a portion of the feature data 171 (e.g., one or more of the LPC 141, the pitch gain 173, or the pitch estimate 175) is estimated independently of a corresponding encoded version of that portion of the feature data 171 (e.g., the LPC 141, the pitch gain 173, or the pitch estimate 175). To illustrate, one or more processor 190 components (e.g., FRAE decoder 140, a digital signal processor (DSP) block, or another component) may estimate a portion of feature data 171 based on an encoded version of another portion of feature data 171. For example, pitch estimate 175 may be estimated based on a speech cepstrum. As another example, LPC 141 may be generated by processing various audio features, such as pitch lag, pitch correlation, pitch gain 173, pitch estimate 175, a Bark frequency cepstrum of the speech signal, or a combination thereof, of audio frame 103A. In certain aspects, FRAE decoder 140 provides a decoded portion of feature data 171 to sub-band network 162. In certain aspects, one or more processor 190 components (e.g., FRAE decoder 140, a DSP block, or another component) provide an estimated portion of feature data 171 to sub-band network 162.

[0028] The sample generation network 160 generates reconstructed audio samples 167, as will be further described with reference to FIG. 4. For example, the synthesizer 154 synthesizes the neural network inputs 151 to generate embeddings 155 that are provided to the neural network 170. The neural network 170 (e.g., a first stage network) processes the embeddings 155 to generate the neural network outputs 161. In certain aspects, using the neural network 170 to generate the neural network outputs 161 to perform an initial stage of processing for multiple (e.g., all) audio subbands reduces complexity. The neural network 170 provides the neural network outputs 161 to the subband network 162.

[0029] The subband networks 162 process the neural network output 161 and the feature data 171 to generate reconstructed subband audio samples 165. For example, as will be further described with reference to Figures 3-4, each of the subband networks 162 generates one of the reconstructed subband audio samples 165 associated with a corresponding audio subband. Each subsequent subband network of the subband networks 162 generates a reconstructed subband audio sample associated with an audio subband of the reconstructed audio samples 167 that is based on the reconstructed subband audio sample generated by the previous subband network of the subband network 162, thus taking into account any dependencies between the audio subbands.

[0030] The reconstructor 166 combines the reconstructed sub-band audio samples 165 generated during one or more iterations by the sub-band network 162 to generate reconstructed audio samples 167. In a particular aspect, the reconstructor 166 includes a sub-band reconstruction filter bank, such as a quadrature mirror filter (QMF), a pseudo-QMF, or a Gabor filter bank. The reconstructor 166 may perform critically sampled or oversampled sub-band processing. Oversampling enables a relationship between transfer ripple and aliasing operating points that cannot be achieved with critical sampling. For example, for a particular transfer ripple specification, a critically sampled filter bank can limit aliasing to a maximum of a particular threshold level, while an oversampled filter bank can further reduce aliasing while maintaining the same transfer ripple specification. Oversampling relieves some of the burden from the subband network 162 in that the subband network 162 attempts to exactly match aliasing components across audio subbands to achieve aliasing cancellation. Even if the aliasing components are not exactly matched and aliasing is not exactly cancelled, the final output quality of the reconstructed audio samples 167 is likely to be acceptable if the aliasing within each subband is relatively low to begin with.

[0031] In a particular aspect, the reconstructed audio samples 167 correspond to a reconstruction of the audio samples 107 of the audio frame 103A of the audio signal 105. The audio synthesizer 150 generates the reconstructed audio frame 153A, which includes at least the reconstructed audio samples 167.

[0032] Similarly, audio synthesizer 150 (e.g., sample generation network 160) generates reconstructed audio frame 153B corresponding to a reconstruction or estimate of audio frame 103B, one or more additional reconstructed audio frames, reconstructed audio frame 153N corresponding to a reconstruction or estimate of audio frame 103N, or a combination thereof. Reconstructed audio signal 177 includes reconstructed audio frame 153A, reconstructed audio frame 153B, one or more additional reconstructed audio frames, reconstructed audio frame 153N, or a combination thereof.

[0033] In some aspects, audio synthesizer 150 outputs reconstructed audio signal 177 via one or more speakers 136. In some examples, device 102 provides reconstructed audio signal 177 to another device, such as a storage device, a user device, a network device, a playback device, or a combination thereof. In some aspects, reconstructed audio signal 177 includes a reconstructed speech signal, a reconstructed music signal, a reconstructed animal sound signal, a reconstructed noise signal, or a combination thereof.

[0034] In some implementations, the subband network 162 provides reconstructed subband audio samples 165, the reconstructor 166 provides reconstructed audio samples 167, or both, to the synthesizer 154 as part of the neural network input 151 for subsequent iterations.

[0035] By having the neural network 170 perform an early stage of processing to generate the neural network output 161, the system 100 reduces complexity, thereby reducing processing time, memory usage, or both. By having each subsequent subband network of the subband network 162 generate reconstructed audio samples associated with a corresponding audio subband that is based on reconstructed audio samples generated by a previous subband network in the subband network 162, the system 100 eliminates dependencies between subbands, thereby reducing discontinuities between the subbands.

[0036] 2, a diagram of an example embodiment of a system 200 operable to perform audio sample reconstruction using a sample generation network 160 is shown. In some embodiments, the system 100 includes one or more components of the system 200.

[0037] The system 200 includes a device 202 configured to communicate with the device 102. The device 202 includes an encoder 204 coupled to a transmitter 208 via a modem 206. The device 102 includes a receiver 238 coupled to a FRAE decoder 140 via a modem 240. The audio synthesizer 150 includes a frame rate network 250 coupled to a sample generation network 160. The FRAE decoder 140 is coupled to the frame rate network 250.

[0038] In some aspects, the encoder 204 of the device 202 uses an audio coding algorithm to process the audio signal 105 of FIG. 1. For example, the audio signal 105 may include a digitized audio signal. In some implementations, the digitized audio signal is generated using a filter to remove aliasing, a sampler to convert to discrete time, and an analog-to-digital converter to convert the analog signal to the digital domain. The resulting digitized audio signal is still a discrete-time audio signal with discretized samples. Using the audio coding algorithm, the encoder 204 may generate a compressed audio signal that represents the audio signal 105 using as few bits as possible while attempting to maintain a constant quality level for the audio. The audio coding algorithm may include a linear predictive coding algorithm (e.g., CELP, ACELP, or other linear predictive techniques) or other voice coding algorithms.

[0039] As an example, the encoder 204 uses an audio coding algorithm to encode the audio frame 103A of the audio signal 105 to generate the compressed audio signal encoded audio data 241. The modem 206 begins transmitting the compressed audio signal (e.g., the encoded audio data 241) via the transmitter 208. The modem 240 of the device 102 receives the compressed audio signal (e.g., the encoded audio data 241) via the receiver 238 and provides the compressed audio signal (e.g., the encoded audio data 241) to the FRAE decoder 140.

[0040] The FRAE decoder 140 decodes the compressed audio signal to extract features representative of the audio signal 105 and provides those features to the audio synthesizer 150 to generate a reconstructed audio signal 177. For example, the FRAE decoder 140 decodes the encoded audio data 241 to generate features 253 representative of the audio frame 103A.

[0041] The features 253 may include any set of features of the audio frame 103A generated by the encoder 204. In some implementations, the features 253 may include quantized features. In other implementations, the features 253 may include dequantized features. In certain aspects, the features 253 include the LPC 141, the pitch gain 173, the pitch estimate 175, a pitch lag with fractional precision, a Bark cepstrum of the speech signal, an 18-band Bark frequency cepstrum, an integer pitch period (or lag) (e.g., between 16 samples and 256 samples), a fractional pitch period (or lag), a pitch correlation (e.g., between 0 and 1), or a combination thereof. In some implementations, features 253 may include features for one or more (e.g., two) audio frames preceding audio frame 103A in a sequence of audio frames, audio frame 103A, one or more (e.g., two) audio frames following audio frame 103A in the sequence of audio frames, or a combination thereof.

[0042] In certain aspects, the features 253 explicitly include at least a portion of the feature data 171 (e.g., the LPC 141, the pitch gain 173, the pitch estimate 175, or a combination thereof), and the FRAE decoder 140 provides at least a portion of the feature data 171 (e.g., the LPC 141, the pitch gain 173, the pitch estimate 175, or a combination thereof) to the sample generation network 160.

[0043] In certain aspects, the features 253 extracted from the encoded audio data 241 do not explicitly include the particular feature (e.g., the LPC 141, the pitch gain 173, the pitch estimate 175, or a combination thereof), and the particular feature is estimated based on other features explicitly included in the features 253. For example, the FRAE decoder 140 provides one or more features explicitly included in the features 253 to another component (e.g., a DSP block) of the one or more processors 190 to generate the particular feature, and the other component provides the particular feature to the sample generation network 160. By way of example, in an implementation in which the features 253 do not explicitly include the LPC 141, but do include a Bark Cepstrum, the LPC 141 may be estimated based on the Bark Cepstrum. To illustrate, the LPC 141 is estimated by converting the 18-band Bark frequency cepstrum to a linear frequency spectral density (e.g., power spectral density (PSD)), using an inverse Fast Fourier Transform (iFFT) to convert the linear frequency spectral density (e.g., PSD) to an autocorrelation, and using the Levinson-Durbin algorithm on the autocorrelation to determine the LPC 141. As another example, in an implementation where the features 253 do not explicitly include a pitch estimate 175, but rather include a speech cepstrum for the audio frame 103A, the pitch estimate 175 may be estimated based on the speech cepstrum.

[0044] In some aspects, the FRAE decoder 140 provides one or more features 243 of the features 253 to the frame rate network 250 to generate the adjustment vector 251. In a particular implementation, the frame rate network 250 includes a convolutional (conv.) layer 270, a convolutional layer 272, a fully connected (FC) layer 276, and a fully connected layer 278. The convolutional layer 270 processes the features 243 to generate an output that is provided to the convolutional layer 272. In some cases, the convolutional layer 270 and the convolutional layer 272 include filters of the same size. For example, the convolutional layer 270 and the convolutional layer 272 may include a filter size of three, resulting in a receptive field of five audio frames (e.g., features of two preceding audio frames, audio frame 103A, and two subsequent audio frames). The output of convolutional layer 272 is added to features 243 and then processed by fully connected layer 276 to generate outputs that are provided as inputs to fully connected layer 278. Fully connected layer 278 processes the inputs to generate adjustment vectors 251.

[0045] The frame rate network 250 provides the adjustment vectors 251 to the sample generation network 160. In one illustrative example, the adjustment vectors 251 are 128-dimensional vectors. In some aspects, the adjustment vectors 251, the feature data 171 (e.g., the LPC 141, the pitch gain 173, the pitch estimate 175, or a combination thereof), or both, may be held constant for the duration of processing each audio frame. The sample generation network 160 generates reconstructed audio samples 167 based on the adjustment vectors 251, the feature data 171, or both, as further described with reference to FIGS. 3-4. In certain aspects, the reconstructed audio frame 153A includes at least the reconstructed audio samples 167.

[0046] In some implementations, the FRAE decoder 140 and the frame rate network 250 are each configured to process data at a frame rate (e.g., once every 10 ms audio frame). In some implementations, the sample generation network 160 processes data at a sample rate (e.g., one reconstructed audio sample is generated per iteration).

[0047] 3, there is shown a diagram of an example implementation of a sample generation network 160. The sample generation network 160 includes a synthesizer 154 coupled to a sub-band network 162 via a neural network 170.

[0048] In certain aspects, neural network 170 is coupled to one or more of sub-band networks 162 via one or more combiners. For example, neural network 170 is coupled to sub-band network 162A, and neural network 170 is coupled to sub-band network 162B via combiner 368A.

[0049] In some aspects, the neural network 170 corresponds to a first stage during which the embeddings 155 representing the neural network inputs 151 are processed using a synthesized network, and the sub-band networks 162 correspond to a second stage during which each set of sub-band network inputs (based on the neural network outputs 161) is processed separately using a respective sub-band network to generate corresponding reconstructed sub-band audio samples.

[0050] Neural network 170 is configured to process embeddings 155 to generate neural network output 161. Neural network 170 includes multiple recurrent layers. The recurrent layers include a gated recurrent unit (GRU), such as GRU 356. In certain aspects, the multiple recurrent layers include a first recurrent layer including GRU 356, a second recurrent layer including GRU 358, one or more additional recurrent layers, or a combination thereof.

[0051] The combiner 154 is coupled to a first recurrent layer (e.g., GRU 356) of the multiple recurrent layers, the GRUs of each previous recurrent layer are coupled to the GRUs of the subsequent recurrent layer, and the GRU of the last recurrent layer (e.g., the second recurrent layer) is coupled to the subband network 162 (e.g., GRU 358).

[0052] As an illustrative example, neural network 170 is provided that includes two recurrent layers. In other examples, neural network 170 may include fewer or more than two recurrent layers. In some implementations, neural network 170 may include one or more additional layers, one or more additional connections, or combinations thereof that are not shown for ease of illustration.

[0053] The synthesizer 154 is configured to process one or more neural network inputs 151 to generate the embedding 155. The one or more neural network inputs 151 include the adjustment vectors 251, the previous sub-band audio samples 311A, the previous sub-band audio samples 311B, the previous audio samples 371, the predicted audio data 353, or a combination thereof.

[0054] In certain aspects, the previous sub-band audio samples 311A ​​are generated by the sub-band network 162A during a previous iteration. In certain aspects, the previous sub-band audio samples 311B are generated by the sub-band network 162B during a previous iteration. In certain aspects, the predicted audio data 353 includes predicted audio data generated by the LP module of the sub-band network 162A during one or more previous iterations, predicted audio data generated by the LP module of the sub-band network 162B during one or more previous iterations, or both.

[0055] The multiple recurrent layers of the neural network 170 are configured to process the embedding 155. In some implementations, the GRU 356 determines a first hidden state based on a previous first hidden state and the embedding 155. The previous first hidden state is generated by the GRU 356 during a previous iteration. The GRU 356 outputs the first hidden state to the GRU 358. The GRU 358 determines a second hidden state based on the first hidden state and a previous second hidden state. The previous second hidden state is generated by the GRU 358 during a previous iteration. Each previous GRU outputs a hidden state to a subsequent GRU of the multiple recurrent layers, which generates a hidden state based on the received hidden state and the previous hidden state. The neural network output 161 is based on the hidden state of the GRU (e.g., the GRU 358) of the last recurrent layer. Neural network 170 outputs neural network output 161 to subband network 162A and to combiner 368A.

[0056] In some examples, one or more neural network inputs 151 may be mu-law encoded and embedded using a network embedding layer of the synthesizer 154 to generate embeddings 155. For example, embeddings 155 may map each mu-law level to a vector (e.g., in an embedding matrix) and essentially learn a set of nonlinear functions to be applied to the mu-law values. The embedding matrix (e.g., embeddings 155) may be sent to one or more of multiple recurrent layers (e.g., GRU 356, GRU 358, or a combination thereof). For example, the embedding matrix (e.g., embeddings 155) may be input to GRU 356 and the output of GRU 356 may be input to GRU 358. In another example, the embedding matrix (e.g., embeddings 155) may be input to GRU 356, GRU 358, or both separately.

[0057] In some aspects, a product of an embedding matrix being input to the GRU may be calculated with a corresponding sub-matrix of the GRU's non-recurrent weights. The transformation may be applied to all of the GRU's gates (e.g., update gates (u), reset gates (r), and hidden states (h)) and embedded inputs (e.g., one or more neural network inputs 151). In some cases, one or more of the one or more neural network inputs 151, such as adjustment vectors 251, may not be embedded. Using the previous sub-band audio sample 311A ​​as an example of an embedded input, E may denote the embedding matrix, and U may denote the embedding matrix. (u,s) is a sequence U that contains the columns applied to the embedding of the previous subband audio samples 311A. (n) , and a new embedding matrix V that maps the previous subband audio sample 311A ​​directly to the non-recursive terms of the update gate calculation. (u,s) =U (u,s) E can be derived.

[0058] The output from GRU 358, or the outputs from GRU 356 and GRU 358, when an embedding matrix (e.g., embedding 155) is input separately to GRU 356 and to GRU 358, is provided as neural network output 161 to sub-band network 162 and to combiner 368A. For example, neural network 170 provides neural network output 161 as one or more sub-band neural network inputs 361A to sub-band network 162A and to combiner 368A.

[0059] Each of the subband networks 162 generates reconstructed subband audio samples of a reconstructed subband audio signal of the reconstructed audio signal 177. For illustration purposes, a first reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a first audio subband and a second reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a second audio subband. As will be further explained with reference to Figures 7-9, the first audio subband is associated with a first frequency range and the second audio subband is associated with a second frequency range.

[0060] For example, the sub-band network 162A processes one or more sub-band neural network inputs 361A based at least in part on the feature data 171 to generate reconstructed sub-band audio samples 165A of a first reconstructed sub-band audio signal of the reconstructed audio signal 177. For example, as will be further described with reference to Figures 5 and 6, the sub-band network 162A generates the reconstructed sub-band audio samples 165A based on the feature data 171, the previous sub-band audio samples 311A, the previous audio samples 371, the predicted audio data (e.g., at least a portion of the predicted audio data 353), or a combination thereof.

[0061] The synthesizer 368A synthesizes the one or more sub-band neural network inputs 361A and the reconstructed sub-band audio samples 165A to generate one or more sub-band neural network inputs 361B. The sub-band network 162B processes the one or more sub-band neural network inputs 361B based at least in part on the feature data 171 to generate the reconstructed sub-band audio samples 165B of a second reconstructed sub-band audio signal of the reconstructed audio signal 177. For example, as further described with reference to Figures 5 and 6, the sub-band network 162B generates the reconstructed sub-band audio samples 165B based on the feature data 171, the previous sub-band audio samples 311A, the previous sub-band audio samples 311B, the previous audio samples 371, the predicted audio data (e.g., at least a portion of the predicted audio data 353), the reconstructed sub-band audio samples 165A, or a combination thereof.

[0062] As an illustrative example, sub-band network 162 including two sub-band networks is provided. In other examples, sub-band network 162 includes more than two sub-band networks (i.e., a particular number of sub-band networks greater than two, such as a four sub-band network).

[0063] The reconstructor 166 combines the reconstructed sub-band audio samples generated by the sub-band network 162 during one or more iterations to generate the reconstructed audio sample 167. For example, the reconstructor 166 combines the reconstructed sub-band audio sample 165A, the reconstructed sub-band audio sample 165B, one or more additional sub-band audio samples, or a combination thereof, to generate the reconstructed audio sample 167.

[0064] In a particular implementation, the reconstructor 166 synthesizes one or more subband audio samples (e.g., the previous subband audio sample 311A, the previous subband audio sample 311B, one or more additional subband audio samples, or a combination thereof) generated in a previous iteration to generate a previous reconstructed audio sample. In another particular implementation, the reconstructor 166 synthesizes one or more subband audio samples (e.g., the previous subband audio sample 311A, the previous subband audio sample 311B, or both) generated in a previous iteration, one or more subband audio samples (e.g., the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, or both), one or more additional subband audio samples, or a combination thereof generated in a current iteration to generate a reconstructed audio sample 167.

[0065] In certain aspects, the sub-band network 162, the reconstructor 166, or both generate at least a portion of the one or more neural network inputs 151 for a subsequent iteration. For example, the sub-band network 162A provides the reconstructed sub-band audio sample 165A as the previous sub-band audio sample 311A ​​for the subsequent iteration. As another example, the sub-band network 162B provides the reconstructed sub-band audio sample 165B as the previous sub-band audio sample 311B for the subsequent iteration. In certain aspects, the reconstructor 166 provides the reconstructed audio sample 167 as the previous audio sample 371 for the subsequent iteration. In certain implementations, the sub-band network 162A provides at least a first portion of the predicted audio data 353 for the subsequent iteration. In certain implementations, the sub-band network 162B provides at least a second portion of the predicted audio data 353 for the subsequent iteration.

[0066] Sub-band network 162A and sub-band network 162B are illustrated as separate modules for ease of explanation. In another example, the same sub-band network generates reconstructed sub-band audio samples 165A followed by reconstructed sub-band audio samples 165B.

[0067] In some examples, the reconstructor 166 is configured to generate a plurality of reconstructed audio samples of the reconstructed audio signal 177 for each inference of the neural network 170. For example, the reconstructor 166 may generate a plurality of reconstructed audio samples from the reconstructed sub-band audio sample 165A, the reconstructed sub-band audio sample 165B, one or more additional reconstructed audio samples, or a combination thereof. In an illustrative example, the reconstructor 166 includes a critically sampled two-band filter bank. The audio signal 105 (e.g., s[n]) has a first sample rate (e.g., 16 kHz) and is encoded as a first sub-band audio signal (e.g., s_L[n]) and a second sub-band audio signal (e.g., s_H[n]).

[0068] In a particular aspect, the first sub-band audio signal (e.g., s_L[n]) corresponds to a first audio sub-band including a first frequency range. The second sub-band audio signal (e.g., s_H[n]) corresponds to a second audio band including a second frequency range different from the first frequency range. As an example, the first frequency range is from a first start frequency to a first end frequency, and the second frequency range is from a second start frequency to a second end frequency. In a particular example, the second start frequency is adjacent to and follows the first end frequency. Each of the first sub-band audio signal (e.g., s_L[n]) and the second sub-band audio signal (e.g., s_H[n]) has a second sample rate (e.g., 8 kHz) that is half the first sample rate (e.g., 16 kHz).

[0069] The reconstructor 166 generates a first reconstructed subband audio signal (e.g., including reconstructed subband audio samples 165A) and a second reconstructed audio signal (e.g., including reconstructed subband audio samples 165B) that represent reconstructed versions of the first subband audio signal and the second subband audio signal, respectively.

[0070] The reconstructor 166 upsamples and filters each of the first reconstructed subband audio signal and the second reconstructed audio signal, and adds the resulting upsampled and filtered signals to generate a reconstructed audio signal 177 having twice the sample rate of the first reconstructed subband audio signal and the second reconstructed audio signal. Thus, a frame of N reconstructed samples of the first reconstructed subband audio signal (e.g., s_L) and a corresponding frame of N reconstructed samples of the second reconstructed subband audio signal (e.g., s_H) input to the reconstructor 166 results in an output of 2N reconstructed samples of the reconstructed audio signal 177. Thus, the reconstructor 166 can generate a plurality of reconstructed audio samples (e.g., two reconstructed audio samples) based on the reconstructed subband audio samples 165A and the reconstructed subband audio samples 165B in each iteration.

[0071] In some implementations, during a first processing stage of the iteration, the sub-band network 162A generates reconstructed sub-band audio samples 165A that are used by the reconstructor 166 in generating the two reconstructed audio samples. During a second processing stage of the iteration, the sub-band network 162B generates reconstructed sub-band audio samples 165B that are also used by the reconstructor 166 in generating the two reconstructed audio samples. In some aspects, the sub-band network 162B is idle during the first processing stage, and the sub-band network 162A is idle during the second processing stage. Each of the sub-band network 162A and the sub-band network 162B operates at a sample rate (e.g., 8 kHz) that is half the first sample rate (e.g., 16 kHz) of the reconstructed audio signal 177. For example, each of the sub-band network 162A and the sub-band network 162B generates data used to generate two reconstructed audio samples for every two processing stages.

[0072] 4, a diagram of an example implementation of a sample generation network 160 is shown. In a particular aspect, each of the sub-band networks 162 is configured to generate reconstructed sub-band audio samples based at least in part on the neural network output 161.

[0073] Sub-band network 162 includes sub-band network 162A, sub-band network 162B, sub-band network 162C, and sub-band network 162D. Neural network 170 is coupled to combiner 368A, combiner 368B, and combiner 368C. Combiner 368A is coupled to sub-band network 162A and to sub-band network 162B. Combiner 368B is coupled to sub-band network 162B and to sub-band network 162C. Combiner 368C is coupled to sub-band network 162C and to sub-band network 162D. Neural network 170 provides neural network output 161 to each of combiner 368A, combiner 368B, and combiner 368C.

[0074] The subband networks 162 are implemented in a substantially similar manner as described with reference to FIG. 3. Each of the subband networks 162 generates reconstructed subband audio samples of a reconstructed subband audio signal of the reconstructed audio signal 177. For illustration purposes, a first reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a first audio subband, a second reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a second audio subband, a third reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a third audio subband, a fourth reconstructed subband audio signal of the reconstructed audio signal 177 corresponds to at least a fourth audio subband, etc. As will be further described with reference to FIGS. 8-9, the first audio subband is associated with a first frequency range, the second audio subband is associated with a second frequency range, the third audio subband is associated with a third frequency range, and the fourth audio subband is associated with a fourth frequency range.

[0075] For example, subband network 162A generates reconstructed subband audio samples 165A of a first reconstructed subband audio signal of reconstructed audio signal 177. For example, as further described with reference to Figures 5 and 6, subband network 162A generates reconstructed subband audio samples 165A based on feature data 171, previous subband audio samples 311A, previous audio samples 371, predicted audio data (e.g., at least a portion of predicted audio data 353), or a combination thereof.

[0076] The synthesizer 368A synthesizes the one or more sub-band neural network inputs 361A and the reconstructed sub-band audio samples 165A to generate one or more sub-band neural network inputs 361B. The sub-band network 162B generates the reconstructed sub-band audio samples 165B of a second reconstructed sub-band audio signal of the reconstructed audio signal 177. For example, as further described with reference to Figures 5 and 6, the sub-band network 162B generates the reconstructed sub-band audio samples 165B based on the feature data 171, the previous sub-band audio samples 311A, the previous sub-band audio samples 311B, the previous audio samples 371, the predicted audio data (e.g., at least a portion of the predicted audio data 353), the reconstructed sub-band audio samples 165A, or a combination thereof.

[0077] The sub-band network 162B provides reconstructed sub-band audio samples 165B to a combiner 368B, which combines the neural network output 161 and the reconstructed sub-band audio samples 165B to generate one or more sub-band neural network inputs 361C.

[0078] The subband network 162C processes the one or more subband neural network inputs 361C based at least in part on the feature data 171 to generate reconstructed subband audio samples 165C of a third reconstructed subband audio signal of the reconstructed audio signal 177. For example, as will be further described with reference to Figures 5 and 6, the subband network 162C generates the reconstructed subband audio samples 165C based on the feature data 171, the previous subband audio samples 311A, the previous subband audio samples 311B, the previous subband audio samples generated by the subband network 162C during a previous iteration, the predicted audio data (e.g., at least a portion of the predicted audio data 353), the reconstructed subband audio samples 165A, the reconstructed subband audio samples 165B, or a combination thereof.

[0079] The sub-band network 162C provides reconstructed sub-band audio samples 165C to a combiner 368C, which combines the neural network output 161 and the reconstructed sub-band audio samples 165C to generate one or more sub-band neural network inputs 361D.

[0080] The subband network 162D processes one or more subband neural network inputs 361D based at least in part on the feature data 171 to generate reconstructed subband audio samples 165D of a fourth reconstructed subband audio signal of the reconstructed audio signal 177. For example, as will be further described with reference to Figures 5 and 6, the subband network 162D generates the reconstructed subband audio samples 165D based on the feature data 171, the previous subband audio samples 311A, the previous subband audio samples 311B, the previous subband audio samples generated by the subband network 162C during a previous iteration, the previous subband audio samples 311D generated by the subband network 162D during a previous iteration, the predicted audio data (e.g., at least a portion of the predicted audio data 353), the reconstructed subband audio samples 165A, the reconstructed subband audio samples 165B, the reconstructed subband audio samples 165C, or a combination thereof.

[0081] The reconstructor 166 synthesizes the reconstructed subband audio samples generated by the subband network 162 during one or more iterations to generate a reconstructed audio sample 167. For example, the reconstructor 166 generates the reconstructed audio sample 167 by synthesizing the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, the reconstructed subband audio sample 165C, the reconstructed subband audio sample 165D, one or more additional reconstructed subband audio samples, or a combination thereof.

[0082] In certain aspects, the sub-band network 162, the reconstructor 166, or both generate at least a portion of the one or more neural network inputs 151 for a subsequent iteration. For example, each of the sub-band networks 162 provides a reconstructed sub-band audio sample as a previous sub-band audio sample for a subsequent iteration. In certain aspects, the reconstructor 166 provides a reconstructed audio sample 167 as a previous audio sample 371 for a subsequent iteration. In certain implementations, each of the sub-band networks 162 provides at least a portion of the predicted audio data 353 for a subsequent iteration.

[0083] In the example shown in FIG. 4, the previous subband audio sample 311A ​​from subband network 162A and the previous subband audio sample 311D from subband network 162D are provided to the synthesizer 154 as part of the neural network input 151, although it will be appreciated that in another example, the previous subband audio sample 311B from subband network 162B and the previous subband audio sample from subband network 162C are also provided as part of the neural network input 151.

[0084] Sub-band network 162A, sub-band network 162B, sub-band network 162C, and sub-band network 162D are described as separate modules for ease of explanation. In certain aspects, the same sub-band network generates multiple reconstructed audio samples one after the other. For illustrative purposes, in some examples, the same sub-band network generates reconstructed sub-band audio sample 165A followed by reconstructed sub-band audio sample 165B. In some examples, the same sub-band network generates reconstructed sub-band audio sample 165B followed by reconstructed sub-band audio sample 165C. In some examples, the same sub-band network generates reconstructed sub-band audio sample 165C followed by reconstructed sub-band audio sample 165D.

[0085] In some examples, the reconstructor 166 may generate a plurality of reconstructed audio samples from the reconstructed sub-band audio sample 165A, the reconstructed sub-band audio sample 165B, the reconstructed sub-band audio sample 165C, the reconstructed sub-band audio sample 165D, one or more additional reconstructed audio samples, or a combination thereof. In an illustrative example, the reconstructor 166 includes a critically sampled four-band filter bank. The audio signal 105 (e.g., s[n]) has a first sample rate (e.g., 16 kilohertz (kHz)) and is encoded as a first sub-band audio signal, a second sub-band audio signal, a third sub-band audio signal, and a fourth sub-band audio signal. In a particular aspect, the four sub-band audio signals are contiguous (e.g., adjacent and non-overlapping), and each of the four sub-band audio signals has a second sample rate (e.g., 4 kHz) that is a quarter of the first sample rate (e.g., 16 kHz). The reconstructor 166 processes a first reconstructed subband audio signal (e.g., including reconstructed subband audio samples 165A) from subband network 162A, a second reconstructed audio signal (e.g., including reconstructed subband audio samples 165B) from subband network 162B, a third reconstructed subband audio signal (e.g., including reconstructed subband audio samples 165C) from subband network 162C, and a fourth reconstructed audio signal (e.g., including reconstructed subband audio samples 165D) from subband network 162D, which represent reconstructed versions of the first subband audio signal, the second subband audio signal, the third subband audio signal, and the fourth subband audio signal, respectively.

[0086] The reconstructor 166 upsamples and filters each of the first reconstructed subband audio signal, the second reconstructed audio signal, the third reconstructed audio signal, and the fourth reconstructed audio signal, and sums the resulting upsampled and filtered signals to generate a reconstructed audio signal 177 having a sample rate that is four times that of the first reconstructed subband audio signal, the second reconstructed audio signal, the third reconstructed subband audio signal, and the fourth reconstructed audio signal. Thus, a frame of N reconstructed samples of the first reconstructed subband audio signal, a corresponding frame of N reconstructed samples of the second reconstructed subband audio signal, a corresponding frame of N reconstructed samples of the third reconstructed subband audio signal, and a corresponding frame of N reconstructed samples of the fourth reconstructed subband audio signal input to the reconstructor 166 results in an output of 4N reconstructed samples of the reconstructed audio signal 177. Thus, in each iteration, the reconstructor 166 may generate multiple reconstructed audio samples (e.g., four reconstructed audio samples) based on the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, the reconstructed subband audio sample 165C, and the reconstructed subband audio sample 165D.

[0087] Each of sub-band network 162A, sub-band network 162B, sub-band network 162C, and sub-band network 162D operates at a sample rate (e.g., 4 kHz) that is one-quarter of the first sample rate (e.g., 16 kHz) of reconstructed audio signal 177. For example, each of sub-band network 162A, sub-band network 162B, sub-band network 162C, and sub-band network 162D generates data that is used to generate four reconstructed audio samples for each of four processing stages.

[0088] 5, a diagram of an example implementation of a sub-band network 162 of a sample generation network 160 is shown. In a particular aspect, the sub-band network 162 represents one or more example implementations of a sub-band network 162A, a sub-band network 162B, a sub-band network 162C, or a sub-band network 162D.

[0089] The sub-band network 162 includes a neural network 562 coupled to a linear prediction (LP) module 564. The neural network 562 includes one or more recurrent layers, feedforward layers, softmax layers 556, or combinations thereof. The recurrent layers include GRUs, such as the GRU 552. The feedforward layers include fully connected layers, such as the FC layer 554.

[0090] As an illustrative example, a neural network 562 including one recurrent layer is provided. In other examples, the neural network 562 may include multiple recurrent layers. The GRUs of each previous recurrent layer of the multiple recurrent layers are coupled to the GRUs of the subsequent recurrent layer. The GRUs 552 of the last recurrent layer of the one or more recurrent layers are coupled to an FC layer 554. The FC layer 554 is coupled to a softmax layer 556. In some implementations, the neural network 562 may include one or more additional layers, one or more additional connections, or combinations thereof, not shown for ease of explanation.

[0091] The one or more recurrent layers are configured to process one or more sub-band neural network inputs 361. In some implementations, a GRU (e.g., GRU 552) of a first recurrent layer of the one or more recurrent layers determines a first hidden state based on a previous first hidden state and the one or more sub-band neural network inputs 361. The previous first hidden state is generated by the GRU (e.g., GRU 552) of the first recurrent layer during a previous iteration.

[0092] In some implementations, neural network 562 includes multiple recurrent layers, where the GRUs of each previous recurrent layer output a hidden state to a GRU of a subsequent recurrent layer of the multiple recurrent layers, which generates a hidden state based on the received hidden state and the previous hidden state.

[0093] The GRU 552 of the last recurrent layer of the one or more recurrent layers outputs the first hidden state to the FC layer 554. The FC layer 554 is configured to process the output of the one or more recurrent layers. For example, the FC layer 554 includes a dual FC layer. The outputs of the two fully connected layers of the FC layer 554 are combined with an element-wise weighted sum to generate an output. The output of the FC layer 554 is provided to a softmax layer 556 to generate a probability distribution 557. In certain aspects, the probability distribution 557 indicates the probability of various values ​​of the residual data 563.

[0094] In some implementations, one or more recurrent layers receive the embeddings 155 (in addition to the neural network output 161) as one or more subband neural network inputs 361. The output of the GRU 552, or the output of the GRUs of the multiple recurrent layers, is provided to an FC layer 554. In some examples, the FC layer 554 (e.g., a dual FC layer) may include two fully connected layers combined with an element-wise weighted sum. Using a combined fully connected layer may allow for computing the probability distribution 557 without significantly increasing the size of the preceding layers. In one illustrative example, the FC layer 554 is a multi-layered FC layer that computes the probability distribution 557 by using the following formula: dual_fc(x)=a 1 tanh(W 1 x)+a 2 tanh(W 2 x), where W 1 and W 2 is the weight matrix, and a 1 and a 2 is a weighting vector, and tanh is the hyperbolic tangent function that produces values ​​between -1 and 1.

[0095] In some implementations, the output of the FC layer 554 is used with a softmax activation of a softmax layer 556 to compute a probability distribution 557 that represents the probability of possible excitation values ​​for the residual data 563. The residual data 563 may be quantized (e.g., 8-bit mu-law quantization). The 8-bit quantized value is calculated by multiplying the count of possible values ​​(e.g., 2 8 The probability distribution 557 indicates a probability associated with each of the possible values ​​(e.g., 256 values) of the residual data 563. In some implementations, the output of the FC layer 554 indicates a mean and a covariance matrix that corresponds to a probability distribution 557 (e.g., a normal distribution) of the values ​​of the residual data 563. In these implementations, the values ​​of the residual data 563 may correspond to real values ​​(e.g., dequantized values).

[0096] Neural network 562 performs sampling 558 based on probability distribution 557 to generate residual data 563. For example, neural network 562 selects particular values ​​for residual data 563 based on the probabilities indicated by probability distribution 557. Neural network 562 provides residual data 563 to LP module 564.

[0097] The LP module 564 generates the reconstructed sub-band audio samples 165 based on the residual data 563. For example, the LP module 564 generates the reconstructed sub-band audio samples 165 of the reconstructed audio signal 177 based on the residual data 563, the feature data 171, the predicted audio data 559, the previous audio samples 371, one or more reconstructed sub-band audio samples 565, or a combination thereof, as will be further described with reference to Fig. 6. In a particular aspect, the predicted audio data 559 corresponds to a portion of the predicted audio data 353 generated by the LP module 564 during a previous iteration, as will be further described with reference to Fig. 6.

[0098] In a particular aspect, the sub-band network 162 represents an example implementation of the sub-band network 162A, the sub-band network 162B, the sub-band network 162C, or the sub-band network 162D. In this aspect, the one or more sub-band neural network inputs 361 represent sub-band neural network inputs to the represented sub-band network, and the reconstructed sub-band audio samples 165 represent reconstructed sub-band audio samples output by the represented sub-band network. For example, in a particular aspect, the sub-band network 162 represents an example implementation of the sub-band network 162A. In this aspect, the one or more sub-band neural network inputs 361 represent one or more sub-band neural network inputs 361A, and the reconstructed sub-band audio samples 165 represent reconstructed sub-band audio samples 165A.

[0099] In a particular aspect, the sub-band network 162 represents an exemplary implementation of the sub-band network 162B. In this aspect, the one or more sub-band neural network inputs 361 represent one or more sub-band neural network inputs 361B, and the reconstructed sub-band audio samples 165 represent reconstructed sub-band audio samples 165B. Similarly, in a particular aspect, the sub-band network 162 represents an exemplary implementation of the sub-band network 162C. In this aspect, the one or more sub-band neural network inputs 361 represent one or more sub-band neural network inputs 361C, and the reconstructed sub-band audio samples 165 represent reconstructed sub-band audio samples 165C. In a particular aspect, the sub-band network 162 represents an exemplary implementation of the sub-band network 162D. In this aspect, the one or more sub-band neural network inputs 361 represent one or more sub-band neural network inputs 361D, and the reconstructed sub-band audio samples 165 represent reconstructed sub-band audio samples 165D.

[0100] As an illustrative example, each sub-band network 162 is provided that includes an LP module 564. In some implementations, each of the sub-band networks 162 (e.g., sub-band network 162A, sub-band network 162B, sub-band network 162C, sub-band network 162D, or a combination thereof) provides residual data to the reconstructor 166 of FIG. 1, which processes the residual data to generate reconstructed residual data and provides the reconstructed residual data to the LP module. The LP module generates reconstructed audio samples 167 based on the reconstructed residual data.

[0101] In a particular aspect, the reconstructor 166 receives the first residual data 563 from the sub-band network 162A and the second residual data 563 from the sub-band network 162B, and processes the first residual data and the second residual data to generate reconstructed residual data. The LP module processes the reconstructed residual data based on the LPC 141 and the feature data 171 to generate reconstructed audio samples 167.

[0102] In another particular aspect, the reconstructor 166 receives the first residual data 563 from the sub-band network 162A, the second residual data 563 from the sub-band network 162B, the third residual data 563 from the sub-band network 162C, and the fourth residual data 563 from the sub-band network 162D. The reconstructor 166 processes the first residual data, the second residual data, the third residual data, and the fourth residual data to generate reconstructed residual data. The LP module processes the reconstructed residual data based on the LPC 141 and the feature data 171 to generate reconstructed audio samples 167.

[0103] 6, a diagram 600 of an example implementation of the LP module 564 is shown. The LP module 564 includes a long-term prediction (LTP) engine 610 coupled to a short-term LP engine 630. The LTP engine 610 includes an LTP filter 612, and the short-term LP engine 630 includes a short-term LP filter 632.

[0104] In a particular aspect, the residual data 563 corresponds to an excitation signal, and the predicted audio data 657 and the predicted audio data 659 correspond to predictions, and the LP module 564 is configured to combine the excitation signal (e.g., the residual data 563) with the predictions (e.g., the predicted audio data 657 and the predicted audio data 659) to generate the reconstructed sub-band audio samples 165. For example, the LTP engine 610 combines the predicted audio data 657 with the residual data 563 to generate the synthesized residual data 611 (e.g., LP residual data). The short-term LP engine 630 combines the synthesized residual data 611 with the predicted audio data 659 to generate the reconstructed sub-band audio samples 165. In a particular aspect, the predicted audio data 559 of FIG. 5 includes the predicted audio data 657 and the predicted audio data 659.

[0105] In some implementations, the LTP engine 610 combines the predictive audio data 657 with residual data associated with another audio sample to generate the synthesized residual data 611. For example, the LTP engine 610 combines the predictive audio data 657 with the residual data 563 and residual data 663 associated with one or more other sub-band audio samples to generate the synthesized residual data 611. In a particular implementation, the residual data 563 is generated by a neural network 562 of one of the sub-band networks 162, and the residual data 663 is generated by a neural network 562 of another of the sub-band networks 162. For example, residual data 563 is generated by neural network 562 of sub-band network 162A, and residual data 663 includes first residual data generated by neural network 562 of sub-band network 162B, second residual data generated by neural network 562 of sub-band network 162C, third residual data generated by neural network 562 of sub-band network 162D, or a combination thereof.

[0106] The LP module 564 is configured to generate a prediction for a subsequent iteration. For example, the LTP filter 612 generates next predicted audio data 667 (e.g., next long-term predicted data) based on the synthesis residual data 611, the pitch gain 173, the pitch estimate 175, or a combination thereof. In certain aspects, the next predicted audio data 667 is used as the predicted audio data 657 in the subsequent iteration.

[0107] The short-term LP filter 632 generates next predicted audio data 669 (e.g., next short-term predicted data) based on the reconstructed sub-band audio sample 165, the LPC 141, the previous audio sample 371, one or more reconstructed sub-band audio samples 665 received from the LP module of the other sub-band network, or a combination thereof. For example, the short-term LP filter 632 of the sub-band network 162A generates next predicted audio data 669 (e.g., next short-term predicted data) based on the reconstructed sub-band audio sample 165A, the LPC 141, the previous audio sample 371, or a combination thereof. In this example, the short-term LP filter 632 does not receive any reconstructed sub-band audio sample 665 from the LP module of the other sub-band network, and the one or more reconstructed sub-band audio samples 565 of FIG. 5 include the reconstructed sub-band audio sample 165A.

[0108] In another example, the short-term LP filter 632 of the subband network 162B generates next predicted audio data 669 (e.g., next short-term predicted data) based on the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, the LPC 141, the previous audio sample 371, or a combination thereof received from the subband network 162A. In this example, the one or more reconstructed subband audio samples 665 include the reconstructed subband audio sample 165A, and the one or more reconstructed subband audio samples 565 of FIG. 5 include the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, or both.

[0109] As a further example, the short-term LP filter 632 of the subband network 162C generates next predicted audio data 669 (e.g., next short-term predicted data) based on the reconstructed subband audio sample 165A received from the subband network 162A, the reconstructed subband audio sample 165B received from the subband network 162B, the reconstructed subband audio sample 165C, the LPC 141, the previous audio sample 371, or a combination thereof. In this example, the one or more reconstructed subband audio samples 665 include the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, or both, and the one or more reconstructed subband audio samples 565 of FIG. 5 include the reconstructed subband audio sample 165A, the reconstructed subband audio sample 165B, the reconstructed subband audio sample 165C, or a combination thereof.

[0110] In certain aspects, the next predicted audio data 669 is used as the predicted audio data 659 in the subsequent iteration. In certain aspects, the LP module 564 outputs the next predicted audio data 667, the next predicted audio data 669, or both, as part of the predicted audio data 353 for the subsequent iteration.

[0111] In certain aspects, the LP module 564 outputs the reconstructed subband audio sample 165 as the previous subband audio sample (e.g., the previous subband audio sample 311A, the previous subband audio sample 311B, the previous subband audio sample generated by the subband network 162C during the previous iteration, or the previous subband audio sample 311D) in the neural network input 151 for the subsequent iteration. In certain aspects, the LP module 564 outputs the residual data 563, the synthesized residual data 611, or both as additional previous subband sample data in the neural network input 151 for the subsequent iteration.

[0112] In some implementations, the LPC 141 includes different LPCs associated with different audio subbands. For example, the LPC 141 includes a first LPC associated with a first audio subband and a second LPC associated with a second audio subband, where the second LPC is different from the first LPC. In these implementations, the short-term LP filter 632 of the subband network 162A generates the next predicted audio data 669 (e.g., next short-term predicted data) based on the first LPC of the LPC 141, the reconstructed subband audio samples 165A, the previous audio samples 371, or a combination thereof. The short-term LP filter 632 of the subband network 162B generates the next predicted audio data 669 (e.g., next short-term predicted data) based on the second LPC of the LPC 141, the reconstructed subband audio samples 165A, the reconstructed subband audio samples 165B, the previous audio samples 371 received from the subband network 162A, or a combination thereof.

[0113] Diagram 600 provides an illustrative, non-limiting example of one implementation of the LP module 564 of the sub-band network 162 of FIG. 5. In other examples, the LP module 564 of the sub-band network 162 may have various other implementations. For example, in a particular implementation, the residual data 563 is processed by the short-term LP engine 630 prior to processing of the output of the short-term LP engine 630 by the LTP engine 610. In this implementation, the output of the LTP engine 610 corresponds to the reconstructed sub-band audio samples 165. In some implementations, the LP module 564 includes the short-term LP engine 630 and does not include the LTP engine 610. For example, the residual data 563 is provided to the short-term LP engine 630, which generates the reconstructed sub-band audio samples 165 based on the residual data 563 and the predicted audio data 659, independent of (e.g., without generating) the synthesized residual data 611.

[0114] 7, there is shown a diagram 700 of illustrative examples of audio sub-bands corresponding to reconstructed sub-band audio samples 165. In a particular aspect, the reconstructed sub-band audio samples 165 are generated by the sample generation network 160 of FIG.

[0115] In certain aspects, the reconstructed subband audio samples 165A of FIGS. 3-4 represent audio of audio subband 711A. Audio subband 711A includes a first range of frequencies (e.g., a first frequency range) from frequency 715A to frequency 715B, where frequency 715B is greater (e.g., higher) than frequency 715A. In certain aspects, the reconstructed subband audio samples 165B represent audio of audio subband 711B. Audio subband 711B includes a second range of frequencies (e.g., a second frequency range) from frequency 715C to frequency 715D, where frequency 715D is greater (e.g., higher) than frequency 715C.

[0116] In example 702, the first frequency range of audio subband 711A and the second frequency range of audio subband 711B are non-overlapping and non-contiguous. For illustration purposes, frequency 715C is higher than frequency 715B.

[0117] In example 704, the first frequency range of audio subband 711A and the second frequency range of audio subband 711B are non-overlapping and contiguous. For purposes of illustration, frequency 715C is equal to frequency 715B.

[0118] In example 706, a first frequency range of audio subband 711A at least partially overlaps with a second frequency range of audio subband 711B. For illustration purposes, frequency 715C is greater (e.g., higher) than frequency 715A and less than (e.g., lower) than frequency 715B.

[0119] As an illustrative example, reconstructed subband audio samples 165A and reconstructed subband audio samples 165B are provided that represent audio subband 711A and audio subband 711B, respectively. In other examples, reconstructed subband audio samples 165A and reconstructed subband audio samples 165B may represent audio subband 711B and audio subband 711A, respectively.

[0120] The first frequency range of audio subband 711A has a first width corresponding to the difference between frequencies 715A and 715B. The second frequency range of audio subband 711B has a second width corresponding to the difference between frequencies 715C and 715D. In some examples, the first frequency range of audio subband 711A has the same width as the second frequency range of audio subband 711B. For example, the first width is equal to the second width. For illustration purposes, the difference between frequencies 715A and 715B is the same as the difference between frequencies 715C and 715D.

[0121] In some examples, the first frequency range of audio subband 711A is wider than the second frequency range of audio subband 711B. For example, the first width is greater than the second width. For illustration purposes, the difference between frequency 715A and frequency 715B is greater than the difference between frequency 715C and frequency 715D. In some examples, the first frequency range of audio subband 711A is narrower than the second frequency range of audio subband 711B. For example, the first width is less than the second width. For illustration purposes, the difference between frequency 715A and frequency 715B is less than the difference between frequency 715C and frequency 715D. In some examples, the first width is greater than or equal to the second width. For illustration purposes, the difference between frequency 715A and frequency 715B is greater than or equal to the difference between frequency 715C and frequency 715D.

[0122] 8, a diagram 800 of illustrative examples of audio sub-bands corresponding to reconstructed sub-band audio samples 165 is shown. In a particular aspect, the reconstructed sub-band audio samples 165 are generated by the sample generation network 160 of FIG.

[0123] Audio sub-band 811A includes a first frequency range from frequency 815A to frequency 815B, where frequency 815B is greater (e.g., higher) than frequency 815A. Audio sub-band 811B includes a second frequency range from frequency 815C to frequency 815D, where frequency 815D is greater (e.g., higher) than frequency 815C. Audio sub-band 811C includes a third frequency range from frequency 815E to frequency 815F, where frequency 815F is greater (e.g., higher) than frequency 815E. Audio sub-band 811D includes a fourth frequency range from frequency 815G to frequency 815H, where frequency 815H is greater (e.g., higher) than frequency 815G. As an illustrative example, four audio sub-bands are shown. In other examples, the audio band may be subdivided into fewer or more than four sub-bands.

[0124] 4 represents audio subband 811A, reconstructed subband audio sample 165B represents audio subband 811B, reconstructed subband audio sample 165C represents audio subband 811C, and reconstructed subband audio sample 165D represents audio subband 811D. As illustrative examples, reconstructed subband audio sample 165A representing audio subband 811A, reconstructed subband audio sample 165B representing audio subband 811B, reconstructed subband audio sample 165C representing audio subband 811C, and reconstructed subband audio sample 165D representing audio subband 811D are provided. In other examples, any one of reconstructed subband audio sample 165A, reconstructed subband audio sample 165B, reconstructed subband audio sample 165C, or reconstructed subband audio sample 165D may represent audio of any one of audio subband 811A, audio subband 811B, audio subband 811C, or audio subband 811D.

[0125] In example 802, the first frequency range of audio subband 811A, the second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D are non-overlapping and non-contiguous. For illustration purposes, frequency 815C is greater (e.g., higher) than frequency 815B, frequency 815E is greater (e.g., higher) than frequency 815D, and frequency 815G is greater (e.g., higher) than frequency 815F.

[0126] In example 804, the first frequency range of audio subband 811A, the second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D are non-overlapping and contiguous. For illustration purposes, frequency 815C is equal to frequency 815B, frequency 815E is equal to frequency 815D, and frequency 815G is equal to frequency 815F.

[0127] In example 806, a first frequency range of audio subband 811A at least partially overlaps with a second frequency range of audio subband 811B, the second frequency at least partially overlaps with a third frequency of audio subband 811C, and the third frequency range at least partially overlaps with a fourth frequency range of audio subband 811D. To illustrate, frequency 815C is greater (e.g., higher) than frequency 815A and less (e.g., lower) than frequency 815B, frequency 815E is greater (e.g., higher) than frequency 815C and less (e.g., lower) than frequency 815D, and frequency 815G is greater (e.g., higher) than frequency 815E and less (e.g., lower) than frequency 815F.

[0128] In some examples, the first frequency range of audio subband 811A, the second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D each have the same width. In other examples, at least one of the first frequency range, the second frequency range, the third frequency range, or the fourth frequency range is wider than at least another one of the first frequency range, the second frequency range, the third frequency range, or the fourth frequency range.

[0129] 9, a diagram 900 of illustrative examples of audio subbands corresponding to reconstructed subband audio samples 165 is shown. In certain aspects, the reconstructed subband audio samples 165 are generated by the sample generation network 160 of FIG 1. The audio bands may be divided into subbands that are combinations of non-overlapping, non-contiguous, contiguous, or partially overlapping frequency ranges.

[0130] In example 902, the first frequency range of audio subband 811A, the second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D are non-overlapping. The first frequency range of audio subband 811A, the second frequency range of audio subband 811B, and the third frequency range of audio subband 811C are non-contiguous. For illustration purposes, frequency 815C is greater (e.g., higher) than frequency 815B, and frequency 815E is greater (e.g., higher) than frequency 815D. The third frequency range of audio subband 811C and the fourth frequency range of audio subband 811D are contiguous. For example, frequency 815G is equal to frequency 815F.

[0131] In example 904, the first frequency range of audio subband 811A, the second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D are non-overlapping. The first frequency range of audio subband 811A is contiguous with the second frequency range of audio subband 811B, and the second frequency range is contiguous with the third frequency range of audio subband 811C. For illustration purposes, frequency 815C is equal to frequency 815B, and frequency 815E is equal to frequency 815D. The third frequency range of audio subband 811C and the fourth frequency range of audio subband 811D are non-contiguous. For example, frequency 815G is greater (e.g., higher) than frequency 815F.

[0132] In example 906, a first frequency range of audio subband 811A at least partially overlaps with a second frequency range of audio subband 811B. For illustration purposes, frequency 815C is greater (e.g., higher) than frequency 815A and less (e.g., lower) than frequency 815B. The second frequency range of audio subband 811B, the third frequency range of audio subband 811C, and the fourth frequency range of audio subband 811D are non-overlapping and non-contiguous. For illustration purposes, frequency 815E is greater (e.g., higher) than frequency 815D, and frequency 815G is greater (e.g., higher) than frequency 815F.

[0133] Diagram 900 provides some illustrative non-limiting examples of combinations of sub-bands with non-overlapping, non-contiguous, contiguous, or partially overlapping frequency ranges. Audio bands may include various other combinations of sub-bands with non-overlapping, non-contiguous, contiguous, or partially overlapping frequency ranges.

[0134] 10 illustrates an implementation 1000 of the device 102 as an integrated circuit 1002 that includes one or more processors 190. The one or more processors 190 include the sample generation network 160. The integrated circuit 1002 also includes a signal input 1004, such as one or more bus interfaces, that allows input data 1051 to be received for processing. For example, the input data 1051 includes at least a portion of one or more neural network inputs 151 of FIG. 1, pitch gain 173, pitch estimate 175, LPC 141, feature data 171, encoded audio data 241, features 243, features 253, adjustment vectors 251 of FIG. 2, or combinations thereof. The integrated circuit 1002 also includes a signal output 1006, such as a bus interface, that allows transmission of an output signal, such as reconstructed audio samples 167, reconstructed audio signals 177, or combinations thereof. The integrated circuit 1002 enables implementations that perform audio sample reconstruction using neural networks and multiple subband networks as components in a system, such as a mobile phone or tablet as shown in FIG. 11, a headset as shown in FIG. 12, a wearable electronic device as shown in FIG. 13, a voice-controlled speaker system as shown in FIG. 14, a camera as shown in FIG. 15, a virtual reality headset, mixed reality headset, or augmented reality headset as shown in FIG. 16, or a vehicle as shown in FIG. 17 or FIG. 18.

[0135] 11 illustrates an implementation 1100 in which the device 102 includes a mobile device 1102, such as a phone or tablet, as an illustrative, non-limiting example. The mobile device 1102 includes a display screen 1104. Components of one or more processors 190, including a sample generation network 160, are incorporated within the mobile device 1102 and are illustrated using dashed lines to indicate internal components that are generally invisible to a user of the mobile device 1102. In a particular example, the sample generation network 160 operates to perform audio sample reconstruction to generate reconstructed audio samples 167 (e.g., reconstructed audio signal 177), which are then processed to perform one or more operations on the mobile device 1102, such as to launch a graphical user interface or, in some cases, to display other information associated with speech detected in the reconstructed audio signal 177 on the display screen 1104 (e.g., via an incorporated “smart assistant” application).

[0136] 12 illustrates an implementation 1200 in which the device 102 includes a headset device 1202. One or more components of the processor 190, including a sample generation network 160, are incorporated within the headset device 1202. In a particular example, the sample generation network 160 operates to generate reconstructed audio samples 167 (e.g., reconstructed audio signal 177) that can cause the headset device 1202 to output the reconstructed audio signal 177 via one or more speakers 136, to perform one or more operations at the headset device 1202, to transmit audio data corresponding to voice activity detected in the reconstructed audio signal 177 to a second device (not shown) for further processing, or a combination thereof.

[0137] 13 illustrates an implementation 1300 in which the device 102 includes a wearable electronic device 1302, illustrated as a "smart watch." The sample generation network 160 is incorporated within the wearable electronic device 1302. In a particular example, the sample generation network 160 operates to generate a reconstructed audio sample 167 (e.g., a reconstructed audio signal 177). In some implementations, the wearable electronic device 1302 outputs the reconstructed audio signal 177 via one or more speakers 136. In some implementations, the reconstructed audio sample 167 is processed to perform one or more operations on the wearable electronic device 1302, such as to launch a graphical user interface or, in some cases, to display other information associated with the audio detected in the reconstructed audio signal 177 (e.g., song title, artist name, etc.) on a display screen 1304 of the wearable electronic device 1302. To illustrate, the wearable electronic device 1302 may include a display screen configured to display a notification based on audio detected by the wearable electronic device 1302. In a particular example, the wearable electronic device 1302 includes a haptic device that provides a haptic notification (e.g., vibrates) in response to detecting the audio. For example, the haptic notification may cause the user to look at the wearable electronic device 1302 to see a displayed notification that shows information associated with the audio (e.g., song title, artist name, etc.).

[0138] FIG. 14 is an implementation 1400 in which the device 102 includes a wireless speaker and voice-activated device 1402. The wireless speaker and voice-activated device 1402 may have wireless network connectivity and is configured to perform assistant operations. One or more processors 190 including a sample generation network 160 are included in the wireless speaker and voice-activated device 1402. The wireless speaker and voice-activated device 1402 also includes one or more speakers 136. In operation, the wireless speaker and voice-activated device 1402 outputs a reconstructed audio signal 177 generated via operation of the sample generation network 160 via the one or more speakers 136. In some implementations, in response to a verbal command identified as a user voice in the reconstructed audio signal 177, the wireless speaker and voice-activated device 1402 may perform an assistant operation, such as via execution of an embedded assistant application. The assistant operation may include adjusting the temperature, playing music, turning on the lights, etc. For example, an assistant action may be performed in response to detecting a command following a keyword or key phrase (e.g., "Hello, assistant").

[0139] 15 illustrates an implementation 1500 in which the device 102 includes a portable electronic device corresponding to a camera device 1502. The sample generation network 160 is included within the camera device 1502. In operation, the camera device 1502 outputs a reconstructed audio signal 177 generated via operation of the sample generation network 160 via one or more speakers 136. In some implementations, as an illustrative example, in response to detecting a verbal command identified in the reconstructed audio signal 177, the camera device 1502 may perform an action in response to the verbal command, such as to adjust image or video capture settings, image or video playback settings, or image or video capture instructions.

[0140] FIG. 16 illustrates an implementation 1600 in which the device 102 includes a portable electronic device corresponding to a virtual reality headset, a mixed reality headset, or an augmented reality headset 1602. The sample generation network 160 is incorporated within the headset 1602. In certain aspects, the headset 1602 outputs a reconstructed audio signal 177 generated via operation of the sample generation network 160 through one or more speakers 136. In some implementations, voice activity detection may be performed based on the reconstructed audio signal 177. The visual interface device is placed in front of the user to enable the user to display an augmented reality, mixed reality, or virtual reality image or scene while the headset 1602 is worn. In certain examples, the visual interface device is configured to display a notification indicating audio detected within the reconstructed audio signal 177.

[0141] 17 illustrates an implementation 1700 in which the device 102 corresponds to or is incorporated within a vehicle 1702, shown as a manned or unmanned aerial device (e.g., a package delivery drone). The sample generation network 160 is incorporated within the vehicle 1702. The vehicle 1702 outputs a reconstructed audio signal 177 via operation of the sample generation network 160, such as for assembly or installation instructions for a package recipient, via one or more speakers 136.

[0142] FIG. 18 illustrates another implementation 1800 in which the device 102 corresponds to or is incorporated in a vehicle 1802, depicted as a car. The vehicle 1802 includes one or more processors 190, including a sample generation network 160. Voice recognition may be performed based on a reconstructed audio signal 177 generated via operation of the sample generation network 160. In a particular implementation, the vehicle 1802 outputs the reconstructed audio signal 177 generated via operation of the sample generation network 160 via one or more speakers 136. For example, the reconstructed audio signal 177 corresponds to an audio signal received during a phone call with another device. In another example, the reconstructed audio signal 177 corresponds to an audio signal output by an entertainment system of the vehicle 1802. In some examples, the vehicle 1802 provides information associated with the reconstructed audio signal 177 (e.g., caller identification, song title, etc.) via a display 1820.

[0143] 19, a particular implementation of a method 1900 for performing audio sample reconstruction using a neural network and multiple sub-band networks is shown. In a particular aspect, one or more operations of the method 1900 are performed by at least one of the neural network 170, the sub-band network 162, the reconstructor 166, the sample generation network 160, the audio synthesizer 150, one or more processors 190, the device 102, the system 100 of FIG. 1, the system 200 of FIG. 2, or a combination thereof.

[0144] The method 1900 includes, at 1902, processing one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output. For example, as described with reference to Figure 1, the sample generation network 160 uses a neural network 170 to process an embedding 155 based on the one or more neural network inputs 151 to generate a neural network output 161. As described with reference to Figure 3, the one or more neural network inputs 151 include at least a previous sub-band audio sample 311A, a previous sub-band audio sample 311B, a previous audio sample 371, or a combination thereof.

[0145] The method 1900 also includes, at 1904, processing the one or more first sub-band network inputs using the first sub-band neural network to generate at least one first sub-band audio sample of the first reconstructed sub-band audio signal. For example, as described with reference to FIG. 3, the sample generating network 160 uses the sub-band network 162A to process the one or more sub-band neural network inputs 361A to generate at least one reconstructed sub-band audio sample 165A of the first reconstructed sub-band audio signal. In certain aspects, the one or more sub-band neural network inputs 361A include a previous audio sample 371, a previous sub-band audio sample 311A, a previous sub-band audio sample 311B, a neural network output 161, or a combination thereof.

[0146] The method 1900 further includes, at 1906, processing the one or more second sub-band network inputs using the second sub-band neural network to generate at least one second sub-band audio sample of the second reconstructed sub-band audio signal. For example, the sample generating network 160 uses the sub-band network 162B to process the one or more sub-band neural network inputs 361B to generate at least one reconstructed sub-band audio sample 165B of the second reconstructed sub-band audio signal, as described with reference to FIG. 3. In certain aspects, the one or more sub-band neural network inputs 361B include a previous sub-band audio sample 311B, a previous audio sample 371, a reconstructed sub-band audio sample 165A, a previous sub-band audio sample 311A, a neural network output 161, or a combination thereof.

[0147] The method 1900 also includes, at 1908, using a reconstructor to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample. For example, the sample generation network 160 uses a reconstructor 166 to generate at least a reconstructed audio sample 167 of a reconstructed audio frame 153A of the reconstructed audio signal 177 based on the reconstructed subband audio sample 165A and the reconstructed subband audio sample 165B, as described with reference to FIG.

[0148] Method 1900 thus enables the generation of reconstructed audio samples 167 using neural network 170, sub-band networks 162 (e.g., sub-band networks 162A and 162B), and reconstructor 166. Using neural network 170 as an initial stage of neural network processing reduces complexity, thereby reducing processing time, memory usage, or both. Having separate sub-band networks accounts for any dependencies between audio sub-bands to accommodate adjustments across bands.

[0149] The method 1900 of Figure 19 may be implemented by a field-programmable gate array (FPGA) device, an application-specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a DSP, a graphics processing unit (GPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, the method 1900 of Figure 19 may be performed by a processor executing instructions as described with reference to Figure 20.

[0150] 20, a block diagram of a particular example implementation of a device is shown, generally designated 2000. In various implementations, the device 2000 may have more or less components than those shown in FIG. 20. In an example implementation, the device 2000 may correspond to the device 102. In an example implementation, the device 2000 may perform one or more of the operations described with reference to FIGS. 1-19.

[0151] In certain implementations, the device 2000 includes a processor 2006 (e.g., a CPU). The device 2000 may include one or more additional processors 2010 (e.g., one or more DSPs, one or more GPUs, or a combination thereof). In certain aspects, the one or more processors 190 of FIG. 1 correspond to the processor 2006, the processor 2010, or a combination thereof. The processor 2010 may include a voice and music coder-decoder (CODEC) 2008 including a voice coder ("vocoder") encoder 2036, a vocoder decoder 2038, or a combination thereof. In certain aspects, the processor 2010 may include the sample generation network 160. In certain aspects, the vocoder encoder 2036 may include the encoder 204. In certain aspects, the vocoder decoder 2038 may include the FRAE decoder 140.

[0152] The device 2000 may include a memory 2086 and a CODEC 2034. The memory 2086 may include instructions 2056 executable by one or more additional processors 2010 (or processor 2006) to perform functions described with reference to the sample generation network 160. The device 2000 may include a modem 2048 coupled to an antenna 2052 via a transceiver 2050. In certain aspects, the modem 2048 may correspond to the modem 206, the modem 240, or both of FIG. 2. In certain aspects, the transceiver 2050 may include the transmitter 208, the receiver 238, or both of FIG. 2.

[0153] The device 2000 may include a display 2028 coupled to a display controller 2026. One or more speakers 136, one or more microphones 2090, or a combination thereof may be coupled to the CODEC 2034. The CODEC 2034 may include a digital-analog converter (DAC) 2002, an analog-to-digital converter (ADC) 2004, or both. In certain implementations, the CODEC 2034 may receive analog signals from the one or more microphones 2090, convert the analog signals to digital signals using the analog-to-digital converter 2004, and provide the digital signals to the speech and music codec 2008. In certain implementations, the speech and music codec 2008 may provide the digital signals to the CODEC 2034. For example, the speech and music codec 2008 may provide the reconstructed audio signal 177 generated by the sample generation network 160 to the CODEC 2034. The CODEC 2034 may convert the digital signal to an analog signal using a digital-to-analog converter 2002 and may provide the analog signal to one or more speakers 136 .

[0154] In certain implementations, the device 2000 may be included in a system-in-package or system-on-chip device 2022. In certain implementations, the memory 2086, the processor 2006, the processor 2010, the display controller 2026, the CODEC 2034, and the modem 2048 are included in the system-in-package or system-on-chip device 2022. In certain implementations, the input device 2030 and the power supply 2044 are coupled to the system-in-package or system-on-chip device 2022. Moreover, in certain implementations, the display 2028, the input device 2030, the one or more speakers 136, the one or more microphones 2090, the antenna 2052, and the power supply 2044 are external to the system-in-package or system-on-chip device 2022, as illustrated in FIG. In particular implementations, each of the display 2028, the input device 2030, the one or more speakers 136, the one or more microphones 2090, the antenna 2052, and the power source 2044 may be coupled to a component of the system-in-package or system-on-chip device 2022, such as an interface or controller.

[0155] The device 2000 may include a smart speaker, a speaker bar, a mobile communications device, a smartphone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, an automobile, a computing device, a communications device, an internet-of-thing (IoT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0156] In accordance with the described implementation, the apparatus includes means for processing one or more neural network inputs, including at least one previous audio sample, using the neural network to generate a neural network output. For example, the means for processing one or more neural network inputs may correspond to the neural network 170, the sample generating network 160, the audio synthesizer 150, one or more processors 190, the device 102, the system 100 of FIG. 1, the system 200 of FIG. 2, the processor 2006, one or more processors 2010, one or more other circuits or components configured to process one or more neural network inputs to generate a neural network output, or any combination thereof. In a particular aspect, the at least one previous audio sample includes the previous sub-band audio sample 311A, the previous sub-band audio sample 311B, the previous audio sample 371, or a combination thereof.

[0157] The apparatus also includes means for processing one or more first sub-band network inputs using the first sub-band neural network to generate at least one first sub-band audio sample of the first reconstructed sub-band audio signal. For example, the means for processing one or more first sub-band network inputs may correspond to the sub-band network 162A, the sub-band network 162, the neural network 170, the sample generation network 160, the audio synthesizer 150, one or more processors 190, the device 102, the system 100 of FIG. 1, the system 200 of FIG. 2, the processor 2006, the one or more processors 2010, one or more other circuits or components configured to process the one or more first sub-band network inputs using the first sub-band neural network to generate at least one first sub-band audio sample, or any combination thereof.

[0158] In certain aspects, the one or more first sub-band network inputs correspond to one or more sub-band neural network inputs 361A. The one or more sub-band neural network inputs 361A include previous audio samples 371, previous sub-band audio samples 311A, previous sub-band audio samples 311B, neural network output 161, or a combination thereof. In certain aspects, the first reconstructed sub-band audio signal corresponds to audio sub-band 711A.

[0159] The apparatus further includes means for processing one or more second sub-band network inputs using a second sub-band neural network to generate at least one second sub-band audio sample of the second reconstructed sub-band audio signal. For example, the means for processing one or more second sub-band network inputs may correspond to the sub-band network 162B, the sub-band network 162, the neural network 170, the sample generation network 160, the audio synthesizer 150, one or more processors 190, the device 102, the system 100 of FIG. 1, the system 200 of FIG. 2, the processor 2006, the one or more processors 2010, one or more other circuits or components configured to process one or more second sub-band network inputs using a second sub-band neural network to generate at least one second sub-band audio sample, or any combination thereof.

[0160] In certain aspects, the one or more second sub-band network inputs correspond to one or more sub-band neural network inputs 361B. The one or more sub-band neural network inputs 361B include previous sub-band audio samples 311B, previous audio samples 371, reconstructed sub-band audio samples 165A, previous sub-band audio samples 311A, neural network outputs 161, or a combination thereof. In certain aspects, the second reconstructed sub-band audio signal corresponds to audio sub-band 711B.

[0161] The apparatus also includes means for generating at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on at least one first sub-band audio sample and at least one second sub-band audio sample. For example, the means for generating at least one reconstructed audio sample may correspond to the reconstructor 166, the neural network 170, the sample generating network 160, the audio synthesizer 150, the one or more processors 190, the device 102, the system 100 of FIG. 1, the system 200 of FIG. 2, the processor 2006, the one or more processors 2010, one or more other circuits or components configured to generate at least one reconstructed audio sample based on the at least one first sub-band audio sample and the at least one second sub-band audio sample, or any combination thereof.

[0162] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device such as memory 2086) includes instructions (e.g., instructions 2056) that, when executed by one or more processors (e.g., one or more processors 2010 or processor 2006), cause the one or more processors to process, using a neural network (e.g., neural network 170), one or more neural network inputs (e.g., one or more neural network inputs 151 as represented by embedding 155), including at least one previous audio sample (e.g., previous sub-band audio sample 311A, previous sub-band audio sample 311B, previous audio sample 371, or a combination thereof), to generate a neural network output (e.g., neural network output 161).

[0163] The instructions, when executed by the one or more processors, also cause the one or more processors to process one or more first sub-band network inputs (e.g., one or more sub-band neural network inputs 361A) using a first sub-band neural network (e.g., sub-band network 162A) to generate at least one first sub-band audio sample (e.g., reconstructed sub-band audio sample 165A) of a first reconstructed sub-band audio signal. The one or more first sub-band network inputs include at least a neural network output. The first reconstructed sub-band audio signal corresponds to a first audio sub-band (e.g., audio sub-band 711A). The instructions, when executed by the one or more processors, further cause the one or more processors to process one or more second sub-band network inputs (e.g., one or more sub-band neural network inputs 361B) using a second sub-band neural network (e.g., sub-band network 162B) to generate at least one second sub-band audio sample (e.g., reconstructed sub-band audio sample 165B) of a second reconstructed sub-band audio signal. The one or more second sub-band network inputs include at least a neural network output. The second reconstructed sub-band audio signal corresponds to a second audio sub-band (e.g., 711B) different from the first audio sub-band.

[0164] The instructions, when executed by the one or more processors, further cause the one or more processors to generate at least one reconstructed audio sample (e.g., reconstructed audio sample 167) of an audio frame (e.g., reconstructed audio frame 153A) of a reconstructed audio signal (e.g., reconstructed audio signal 177) based on the at least one first subband audio sample and the at least one second subband audio sample.

[0165] The at least one previous audio sample may include at least one previous first subband audio sample of the first reconstructed subband audio signal (e.g., previous subband audio sample 311A), at least one previous second subband audio sample of the second reconstructed subband audio signal (e.g., previous subband audio sample 311B), at least one previous reconstructed audio sample of the reconstructed audio signal (e.g., previous audio sample 371), or a combination thereof.

[0166] Certain aspects of the disclosure are described below in interrelated examples.

[0167] According to example 1, the device includes a neural network configured to process one or more neural network inputs, the neural network inputs including at least one previous audio sample, to generate a neural network output; a first subband neural network configured to process one or more first subband network inputs, the neural network output including at least one previous audio sample, to generate at least one first subband audio sample of a first reconstructed subband audio signal corresponding to a first audio subband; and a second subband neural network configured to process one or more first subband network inputs, the neural network output including at least one previous audio sample, to generate at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband. and a reconstructor configured to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, wherein the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0168] Example 2 includes the device of example 1, wherein the reconstructor is configured to generate a plurality of reconstructed audio samples of the reconstructed audio signal for each inference of the neural network, wherein the first sub-band neural network operates at a sample rate of the reconstructed audio signal, and the second sub-band neural network operates at a sample rate of the reconstructed audio signal.

[0169] Example 3 includes the device of example 1 or example 2, wherein the one or more first subband network inputs to the first subband neural network further include at least one previous first subband audio sample, at least one previous second subband audio sample, at least one previous reconstructed audio sample, or a combination thereof, and the one or more second subband network inputs to the second subband neural network further include at least one first subband audio sample, at least one previous second subband audio sample, at least one previous reconstructed audio sample, at least one previous first subband audio sample, or a combination thereof.

[0170] Example 4 includes the device of any of Examples 1 to 3, further including one or more additional subband neural networks configured to generate at least one additional subband audio sample of the one or more additional subband audio signals, wherein the at least one reconstructed audio sample is further based on the at least one additional subband audio sample.

[0171] Example 5 includes the device of any of Examples 1 to 4, further including: a third subband neural network configured to process the one or more third subband network inputs to generate at least one third subband audio sample of a third reconstructed subband audio signal; and a fourth subband neural network configured to process the one or more fourth subband network inputs to generate at least one fourth subband audio sample of a fourth reconstructed subband audio signal, wherein the at least one reconstructed audio sample is further based on the at least one third subband audio sample, the at least one fourth subband audio sample, or a combination thereof.

[0172] Example 6 includes the device of example 5, wherein the one or more third sub-band network inputs to the third sub-band neural network include at least one of the second sub-band audio samples and the neural network output, and the one or more fourth sub-band network inputs to the fourth sub-band neural network include at least one of the third sub-band audio samples and the neural network output.

[0173] Example 7 includes the device of example 5 or example 6, wherein the third reconstructed subband audio signal corresponds to a third audio subband, the fourth reconstructed subband audio signal corresponds to a fourth audio subband, the third audio subband being different from the first audio subband and the second audio subband, and the fourth audio subband being different from the first audio subband, the second audio subband, and the third audio subband.

[0174] Example 8 includes the device of any of Examples 1 to 7, wherein the first specific audio subband corresponds to a first range of frequencies, the second specific audio subband corresponds to a second range of frequencies, the first specific audio subband includes one of the first audio subband, the second audio subband, the third audio subband, or the fourth audio subband, and the second specific audio subband includes another one of the first audio subband, the second audio subband, the third audio subband, or the fourth audio subband.

[0175] Example 9 includes the device of example 8, wherein the first frequency range has a first width that is greater than or equal to a second width of the second frequency range.

[0176] Example 10 includes the device of example 8 or example 9, wherein the first range of frequencies at least partially overlaps with the second range of frequencies.

[0177] Example 11 includes the device of example 8 or example 9, wherein the first frequency range is adjacent to the second frequency range.

[0178] Example 12 includes the device of any of examples 1 to 11, wherein the recurrent layer of the neural network includes a gated recurrent unit (GRU).

[0179] Example 13 includes the device of any of examples 1 to 12, wherein the one or more neural network inputs also include predicted audio data.

[0180] Example 14 includes the device of example 13, wherein the predictive audio data includes long-term prediction (LTP) data, linear prediction (LP) data, or a combination thereof.

[0181] Example 15 includes the device of any of Examples 1 to 14, wherein the one or more neural network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, or a combination thereof.

[0182] Example 16 includes the device of any of Examples 1 to 15, wherein the first sub-band neural network includes a first neural network configured to process the one or more first sub-band network inputs to generate first residual data.

[0183] Example 17 includes the device of Example 16, wherein the first subband neural network further includes a first linear prediction (LP) filter configured to process the first residual data based on linear prediction coefficients (LPCs) to generate at least one first subband audio sample.

[0184] Example 18 includes the device of example 17, wherein the first LP filter includes a long-term prediction (LTP) filter, a short-term LP filter, or both.

[0185] Example 19 includes the device of Example 17 or Example 18, further including: a modem configured to receive the encoded audio data from the second device; and a decoder configured to decode the encoded audio data and estimate the LPC based on the feature data to generate feature data for the audio frame.

[0186] Example 20 includes the device of example 17 or example 18, further including a modem configured to receive the encoded audio data from the second device and a decoder configured to decode the encoded audio data to generate the LPC.

[0187] Example 21 includes the device of any of Examples 1 to 20, wherein the one or more second subband network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, an LP residual of the at least one first subband audio sample, at least one first subband audio sample, or a combination thereof.

[0188] Example 22 includes the device of any of Examples 1 to 21, wherein the one or more first subband network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, or a combination thereof.

[0189] Example 23 includes the device of any of examples 1 to 22, wherein the reconstructor is further configured to provide the audio frames to a speaker.

[0190] Example 24 includes the device of any of examples 1 to 23, wherein the reconstructor includes a subband reconstruction filter bank.

[0191] Example 25 includes the device of any of examples 1 to 24, wherein the at least one reconstructed audio sample includes a plurality of audio samples.

[0192] Example 26 includes the device of any of examples 1 to 25, wherein the reconstructed audio signal includes a reconstructed speech signal.

[0193] According to Example 27, a method may include processing one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; processing one or more first subband network inputs, including at least the neural network output, using a first subband neural network to generate at least one first subband audio sample of a first reconstructed subband audio signal corresponding to a first audio subband; and processing one or more first subband network inputs, including at least the neural network output, using a second subband neural network to generate at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband. and using a reconstructor to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, wherein the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0194] Example 28 includes the method of Example 27, further including using the reconstructor to generate a plurality of reconstructed audio samples of the reconstructed audio signal for each inference of the neural network, wherein the first subband neural network operates at a sample rate of the reconstructed audio signal and the second subband neural network operates at a sample rate of the reconstructed audio signal.

[0195] Example 29 includes the method of example 27 or example 28, wherein the one or more first subband network inputs to the first subband neural network further include at least one previous first subband audio sample, at least one previous second subband audio sample, at least one previous reconstructed audio sample, or a combination thereof.

[0196] Example 30 includes the method of any of Examples 27 to 29, wherein the one or more second subband network inputs to the second subband neural network further include at least one first subband audio sample, at least one previous second subband audio sample, at least one previous reconstructed audio sample, at least one previous first subband audio sample, or a combination thereof.

[0197] Example 31 includes the method of any of Examples 26 to 30, further including generating at least one additional subband audio sample of the one or more additional subband audio signals using one or more additional subband neural networks, wherein the at least one reconstructed audio sample is further based on the at least one additional subband audio sample.

[0198] Example 32 includes the method of any of Examples 27 to 31, further including: processing the one or more third subband network inputs using a third subband neural network to generate at least one third subband audio sample of a third reconstructed subband audio signal; and processing the one or more fourth subband network inputs using a fourth subband neural network to generate at least one fourth subband audio sample of a fourth reconstructed subband audio signal, wherein the at least one reconstructed audio sample is further based on the at least one third subband audio sample, the at least one fourth subband audio sample, or a combination thereof.

[0199] Example 33 includes the method of example 32, wherein the one or more third sub-band network inputs to the third sub-band neural network include at least one of the second sub-band audio samples and the neural network output, and the one or more fourth sub-band network inputs to the fourth sub-band neural network include at least one of the third sub-band audio samples and the neural network output.

[0200] Example 34 includes the method of Example 32 or Example 33, wherein the third reconstructed subband audio signal corresponds to a third audio subband, the fourth reconstructed subband audio signal corresponds to a fourth audio subband, the third audio subband is different from the first audio subband and the second audio subband, and the fourth audio subband is different from the first audio subband, the second audio subband, and the third audio subband.

[0201] Example 35 includes any of the methods of Examples 27 to 34, wherein the first specific audio subband corresponds to a first range of frequencies, the second specific audio subband corresponds to a second range of frequencies, the first specific audio subband includes one of the first audio subband, the second audio subband, the third audio subband, or the fourth audio subband, and the second specific audio subband includes another one of the first audio subband, the second audio subband, the third audio subband, or the fourth audio subband.

[0202] Example 36 includes the method of example 35, wherein the first range of frequencies has a first width that is greater than or equal to a second width of the second range of frequencies.

[0203] Example 37 includes the method of example 35 or example 36, wherein the first range of frequencies at least partially overlaps with the second range of frequencies.

[0204] Example 38 includes the method of example 35 or example 36, wherein the first range of frequencies is adjacent to the second range of frequencies.

[0205] Example 39 includes the method of any of examples 27 to 38, wherein the recurrent layer of the neural network includes a gated recurrent unit (GRU).

[0206] Example 40 includes the method of any of examples 27-39, wherein the one or more neural network inputs also include predicted audio data.

[0207] Example 41 includes the method of example 40, wherein the predictive audio data includes long-term prediction (LTP) data, linear prediction (LP) data, or a combination thereof.

[0208] Example 42 includes the method of any of Examples 27 to 41, wherein the one or more neural network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, or a combination thereof.

[0209] Example 43 includes the method of any of Examples 27 to 42, wherein the first sub-band neural network includes a first neural network configured to process the one or more first sub-band network inputs to generate first residual data.

[0210] Example 44 includes the method of Example 43, wherein the first subband neural network further includes a first linear prediction (LP) filter configured to process the first residual data based on linear prediction coefficients (LPCs) to generate at least one first subband audio sample.

[0211] Example 45 includes the method of example 44, wherein the first LP filter includes a long-term prediction (LTP) filter, a short-term LP filter, or both.

[0212] Example 46 includes the method of example 44 or example 45, further including receiving encoded audio data from the second device via the modem, decoding the encoded audio data to generate feature data for the audio frame, and estimating the LPC based on the feature data.

[0213] Example 47 includes the method of example 44 or example 45, further including receiving the encoded audio data from the second device via the modem and decoding the encoded audio data to generate the LPC.

[0214] Example 48 includes the method of any of Examples 27 to 47, wherein the one or more second subband network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, an LP residual of the at least one first subband audio sample, at least one first subband audio sample, or a combination thereof.

[0215] Example 49 includes the method of any of Examples 27 to 48, wherein the one or more first subband network inputs also include a linear prediction (LP) prediction of the at least one subband audio sample, an LP residual of the at least one previous subband audio sample, at least one previous subband audio sample, at least one previous reconstructed audio sample, or a combination thereof.

[0216] Example 50 includes the method of any of examples 27-49, wherein the reconstructor is further configured to provide the audio frames to a speaker.

[0217] Example 51 includes the method of any of examples 27-50, wherein the reconstructor includes a subband reconstruction filter bank.

[0218] Example 52 includes the method of any of examples 27 to 51, wherein the at least one reconstructed audio sample includes a plurality of audio samples.

[0219] Example 53 includes the method of any of examples 27 to 52, wherein the reconstructed audio signal includes a reconstructed speech signal.

[0220] According to Example 54, a device includes a memory configured to store instructions and a processor configured to execute the instructions to perform a method according to any of Examples 27 to 53.

[0221] According to Example 55, a non-transitory computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform a method as described in any of Examples 27 to 53.

[0222] According to Example 56, a computer program product comprises computer program instructions which, when executed by a processor, cause the processor to perform a method according to any of Examples 27 to 53.

[0223] According to Example 57, the apparatus comprises means for carrying out the method according to any of Examples 27 to 53.

[0224] According to Example 58, a non-transitory computer readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to process one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; process one or more first sub-band network inputs, using a first sub-band neural network to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal; process one or more second sub-band network inputs, using a second sub-band neural network to generate at least one second sub-band audio sample of a second reconstructed sub-band audio signal; and process the at least one first sub-band audio sample and the at least one second sub-band audio sample of a second reconstructed sub-band audio signal. and generating at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the audio sample, wherein the at least one previous audio sample comprises at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof; and the one or more second subband network inputs to the second subband neural network comprise at least one previous second subband audio sample, at least one previous reconstructed audio sample, at least one first subband audio sample, at least one previous first subband audio sample, a neural network output, or a combination thereof.

[0225] According to Example 59, a non-transitory computer-readable medium stores instructions that, when executed by one or more processors, cause the one or more processors to process one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; process one or more first subband network inputs, including at least the neural network output, using a first subband neural network to generate at least one first subband audio sample of a first reconstructed subband audio signal corresponding to a first audio subband; and process at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband. and processing one or more second subband network inputs, including at least the neural network output, using a second subband neural network to generate at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, wherein the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0226] Example 60 includes the non-transitory computer-readable medium of example 58, wherein the instructions, when executed by one or more processors, cause the one or more processors to process the one or more third subband network inputs using a third subband neural network to generate at least one third subband audio sample of a third reconstructed subband audio signal, and process the one or more fourth subband network inputs using a fourth subband neural network to generate at least one fourth subband audio sample of a fourth reconstructed subband audio signal, wherein the at least one reconstructed audio sample is further based on the at least one third subband audio sample, the at least one fourth subband audio sample, or a combination thereof.

[0227] According to Example 61, an apparatus includes means for processing one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; means for processing the one or more first subband network inputs using a first subband neural network to generate at least one first subband audio sample of a first reconstructed subband audio signal; means for processing the one or more second subband network inputs using a second subband neural network to generate at least one second subband audio sample of a second reconstructed subband audio signal; and means for processing a reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample. and means for generating at least one reconstructed audio sample of the audio frame, wherein the at least one previous audio sample comprises at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof, and the one or more second subband network inputs to the second subband neural network comprise at least one previous second subband audio sample, at least one previous reconstructed audio sample, at least one first subband audio sample, at least one previous first subband audio sample, a neural network output, or a combination thereof.

[0228] According to Example 62, an apparatus includes means for processing one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; means for processing one or more first subband network inputs, including at least the neural network output, using a first subband neural network to generate at least one first subband audio sample of a first reconstructed subband audio signal corresponding to a first audio subband; and means for processing one or more first subband network inputs, including at least the neural network output, using a second subband neural network to generate at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband. and means for processing one or more second subband network inputs, including at least one neural network output, using a neural network; and means for generating at least one reconstructed audio sample of an audio frame of the reconstructed audio signal based on the at least one first subband audio sample and the at least one second subband audio sample, wherein the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, at least one previous reconstructed audio sample of the reconstructed audio signal, or a combination thereof.

[0229] Example 63 includes the apparatus of Example 62, wherein the means for processing using a neural network, the means for processing using a first subband neural network, the means for processing using a second subband neural network, and the means for generating are incorporated within at least one of a smart speaker, a speaker bar, a computer, a tablet, a display device, a television, a game console, a music player, a radio, a digital video player, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aviation vehicle, a home automation system, a voice-activated device, a wireless speaker and a voice-activated device, a portable electronic device, a communications device, an Internet of Things (IoT) device, a virtual reality (VR) device, a base station, or a mobile device.

[0230] Those skilled in the art will further appreciate that the various exemplary logical blocks, configurations, modules, circuits, and algorithmic steps described with respect to the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of both. Various exemplary components, blocks, configurations, modules, circuits, and steps have been described above generally with respect to their functionality. Whether such functionality is implemented as hardware or as processor-executable instructions depends on the particular application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each particular application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0231] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPRO), a register, a hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transitory storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from and write information to the storage medium. Alternatively, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or user terminal In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

[0232] The foregoing description of the disclosed aspects is provided to enable any person skilled in the art to make or use the disclosed aspects. Various modifications of these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the present disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein, but is to be accorded the widest possible scope consistent with the principles and novel features defined by the following claims.

Claims

1. a neural network configured to process one or more neural network inputs, the neural network inputs including at least one previous audio sample, to generate a neural network output; a first subband neural network configured to process one or more first subband network inputs, including at least the neural network output, to generate at least one first subband audio sample of a first reconstructed subband audio signal corresponding to a first audio subband; a second subband neural network configured to process one or more second subband network inputs, including at least the neural network output, to generate at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband; and a reconstructor configured to generate at least one reconstructed audio sample of an audio frame of a reconstructed audio signal based on the at least one first sub-band audio sample and the at least one second sub-band audio sample; A device comprising: the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, or a combination thereof; device.

2. 2. The device of claim 1, wherein the reconstructor is configured to generate a plurality of reconstructed audio samples of the reconstructed audio signal for each inference of the neural network, the first sub-band neural network operating at a sample rate of the reconstructed audio signal, and the second sub-band neural network operating at the sample rate of the reconstructed audio signal.

3. 2. The device of claim 1, wherein the one or more first sub-band network inputs to the first sub-band neural network further comprise the at least one previous first sub-band audio sample, the at least one previous second sub-band audio sample, the at least one previous reconstructed audio sample, or a combination thereof; and the one or more second sub-band network inputs to the second sub-band neural network further comprise the at least one first sub-band audio sample, the at least one previous second sub-band audio sample, the at least one previous reconstructed audio sample, the at least one previous first sub-band audio sample, or a combination thereof.

4. 10. The device of claim 1, further comprising: one or more additional subband neural networks configured to generate at least one additional subband audio sample of one or more additional subband audio signals, wherein the at least one reconstructed audio sample is further based on the at least one additional subband audio sample.

5. a third sub-band neural network configured to process the one or more third sub-band network inputs to generate at least one third sub-band audio sample of a third reconstructed sub-band audio signal; a fourth sub-band neural network configured to process the one or more fourth sub-band network inputs to generate at least one fourth sub-band audio sample of a fourth reconstructed sub-band audio signal; Further provided with the at least one reconstructed audio sample is further based on the at least one third sub-band audio sample, the at least one fourth sub-band audio sample, or a combination thereof. The device of claim 1 .

6. 6. The device of claim 5, wherein the one or more third sub-band network inputs to the third sub-band neural network include the at least one second sub-band audio sample and the neural network output, and the one or more fourth sub-band network inputs to the fourth sub-band neural network include the at least one third sub-band audio sample and the neural network output.

7. 6. The device of claim 5, wherein the third reconstructed subband audio signal corresponds to a third audio subband, the fourth reconstructed subband audio signal corresponds to a fourth audio subband, the third audio subband being different from the first audio subband and the second audio subband, and the fourth audio subband being different from the first audio subband, the second audio subband, and the third audio subband.

8. 2. The device of claim 1, wherein a first specific audio subband corresponds to a first range of frequencies, a second specific audio subband corresponds to a second range of frequencies, the first specific audio subband comprises one of the first audio subband, the second audio subband, a third audio subband, or a fourth audio subband, and the second specific audio subband comprises another one of the first audio subband, the second audio subband, the third audio subband, or the fourth audio subband.

9. The first frequency range is: having a first width that is equal to or greater than a second width of the second frequency range; or at least partially overlapping with the second frequency range; or The device of claim 8 adjacent to the second range of frequencies.

10. The device of claim 1 , wherein a recurrent layer of the neural network comprises a gated recurrent unit (GRU).

11. the one or more neural network inputs further include predicted audio data; The device of claim 1 , wherein the predictive audio data comprises long-term prediction (LTP) data, linear prediction (LP) data, or a combination thereof.

12. 2. The device of claim 1, wherein the one or more neural network inputs further comprise a linear prediction (LP) prediction of at least one sub-band audio sample, an LP residual of at least one previous sub-band audio sample, the at least one previous sub-band audio sample, the at least one previous reconstructed audio sample, or a combination thereof.

13. the first sub-band neural network comprises a first neural network configured to process the one or more first sub-band network inputs to generate first residual data; 2. The device of claim 1, wherein the first sub-band neural network further comprises a first linear prediction (LP) filter configured to process the first residual data based on linear prediction coefficients (LPC) to generate the at least one first sub-band audio sample.

14. processing one or more neural network inputs, including at least one previous audio sample, using a neural network to generate a neural network output; processing one or more first sub-band network inputs, including at least the neural network output, using a first sub-band neural network to generate at least one first sub-band audio sample of a first reconstructed sub-band audio signal corresponding to a first audio sub-band; processing one or more second subband network inputs, including at least the neural network output, using a second subband neural network to generate at least one second subband audio sample of a second reconstructed subband audio signal corresponding to a second audio subband different from the first audio subband; using a reconstructor to generate at least one reconstructed audio sample of an audio frame of a reconstructed audio signal based on the at least one first sub-band audio sample and the at least one second sub-band audio sample; A method comprising: the at least one previous audio sample includes at least one previous first subband audio sample of the first reconstructed subband audio signal, at least one previous second subband audio sample of the second reconstructed subband audio signal, or a combination thereof; method.

15. 15. A non-transitory computer-readable medium storing instructions that, when executed by one or more processors, cause the one or more processors to perform the method of claim 14.

Images