Bundle encoding codec

Abstract

A device includes a memory configured to store audio data. The device also includes one or more processors configured to obtain N feature arrays, each feature array representing a frame of the audio data (where N is an integer corresponding to a bundle frame count). The one or more processors are configured to perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. The one or more processors are configured to encode the intra-bundle context array to generate a bundle encoding.

Description

BUNDLE ENCODING CODECI. Cross-Reference to Related Applications

[0001] The present application claims the benefit of priority from the commonly owned Greek Provisional Patent Application No. 20240100526, filed July 26, 2024, the contents of which are expressly incorporated herein by reference in their entirety.IL Field

[0002] The present disclosure is generally related to machine-learning codecs.III. Description of Related Art

[0003] Advances in technology have resulted in smaller and more powerful computing devices. For example, there currently exist a variety of portable personal computing devices, including wireless telephones such as mobile and smart phones, tablets and laptop computers that are small, lightweight, and easily carried by users. These devices can communicate voice and data packets over wireless networks. Further, many such devices incorporate additional functionality such as a digital still camera, a digital video camera, a digital recorder, and an audio file player. Also, such devices can process executable instructions, including software applications, such as a web browser application, that can be used to access the Internet. As such, these devices can include significant computing capabilities.

[0004] Such computing devices often incorporate functionality to exchange data with other devices via wireless transmissions. During a communication session to exchange data, a transmitting device generally encodes the data using an encoder and sends the encoded data to a receiving device via a communications channel. The receiving device receives signals representing the encoded data and uses a decoder to decode the data.

[0005] For reasons related to efficiency and / or quality of data capture and playback, communication of streaming data types (e.g., audio or video streams) often use constant bitrate (CBR) transmission. In a CBR system, each data packet includes encodedrepresentations of a specific number (e.g., a bundle count) of segments (e.g., frames) of the data being transmitted. For example, when the bundle count is three, each data packet includes three encodings representing three frames of data. Using CBR tends to increase latency due to bundling delay. For example, a data packet including an encoding for a first frame is generally not transmitted until the number of encodings available for transmission is equal to at least the bundle count. In this example, the amount of time that the encoding of the first frame is delayed to satisfy the bundle count is the bundling delay for the first frame.IV. Summary

[0006] According to one implementation of the present disclosure, a device includes a memory configured to store audio data. The device includes one or more processors configured to obtain N feature arrays, each feature array representing a frame of the audio data (where N is an integer corresponding to a bundle frame count). The one or more processors are configured to perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. The one or more processors are configured to encode the intra-bundle context array to generate a bundle encoding.

[0007] According to another implementation of the present disclosure, a method includes obtaining N feature arrays, each feature array representing a frame of audio data (where N is an integer corresponding to a bundle frame count). The method includes performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. The method includes encoding the intra-bundle context array to generate a bundle encoding.

[0008] According to another implementation of the present disclosure, a non-transitory computer-readable device stores instructions that are executable by one or more processors to cause the one or more processors to obtain N feature arrays, each feature array representing a frame of audio data (where N is an integer corresponding to a bundle frame count). The instructions cause the one or more processors to perform temporal intermingling operations on the N feature arrays to generate an intra-bundlecontext array representing the N feature arrays. The instructions cause the one or more processors to encode the intra-bundle context array to generate a bundle encoding.

[0009] According to another implementation of the present disclosure, an apparatus includes means for obtaining N feature arrays, each feature array representing a frame of audio data (where N is an integer corresponding to a bundle frame count). The apparatus includes means for performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. The apparatus includes means for encoding the intra-bundle context array to generate a bundle encoding.

[0010] According to another implementation of the present disclosure, a device includes a memory configured to store audio data. The device includes one or more processors configured to decode a bundle encoding to generate an intra-bundle context array. The one or more processors are configured to perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing the audio data (where N is an integer corresponding to a bundle frame count).

[0011] According to another implementation of the present disclosure, a method includes decoding a bundle encoding to generate an intra-bundle context array. The method includes performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays (where N is an integer corresponding to a bundle frame count). The method includes generating N audio frames based on the N feature arrays.

[0012] According to another implementation of the present disclosure, an apparatus includes means for decoding a bundle encoding to generate an intra-bundle context array. The apparatus includes means for performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays (where N is an integer corresponding to a bundle frame count). The apparatus includes means for generating N audio frames based on the N feature arrays.

[0013] According to another implementation of the present disclosure, a non-transitory computer-readable device stores instructions that are executable by one or more processors to cause the one or more processors to decode a bundle encoding to generatean intra-bundle context array. The instructions cause the one or more processors to perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing audio data (where N is an integer corresponding to a bundle frame count). The instructions cause the one or more processors to generate N audio frames based on the N feature arrays.

[0014] Other aspects, advantages, and features of the present disclosure will become apparent after review of the entire application, including the following sections: Brief Description of the Drawings, Detailed Description, and the Claims.V. Brief Description of the Drawings

[0015] FIG. l is a block diagram of a particular illustrative aspect of a system operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0016] FIG. 2 is a block diagram of a particular illustrative aspect of a system operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0017] FIG. 3 is a block diagram of a particular illustrative aspect of a system operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0018] FIG. 4 is a block diagram of a particular illustrative aspect of a system operable to decode audio data of a bundle encoding, in accordance with some examples of the present disclosure.

[0019] FIG. 5 is a block diagram of a particular illustrative aspect of a system operable to decode audio data of a bundle encoding, in accordance with some examples of the present disclosure.

[0020] FIG. 6 is a block diagram of a particular illustrative aspect of a system operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0021] FIG. 7 is a diagram of an illustrative aspect of operations associated with bundle encoding, in accordance with some examples of the present disclosure.

[0022] FIG. 8 is a diagram of an illustrative aspect of operations associated with bundle encoding, in accordance with some examples of the present disclosure.

[0023] FIG. 9 is a flowchart of an example of a method of generating a bundle encoding, in accordance with some examples of the present disclosure.

[0024] FIG. 10 is a flowchart of an example of a method of generating audio data based on a bundle encoding, in accordance with some examples of the present disclosure.

[0025] FIG. 11 illustrates an example of an integrated circuit operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0026] FIG. 12 is a diagram of a mobile device operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0027] FIG. 13 is a diagram of a hearing aid device operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0028] FIG. 14 is a diagram of a headset operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0029] FIG. 15 is a diagram of a wearable electronic device operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0030] FIG. 16 is a diagram of a mixed reality or augmented reality glasses device operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0031] FIG. 17 is a diagram of earbuds operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0032] FIG. 18 is a diagram of a voice-controlled speaker system operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0033] FIG. 19 is a diagram of a camera operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0034] FIG. 20 is a diagram of a headset, such as a virtual reality, mixed reality, or augmented reality headset, operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0035] FIG. 21 is a diagram of a first example of a vehicle operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0036] FIG. 22 is a diagram of a second example of a vehicle operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.

[0037] FIG. 23 is a block diagram of a particular illustrative example of a device that is operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure.VI Detailed Description

[0038] Packet-switched networks often use CBR transmission of data. As an example, data communicated over the internet, which is packet switched, is generally subject to bundling requirements. Bundling can introduce latency, which can negatively impact some types of data communications, such as real-time voice or video communications.

[0039] One problem with both traditional and machine-learning (ML) based CBR speech codecs is that at very low bitrates, they tend to have poor voice quality and poor channel resilience.

[0040] The disclosed embodiments solve quality problems associated with CBR communications with low bitrate constraints. One solution provided by the disclosedembodiments includes moving segment accumulation operations prior to encoding and performs joint encoding of a set of segments for a bundle (referred to herein as bundle encoding). A bundle encoding includes data represents each segment of data assigned to a bundle in a single intermingled and encoded data set, which enables the encoder to take advantage of segment-to-segment similarities during encoding. Thus, a technical advantage of accumulating segments for bundle encoding is high quality data reproduction for a given bitrate, a lower bitrate for a given reproduction quality, or both.

[0041] Another solution provided by the disclosed embodiments includes generating a single bundle encoding for data segments assigned to a bundle such that no particular bit of the bundle encoding is allocated in advance to a particular one of the data segments. That is, any allocation of bits of the bundle encoding to particular data segments is source controlled (e.g. based on the data of the data segments), variable (e.g., can differ from one bundle to the next), and learned (rather than preassigned). Due to training of the codec, this arrangement enables more bits of the bundle encoding to be used to represent data that is more important for high-quality reproduction of the data, and fewer bits of the bundle encoding to be used to represent data that is less important for high-quality reproduction of the data. Thus, a technical advantage of generating a single bundle encoding for data segments such that no particular bit of the bundle encoding is allocated in advance to a particular one of the data segments is ensuring that the bundle encoding emphasizes data important to high quality data reproduction.

[0042] A further aspect of the disclosed embodiments is that the solutions described above, which improve reproduction quality for CBR communications, are provided without increasing latency. For example, the disclosed embodiments incur the same bundling delay as conventional CBR communication techniques, and merely shift segment accumulation to an early stage in the encoding and transmission process to enable the encoder to have access to each segment assigned to a bundle during bundle encoding, rather than bundling previously encoded segments as in conventional CBR communications. Thus, the disclosed embodiments use the bundling delay more efficiently by using the delay to accumulate segments for joint encoding, thereby enabling the bundle encoder to exploit mutual information among the jointly encoded segments.

[0043] Thus, the disclosed embodiments solve challenges faced by communication services that operate over packet-switched networks, such as providing high-quality data reproduction using low-bitrate CBR transmissions. Further, the disclosed embodiments are able to solve these challenges without increasing latency.

[0044] Autoencoders have gained popularity in recent years as they are able to learn efficient representations of input data without the need for labels, i.e., unsupervised learning. Various types of autoencoders exist and are well explained in “Autoencoder and its various variants” (2018 IEEE International Conference on Systems, Man, and Cybernetics), Zhang et al.

[0045] Autoencoders conditioned on spectral features input have been used for speech compression, as described in "Enhancing into the Codec: Noise Robust Speech Coding with Vector-Quantized Autoencoders" (arXiv:2010.06610vl [eess.AS], 12 Feb 2021), Casebeer et al. Speech coding is a lossy compression process. For example, for speech coding, an autoencoder performs dimensionality reduction, wherein an N-dimensional input vector is passed into the encoder of the autoencoder, and a compressed representation is generated to represent the important aspects of the input vector in an M-dimensional vector, where M is smaller than N, e.g., by an order of magnitude.

[0046] A drawback of using conventional, feedforward autoencoders for speech coding is that they cannot necessarily exploit the temporal relationships between sets of input data. U.S. patent 11,526,734 ('"734 patent”) assigned to Qualcomm, Inc., "Method and Apparatus for Recurrent Auto Encoding" (Yang et al.) describes a feedback recurrent autoencoder (FRAE) that improves speech coding by exploiting temporal redundancies and correlations among audio frames. The FRAE in the '734 patent provides details on training and application of compression of sequential data with temporal correlation.The recurrent structure of the FRAE efficiently extracts the redundancy embedded along the time-dimension of sequential data, enabling compact discrete representation of the data at the bottleneck in a sequential fashion.

[0047] The '734 patent describes (e.g., at Table 1) an error function used for training the FRAE. In particular, the '734 patent describes use of MSE (Mean Squared Error), usedas Mel-scale mean-square-error, as a measure of reconstruction loss for training. In Table 1, the MSE of each frequency bin is scaled according to its weight at Mel- frequency for both latent feedback and output feedback.

[0048] The FRAE described in the ‘734 patent has two advantageous features not found in traditional autoencoders: (1) recurrent layers, e.g., long short-term memory (LSTM) or gated recurrent unit (GRU) layers, which retain memory of past inputs; and (2) feedback from the decoder of the autoencoder to the encoder of the autoencoder. The feedback connection 150 in Fig. 2, Fig. 3, and Fig. 7 of the ‘734 patent provides additional historical information from the state (ht) in the decoder to the encoder about how reconstruction of a prior input has fared. This feedback loop is present during both training and inference, which enables the encoder to be trained to respond to particular feedback conditions in a manner that improves reproduction of the input data by the decoder. Thus, the feedback loop is analogous to a mode switch input that serves as an input that influences how the encoder operates on the next input vector.

[0049] In addition, the ‘734 patent describes a second feedback connection (152) from the state (ht) of the decoder to an earlier layer of the decoder. This second feedback connection (152) enables the decoder to learn from its previous reconstruction attempts, providing additional historical context about how reconstruction of a prior input has fared. This second feedback loop is also present during both training and inference, which allows the decoder to be trained to respond to particular feedback conditions in a manner that improves reproduction of the input data by the decoder.

[0050] The ‘734 patent also describes other optional feedback connections. For example, an embedding vector (z) may be fed back via a third feedback connection (356 in Fig. 3 of the ‘734 patent) to the encoder. As another optional example, the '734 patent describes that the FRAE can be a variational autoencoder. In this example, an output of the decoder (754 in Fig. 7 of the ‘734 patent) is sampled and / or parameterized to a generate an autoregressive prior which can be used as a fourth feedback connection to condition a prior model for the next latent space embedded vector (zt+1). Like the previously described feedback connections, each of these optional connections ispresent during both training and inference, and thus are trained as part of the training process thereby improving functionality of the FRAE.

[0051] When used as part of a system to communicate encoded data between two devices, the FRAE of the '734 patent can efficiently encode speech for transmission. In such a system, a transmitting device typically includes the full FRAE, and a receiving device includes at least the decoder of the FRAE. On the transmitting device side, input data (e.g., an input vector) including speech is provided as input to the encoder, and the encoder compresses the input data and provides the compressed data to a bottleneck layer. The bottleneck layer generates a latent space vector, which is optionally quantized, for transmission to the receiving device. At the receiving device, a dequantized version of the latent space vector is provided as input to the decoder, which decompresses the latent space vector to generate a reproduction of the input data.

[0052] Disclosed embodiments expand on the various FRAEs of the '734 patent to perform joint encoding of a set of data segments (e.g., frames of audio data) that are associated with a particular bundle. Thus, each bundle in the payload of a packet includes a bundle encoding representing a set of data segments. The number of data segments associated with each bundle (and therefore jointly encoded to form a bundle encoding) is based on a bundle count of a CBR communication system. As explained above, performing bundle encoding enables the encoder (e.g., the FRAE) to take into account temporal redundancies between the data segments of the bundle, which enables high-quality data reproduction without increasing latency or bitrate.

[0053] Factors that can influence channel conditions (and therefore a count of segments per bundle) include, without limitation, whether the channel includes terrestrial or nonterrestrial (e.g., orbital or aerial) communication links, a mode of operation of the communication links (e.g., best efforts or managed), type(s) of radio access technology used by the communication links, the specific frequency band(s) used for transmissions, whether one or more devices of a communication link are mobile, and many other factors that affect specific channel condition metrics, such as block error rate, delay, jitter, Carrier-to-Interference plus Noise Ratio (CINR), Signal-to-Interference plus Noise Ratio (SINR), Doppler spread, and so forth.

[0054] Particular aspects of the present disclosure are described below with reference to the drawings. In the description, common features are designated by common reference numbers. As used herein, various terminology is used for the purpose of describing particular implementations only and is not intended to be limiting of implementations. For example, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Further, some features described herein are singular in some implementations and plural in other implementations. To illustrate, FIG. 1 depicts a device 102 including one or more processors ("processor(s)" 190 of FIG. 1), which indicates that in some implementations the device 102 includes a single processor 190 and in other implementations the device 102 includes multiple processors 190. For ease of reference herein, such features are generally introduced as "one or more" features and are subsequently referred to in the singular or optional plural (as indicated by "(s)") unless aspects related to multiple of the features are being described.

[0055] In some drawings, multiple instances of a particular type of feature are used. Although these features are physically and / or logically distinct, the same reference number is used for each, and the different instances are distinguished by addition of a letter to the reference number. When the features as a group or a type are referred to herein, e.g., when no particular one of the features is being referenced, the reference number is used without a distinguishing letter. However, when one particular feature of multiple features of the same type is referred to herein, the reference number is used with the distinguishing letter. For example, referring to FIG. 2, multiple packets are illustrated and associated with reference numbers 212A and 212B. When referring to a particular one of these packets, such as a packet 212A, the distinguishing letter "A" is used. However, when referring to any arbitrary one of these packets or to these packets as a group, the reference number 212 is used without a distinguishing letter.

[0056] As used herein, the terms "comprise," "comprises," and "comprising" may be used interchangeably with "include," "includes," or "including." Additionally, the term "wherein" may be used interchangeably with "where." As used herein, "exemplary" indicates an example, an implementation, and / or an aspect, and should not be construed as limiting or as indicating a preference or a preferred implementation. As used herein,an ordinal term (e.g., "first," "second," "third," etc.) used to modify an element, such as a structure, a component, an operation, etc., does not by itself indicate any priority or order of the element with respect to another element, but rather merely distinguishes the element from another element having a same name (but for use of the ordinal term). As used herein, the term "set" refers to one or more of a particular element, and the term "plurality" refers to multiple (e.g., two or more) of a particular element.

[0057] As used herein, "coupled" may include "communicatively coupled," "electrically coupled," or "physically coupled," and may also (or alternatively) include any combinations thereof. Two devices (or components) may be coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) directly or indirectly via one or more other devices, components, wires, buses, networks (e.g., a wired network, a wireless network, or a combination thereof), etc. Two devices (or components) that are electrically coupled may be included in the same device or in different devices and may be connected via electronics, one or more connectors, or inductive coupling, as illustrative, non-limiting examples. In some implementations, two devices (or components) that are communicatively coupled, such as in electrical communication, may send and receive signals (e.g., digital signals or analog signals) directly or indirectly, via one or more wires, buses, networks, etc. As used herein, "directly coupled" may include two devices that are coupled (e.g., communicatively coupled, electrically coupled, or physically coupled) without intervening components.

[0058] In the present disclosure, terms such as "determining," "calculating," "estimating," "shifting," "adjusting," etc. may be used to describe how one or more operations are performed. It should be noted that such terms are not to be construed as limiting and other techniques may be utilized to perform similar operations. Additionally, as referred to herein, "generating," "calculating," "estimating," "using," "selecting," "accessing," and "determining" may be used interchangeably. For example, "generating," "calculating," "estimating," or "determining" a parameter (or a signal) may refer to actively generating, estimating, calculating, or determining the parameter (or the signal) or may refer to using, selecting, or accessing the parameter (or signal) that is already generated, such as by another component or device.

[0059] As used herein, the term "machine learning" should be understood to have any of its usual and customary meanings within the fields of computers science and data science, such meanings including, for example, processes or techniques by which one or more computers can learn to perform some operation or function without being explicitly programmed to do so. As a typical example, machine learning can be used to enable one or more computers to analyze data to identify patterns in data and generate a result based on the analysis. For certain types of machine learning, the results that are generated include data that indicates an underlying structure or pattern of the data itself. Such techniques, for example, include so-called "clustering" techniques, which identify clusters (e.g., groupings of data elements of the data).

[0060] For certain types of machine learning, the results that are generated include a data model (also referred to as a "machine-learning model" or simply a "model"). Typically, a model is generated using a first data set to facilitate analysis of a second data set. For example, a first portion of a large body of data may be used to generate a model that can be used to analyze the remaining portion of the large body of data. As another example, a set of historical data can be used to generate a model that can be used to analyze future data.

[0061] Since a model can be used to evaluate a set of data that is distinct from the data used to generate the model, the model can be viewed as a type of software (e.g., instructions, parameters, or both) that is automatically generated by the computer(s) during the machine learning process. As such, the model can be portable (e.g., can be generated at a first computer, and subsequently moved to a second computer for further training, for use, or both). Additionally, a model can be used in combination with one or more other models to perform a desired analysis. To illustrate, first data can be provided as input to a first model to generate first model output data, which can be provided (alone, with the first data, or with other data) as input to a second model to generate second model output data indicating a result of a desired analysis. Depending on the analysis and data involved, different combinations of models may be used to generate such results. In some examples, multiple models may provide model output that is input to a single model. In some examples, a single model provides model output to multiple models as input.

[0062] Examples of machine-learning models include, without limitation, perceptrons, neural networks, support vector machines, regression models, decision trees, Bayesian models, Boltzmann machines, adaptive neuro-fuzzy inference systems, as well as combinations, ensembles and variants of these and other types of models. Variants of neural networks include, for example and without limitation, prototypical networks, autoencoders, transformers, self-attention networks, convolutional neural networks, deep neural networks, deep belief networks, etc. Variants of decision trees include, for example and without limitation, random forests, boosted decision trees, etc.

[0063] Since machine-learning models are generated by computer(s) based on input data, machine-learning models can be discussed in terms of at least two distinct time windows - a creation / training phase and a runtime phase. During the creation / training phase, a model is created, trained, adapted, validated, or otherwise configured by the computer based on the input data (which in the creation / training phase, is generally referred to as "training data"). Note that the trained model corresponds to software that has been generated and / or refined during the creation / training phase to perform particular operations, such as classification, prediction, encoding, or other data analysis or data synthesis operations. During the runtime phase (or "inference" phase), the model is used to analyze input data to generate model output. The content of the model output depends on the type of model. For example, a model can be trained to perform classification tasks or regression tasks, as non-limiting examples. In some implementations, a model may be continuously, periodically, or occasionally updated, in which case training time and runtime may be interleaved or one version of the model can be used for inference while a copy is updated, after which the updated copy may be deployed for inference.

[0064] In some implementations, a previously generated model is trained (or re-trained) using a machine-learning technique. In this context, "training" refers to adapting the model or parameters of the model to a particular data set. Unless otherwise clear from the specific context, the term "training" as used herein includes "re-training" or refining a model for a specific data set. For example, training may include so called "transfer learning." In transfer learning a base model may be trained using a generic or typicaldata set, and the base model may be subsequently refined (e.g., re-trained or further trained) using a more specific data set.

[0065] A data set used during training is referred to as a "training data set" or simply "training data". The data set may be labeled or unlabeled. "Labeled data" refers to data that has been assigned a categorical label indicating a group or category with which the data is associated, and "unlabeled data" refers to data that is not labeled. Typically, "supervised machine-learning processes" use labeled data to train a machine-learning model, and "unsupervised machine-learning processes" use unlabeled data to train a machine-learning model; however, it should be understood that a label associated with data is itself merely another data element that can be used in any appropriate machinelearning process. To illustrate, many clustering operations can operate using unlabeled data; however, such a clustering operation can use labeled data by ignoring labels assigned to data or by treating the labels the same as other data elements.

[0066] Training a model based on a training data set generally involves changing parameters of the model with a goal of causing the output of the model to have particular characteristics based on data input to the model. To distinguish from model generation operations, model training may be referred to herein as optimization or optimization training. In this context, "optimization" refers to improving a metric, and does not mean finding an ideal (e.g., global maximum or global minimum) value of the metric. Examples of optimization trainers include, without limitation, backpropagation trainers, derivative free optimizers (DFOs), and extreme learning machines (ELMs). As one example of training a model, during supervised training of a neural network, an input data sample is associated with a label. When the input data sample is provided to the model, the model generates output data, which is compared to the label associated with the input data sample to generate an error value. Parameters of the model are modified in an attempt to reduce (e.g., optimize) the error value. As another example of training a model, during unsupervised training of an autoencoder, a data sample is provided as input to the autoencoder, and the autoencoder reduces the dimensionality of the data sample (which is a lossy operation) and attempts to reconstruct the data sample as output data. In this example, the output data is compared to the input data sample togenerate a reconstruction loss, and parameters of the autoencoder are modified in an attempt to reduce (e.g., optimize) the reconstruction loss.

[0067] FIG. 1 is a block diagram of a particular illustrative aspect of a system 100 that includes a device 102 that includes one or more codecs 120. The codec(s) 120 include one or more bundle encoders 126, one or more bundle decoder(s) 128, or both. As described further below, the bundle encoder(s) 126 are configured to jointly encode data representing audio frames for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. Conversely, the bundle decoder(s) 128 are configured to decode a bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. A technical advantage of jointly encoding data representing all of the audio frames of a bundle is that the bundle encoding so generated can be transmitted via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0068] In the example illustrated in FIG. 1, the device 102 includes one or more processors 190, memory 140, one or more microphones ("Mic(s)") 114, a modem 170, and a radio frontend (RFE) 172. The processor(s) 190 are coupled to the microphone(s) 114, the memory 140 and the modem 170. In some embodiments, one or more interface devices or circuits are coupled between the processor(s) 190 and one or more of the microphone(s) 114, the memory, and the modem 170. For example, the microphone(s) 114 can be coupled to microphone interface circuitry that is configured to digitize signals from the microphone(s) 114 and to provide the digitized signals to the processor(s) 190. The modem 170 is coupled to the RFE 172.

[0069] The RFE 172 includes transmit circuitry, receive circuitry, or both. For example, the RFE 172 can include one or more amplifiers, one or more filters, one or more oscillators, gain control circuitry, an analog-to-digital converter (ADC), a digital-to- analog converter (DAC), etc. The RFE 172 is coupled to one or more antennas 174 and to the modem 170. The modem 170 and the RFE 172 are configured to cooperate to receive transmissions 178 including encoded data for decoding by the codec(s) 120, tosend transmissions 178 including encoded data encoded by the codec(s) 120 to one or more other devices 182, or both.

[0070] In FIG. 1, the microphone(s) 114 are configured to capture audio data 146, which can be stored in the memory 140, provided to the processor(s) 190 for processing (e.g., encoding), or both. The microphone(s) 114 can be integrated within the device 102, as illustrated in FIG. 1, or external to the device 102 and coupled to the processor(s) 190 via an interface. In some cases, the microphone(s) 114 can include one or more internal microphones (i.e., microphone(s) integrated within the device 102) and one or more external microphones (i.e., microphone(s) external to the device 102 and communicatively coupled thereto).

[0071] In the example illustrated in FIG. 1, the codec(s) 120 include one or more preprocessors 122, one or more feature extractors 124, the bundle encoder(s) 126, the bundle decoder(s) 128, one or more neural synthesizers 130, and one or more postprocessors 132. The preprocessor(s) 122 and the postprocessor(s) 132 are optional and are omitted in some embodiments.

[0072] In some embodiments, the device 102 includes multiple codecs 120, and the processor(s) 190 select a particular one of the codecs 120 to use for the audio data 146 (or a portion of the audio data 146, such as a particular bundle) based on one or more of several factors. For example, the processor(s) 190 can select a particular one of the codecs 120 to use based on the audio data 146. To illustrate, if the audio data 146 exhibits first characteristics, a first codec can be used, whereas a second codec can be used if the audio data 146 does not exhibit the first characteristics (or if the audio data 146 exhibits second characteristics). For instance, if the audio data 146 represents speech (or particular types of speech, such as voiced or unvoiced speech) the first codec can be used, and the second codec can be used if the audio data 146 does not represent speech.

[0073] As another example, the processor(s) 190 can select a particular one or more of the codecs 120 to use based on channel conditions experienced at the RFE 172 or at one of the other device(s) 182. The channel conditions can indicate, for example, noisecharacteristics of a channel 176, channel capacity of the channel 176, path loss of the channel 176, data loss of the channel 176 (e.g., bit error rates or packet loss rates), etc. In some cases, the channel conditions can be used to determine a bundle frame count (N) 142 indicating a number of audio frames 118 to be included in a bundle. In such cases, the processor(s) 190 can select a particular one of the codecs 120 based on the bundle frame count 142.

[0074] As another example, the processor(s) 190 can select a particular one of the codecs 120 to use based on other factors, such as user settings or service provider settings. Further, in some examples, the processor(s) 190 can select a particular one of the codecs 120 to use based on a combination of any of the above.

[0075] When the device 102 includes multiple codecs 120, some features of the codec(s) 120 can be shared by two or more of the multiple codecs 120. For example, two or more of the codecs 120 can use the same preprocessors 122, the same feature extractors 124, the same neural synthesizers 130, the same postprocessors 132, or a combination thereof.

[0076] When the codec(s) 120 include multiple bundle encoders 126, two or more of the multiple bundle encoders 126 can be associated with different bundle frame counts 142. For example, a first bundle encoder can be associated with a bundle frame count of two (N=2), a second bundle encoder can be associated with a bundle frame count of three (N=3), a third bundle encoder can be associated with a bundle frame count of four (N=4), etc. In some embodiments, the processor(s) 190 are configured to determine a value of N based on channel condition data associated with communication of encoded data 150 (e.g., a bundle encoding) via one or more transmissions 178 over the channel 176.

[0077] The preprocessor(s) 122, if present, are configured to perform one or more operations to prepare the audio data 146 for encoding. For example, the preprocessor(s) 122 can include a denoiser configured to reduce noise in the audio data 146. In other examples, the preprocessor s) 122 include components configured to perform other preprocessing operations, such as audio classification, speech enhancement, voicedetection, etc. The preprocessor s) 122 can be machine-learning (ML) based, procedural, or combinations thereof. To illustrate, the preprocessor s) 122 can include an ML-based audio enhancement component that operates using one or more trained parameters.

[0078] The feature extractor(s) 124 are configured to generate input data for the bundle encoder( s) 126, where the input data include features extracted from or generated based on the audio data 146. Generally, the feature extractor(s) 124 are configured to generate one feature array for each frame of the audio data 146; however, other arrangements are possible. For example, the feature extractor(s) 124 may generate two or more feature arrays for each frame, and the two or more feature arrays can be provided as input to different bundle encoder(s) 126 or combined as input to a single bundle encoder 126. In some embodiments, the feature extractor(s) 124 can include ML-based feature extractor(s), procedural feature extractor(s), or both.

[0079] In some embodiments, the feature extractor(s) 124 include a spectral envelope feature extractor that is configured to generate feature data representing a spectral envelope of one or more segments (e.g., frames, sub-frames, or samples) of the audio data 146. In such embodiments, the feature extractor 124 may generate cepstrum data, cepstral coefficients, liftered cepstrum (e.g., determined using discrete cosine transform truncation to exclude pitch information and only capture spectral envelope information), companded (e.g., log) filterbank energies (e.g., determined by applying inverse discrete cosine transform on liftered cepstrum data), filterbank energies computed by uncompanding (e.g., exp) the companded filterbank energies, a full resolution spectrum, linear prediction (LP) coefficients (e.g., determined from speech frames using an autocorrelation method (e.g., Levinson-Durbin) or a covariance method), line spectral frequencies (LSF), line spectral pairs (LSP), or other data representing the spectral envelope of the audio data 146.

[0080] In some embodiments, the feature extractor(s) 124 include a pitch feature extractor configured to generate feature data representing pitch characteristics of one or more segments (e.g., frames or samples) of the audio data 146. In such embodiments, the feature extractor 124 can generate fundamental frequency (fO) data, pitch correlationdata, pitch lag data, or other frequency-domain data or time-domain data. In some embodiments, the feature extractor(s) 124 can also generate other data, such as an indication of whether speech in the audio data 146 is voiced or unvoiced.

[0081] The bundle encoder(s) 126 are configured to generate encoded data 150 (e.g., a bundle encoding) representing N feature arrays from the feature extractor(s) 124, where each feature array represents an audio frame 118 of the audio data 146, and where N is an integer corresponding to the bundle frame count 142. In some embodiments, each bundle encoder of the bundle encoder(s) 126 is configured to generate a bundle encoding for a specific bundle frame count 142. For example, a first bundle encoder may be configured to generate a bundle encoding for a bundle frame count 142 of two (N=2), a second bundle encoder may be configured to generate a bundle encoding for a bundle frame count 142 of three (N=3), a third bundle encoder may be configured to generate a bundle encoding for a bundle frame count 142 of four (N=4), etc.

[0082] In a particular aspect, a bundle encoder 126 is configured to obtain N feature arrays. For example, the bundle encoder 126 can receive the N feature arrays from the feature extractor(s) 124 or from the memory 140. Each feature array can represent one of the audio frames 118 (or a portion of one of the frames 118) of the audio data 146.

[0083] The bundle encoder 126 is configured to perform several operations to generate a bundle encoding based on the N feature arrays. For example, the bundle encoder 126 is configured to perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays and to encode the intra-bundle context array to generate a bundle encoding. In this context, "temporal intermingling" refers to operations that combine data from different audio frames, where each audio frame is associated with a particular time segment of the audio data 146. For example, for N=2, the temporal intermingling operations include intermingling data of a first feature array representing an audio frame T=1 with data of a second feature array representing an audio frame T=2 (where T is a time index associated with each audio frame). Similarly, for N=3, the temporal intermingling operations include intermingling data of a first feature array representing an audio frame T=l, data of a second featurearray representing an audio frame T=2, and data of a third feature array representing an audio frame T=3, and so forth for other values of N.

[0084] In some embodiments, performing the temporal intermingling (also referred to as intra-bundle context modeling) operations includes performing unshuffling of the temporal dimension in the channel dimension, similar to depth-to-space operation, to obtain a feature array. Temporal pooling is achieved in a subsequent convolutional layer through channel mixing. The feature array can be further processed (e.g., passed through convolutions, fully-connected layers, or other neural network layers) to generate the intermingled array representing the intra-bundle context. Later, the intra- bundle context array can be processed and unstacked / re-arranged to generate N feature arrays using an operation akin to space-to-depth. The operations sandwiched between stacking and unstacking work on the entire intra-context bundle. To illustrate, each of the N feature arrays can include spectral data for different time segments (e.g., frames) of the audio data 146.

[0085] In some embodiments, performing the temporal intermingling (e.g., intra-bundle context modeling) operations includes performing convolution of each feature array to generate N intermediate feature arrays and providing the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays. In some examples, the recurrent layer(s) include bidirectional recurrent layers. For example, the recurrent layer(s) can include bidirectional gated recurrent unit layer(s). In other examples, the recurrent layer(s) include other machine-learning model architectures, such as long-short term memory layer(s), transformer layer(s), other self-attention layer(s), etc. In any of these examples, the N temporal context arrays can represent hidden states of the recurrent layer(s).

[0086] The recurrent layer(s) output temporal context arrays, which can be combined to generate the intra-bundle context array. Combining the N temporal context arrays can include concatenating the temporal context arrays, performing one or more pooling operations using the N temporal context arrays, or performing other mathematical operations to combine the N temporal context arrays.

[0087] In a particular aspect, an intra-bundle context array representing N feature arrays is provided as input to an ML encoder to generate a bundle encoding representing the N feature arrays. The bundle encoding represents the N frames of the N feature arrays using a specified number of bits (e.g., M bits, where M is an integer representing a fixed bundle size). In a particular embodiment, no specific subset of the M bits is mapped in advance to particular frames or portions of the frames. For example, any associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of the ML encoder. Further, associations between specific frames of the N frames and specific bits of the M bits are source controlled. To illustrate, a first feature array representing a first frame includes a first set of values, and a second feature array representing a second frame includes a second set of values. In an M-bit bundle encoding representing the first and second frames, whether a particular set of bits is associated with the first frame, the second frame, or both, depends on the first set of values, the second set of values, and trained parameters of the ML encoder.

[0088] Since the ML encoder is configured to generate an M-bit bundle encoding based on an intra-bundle context array representing N feature arrays, different ML encoders can be used for different values of N (e.g., different bundle frame counts 142), different values of M (e.g., different bundle sizes), or both. In some embodiments, the processor(s) 190 are configured to select a specific ML encoder (from among a plurality of ML encoders) to encode the intra-bundle context array to generate the bundle encoding based on the bundle frame count 142 (e.g., based on the value of N), based on the bundle size (e.g., based on the value of M), or both.

[0089] In some embodiments, the ML encoder is configured as a feed-back recurrent autoencoder that includes a joint encoder, a bottleneck, and a joint decoder. For example, the joint encoder can include one or more gated recurrent unit layers, the bottleneck can include one or more fully connected layers and optionally a quantizer, and the joint decoder can include one or more gated recurrent unit layers. In other examples, the joint encoder and joint decoder include self-attention mechanisms rather than or in addition to gated-recurrent layers.

[0090] In some embodiments, the bundle encoding corresponds to a latent state of the bottleneck. In such embodiments, the joint encoder, the bottleneck, and the joint decoder are trained together (e.g., during training time). During inference time (e.g., during operation) at a transmitting device, the joint encoder and bottleneck can operate to generate the bundle encoding, and operation of the joint decoder can be omitted. Alternatively, the joint encoder, the bottleneck, and the joint decoder can operate together to generate the bundle encoding. For example, the joint decoder can operate to decode the bundle encoding to generate feedback to the joint encoder.

[0091] The bundle encoding can be packetized and provided as encoded data 150 to the modem 170 and the RFE 172 for transmission to the other device(s) 182. Packetizing includes forming a packet that includes a header and a payload, where the bundle encoding is included in the payload.

[0092] During inference time (e.g., during operation) at a receiving device, the joint decoder can operate to decode the bundle encoding to generate an intra-bundle context array. The intra-bundle context array can be processed by the neural synthesize^ s) 130, the postprocessor(s) 132, or both, to generate reconstructed audio frames representing the N audio frames associated with the N feature arrays. When receiving encoded data 150 for decoding, the processor(s) 190 can be configured to select a specific bundle decoder from among a plurality of bundle decoders 128 based on the value of N associated with a bundle encoding of the encoded data 150.

[0093] In a particular aspect, decoding the bundle encoding includes processing the bundle encoding, by the joint decoder, to generate an intra-bundle context array. The intra-bundle context array generated by the joint decoder is representative of the intra- bundle context array input to the joint encoder except for losses due to compression and / or transmission.

[0094] Temporal de-intermingling operations can be performed using the intra-bundle context array to generate N feature arrays representing the audio data. The temporal deintermingling operations are configured to reverse operations performed to generate the intra-bundle context array provided to the joint encoder based on the N feature arrays.

[0095] For example, in some embodiments, the temporal de-intermingling operations include performing convolution (e.g., transposed convolution) of the intra-bundle context array from the joint decoder to generate intermingled segments of N feature arrays. In such embodiments, the intermingled segments of N feature arrays can be processed to generate N feature arrays. To illustrate, the intermingled segments of N feature arrays can be subjected to a channel shuffle operation to generate the N feature arrays. The channel shuffle operation is substantially inverse with respect to the channel unshuffle operation described above. In such embodiments, the temporal deintermingling operations also include performing convolution (e.g., transposed convolution) based on the N feature arrays to generate the N feature arrays.

[0096] In some embodiments, the temporal de-intermingling operations include splitting the intra-bundle context array to generate N temporal context arrays; provide the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and perform convolution (e.g., transpose convolution) of each feature array to generate the N feature arrays. In some such embodiments, the recurrent layer(s) include one or more bidirectional recurrent layers, such as one or more bidirectional gated recurrent unit layers.

[0097] The N feature arrays can be provided to the neural synthesizer(s) 130 to generate audio frames (e.g., reproduced representations of the N audio frames 118 represented by the bundle encoding). The audio frames can be stored at the memory 140 (e.g., as part of the audio data 146), can be output to via one or more speakers or other output devices, or both. In some embodiments, the postprocessor s) 132 can process the audio frames before the audio frames are stored or output.

[0098] Thus, the system 100 is configured to communicate audio data using bundle encoding (e.g., by jointly encoding data representing audio frames for a bundle). A technical advantage of jointly encoding data representing all of the audio frames of a bundle is that the bundle encoding so generated can be transmitted via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0099] FIG. 2 is a block diagram of a particular illustrative aspect of a system 200 operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure. The system 200 is an example of the system 100 of FIG. 1 according to a particular embodiment. In FIG. 2, the system 200 includes two or more devices configured to communicate via transmission of encoded audio. FIG. 2 shows an example of the first device 102 that is configured to encode and transmit audio data and a second device 252 (e.g., one of the other device(s) 182 of FIG. 1) that is configured to receive, decode, and use the audio data. For ease of reference herein, the first device 102 is also referred to herein as an encoding device and / or a transmitting device, and the second device 252 is also referred to herein as a decoding device and / or receiving device. Although the system 200 illustrates one transmitting device 102, the system 200 can include more than one transmitting device 102. For example, a two-way communication system may include two devices (e.g., mobile phones), and each of the devices may transmit audio data to and receive audio data from the other device. That is, each device may act as both a transmitting device 102 and a receiving device 252. In another example, a single receiving device 252 can receive audio data from more than one transmitting device 102. Additionally, or alternatively, the system 200 can include more than one receiving device 252. For example, a single transmitting device 102 may transmit (e.g., multicast or broadcast) audio data to multiple receiving devices 252. Thus, the one-to-one pairing of the transmitting device 102 and the receiving device 252 illustrated in FIG. 2 is merely illustrative of one configuration and is not limiting.

[0100] In the example of FIG. 2, the transmitting device 102 includes a plurality of components arranged to obtain audio data 146 (e.g., audio frames 118) and to process the audio data 146 to generate packets (e.g., a first packet 212A and a second packet 212B) that are transmitted over the channel 176. Each packet 212 includes at least one bundle encoding representing N audio frames 118, where N is a value of the bundle frame count 142.

[0101] In FIG. 2, the components of the transmitting device 102 include a codec 120 A (e.g., an instance of the codec 120 of FIG. 1), a packetizer 210, a modem 170A (e.g., an example of the modem 170 of FIG. 1), and a RFE 172A (e.g., an example of the RFE 172 of FIG. 1). In other examples, the transmitting device 102 may include more, fewer,or different components. To illustrate, in some examples, the transmitting device 102 includes one or more audio generation devices configured to generate the audio data 146. Examples of such audio generation devices include, without limitation, microphones, cameras, game engines, media processors (e.g., computer-generated audio engines), augmented reality engines, sensors, or other devices and / or instructions that are configured to generate the audio data 146.

[0102] The audio data 146 includes a sequence (e.g., a time series) of audio frames 118, where each audio frame 118 represents a time-windowed portion of data. The audio frames 118 are provided as an illustrative example of an audio sample generated by the device 102 from the audio data 146 and processed by the codec 120 A. In other examples, an audio sample corresponding to another portion of audio may be generated (e.g., multiple audio frames 118 or a segment or subset of an audio frame 118) from the audio data 146 and processed by the codec 120 A.

[0103] The codec 120A includes the preprocessor s) 122 ("Prepro." in FIG. 2), the feature extractor(s) 124 ("Feature Ext'r" in FIG. 2), and the bundle encoder(s) 126 ("Bundle Enc'r" in FIG. 2) of FIG. 1. The codec 120 A of FIG. 2 optionally also includes one or more additional data generators 206 ("Add'l Data Generator(s)." in FIG. 2). In other examples, the codec 120A may include more, fewer, or different components. To illustrate, in some examples, the codec 120A does not include the preprocessor s) 122. In the same or different examples, the codec 120A does not include the additional data generator(s) 206.

[0104] As explained with reference to FIG. 1, the preprocessor(s) 122 are optional, and are omitted in some embodiments. The preprocessor s) 122, if present, are configured to perform one or more of various operations to prepare the audio data 146 for feature extraction by the feature extractor(s) 124. For example, the preprocessor s) 122 can reduce particular audio components (such as noise or other sounds), can augment particular audio components (e.g., speech), or both. The preprocessor s) 122 can also, or alternatively, generate data (e.g., the additional features or a portion thereof) that influences encoding of the audio data 146. For example, the preprocessor s) 122 can determine whether one or more audio frames 118 include speech and generate a speechindicator that indicates whether speech is detected. In this example, the speech indicator can be used to determine whether particular preprocessing operations are applied to the audio frames 118, can be used to select a particular feature extractor 124 or a particular bundle encoder 126 to use for the audio frames 118, can be used to determine whether to use one of the additional data generator(s) 206, etc.

[0105] The feature extractor(s) 124 are configured to receive the audio frames 118 from an audio generation device, a memory, or the processor(s) 122 and to generate a feature array (e.g., one of N feature arrays 202) representing each audio frame 118. For example, each feature array can include cepstrum data, cepstral coefficients, liftered cepstrum (e.g., determined using discrete cosine transform truncation to exclude pitch information and only capture spectral envelope information), companded (e.g., log) filterbank energies (e.g., determined by applying inverse discrete cosine transform on liftered cepstrum data), filterbank energies computed by uncompanding (e.g., exp) the companded filterbank energies, a full resolution spectrum, linear prediction (LP) coefficients (e.g., determined from speech frames using an autocorrelation method (e.g., Levinson-Durbin) or a covariance method), line spectral frequencies (LSF), line spectral pairs (LSP), or other data representing the spectral envelope of an audio frame.

[0106] In some embodiments, one or more of the feature extractor(s) 124 can generate additional features 204, such as data representing pitch characteristics of one or more audio frames. For example, the pitch characteristics can include fundamental frequency (fO) data, pitch correlation data, pitch lag data, or other frequency-domain data or timedomain data. In some embodiments, the additional features 204 can include other data, such as an indication of whether speech in the audio data 146 is voiced or unvoiced.

[0107] The bundle encoder(s) 126 are configured to jointly encode N feature arrays 202 (e.g., feature arrays representing N audio frames 118) to generate a bundle encoding 208. Operation of the bundle encoder(s) 126 is described in more detail with reference to FIGS. 1, 7, and 8. The bundle encoding 208 is provided to the packetizer 210.

[0108] In some embodiments, the additional data generator(s) 206 obtain additional features 204 from the feature extractor(s) 124, the preprocessor(s) 122, or both, andgenerate additional data 205 that is provided to the packetizer 210. For example, in some embodiments, the bundle encoding 208 represents spectral envelope data associated with the N audio frames 118, and the additional features 204 represent other information descriptive of the N audio frames 118, such as pitch characteristics. In this example, the additional features 204 can be encoded or quantized by the additional data generator(s) 206 and provided to the packetizer 210 as the additional data 205.

[0109] The packetizer 210 is configured to generate packets 212 (e.g., a first packet 212A, a second packet 212B, etc.) based on bundle encodings 208, and optionally additional data 205. For example, the packetizer 210 can add the bundle encoding 208 (and optionally the additional data 205) to a packet payload and attach a header to the payload to form a packet 212. In some embodiments, the packet 212 can include more than one bundle encoding 208. In some embodiments, the header of the packet 212 can include information indicating the bundle frame count 142. For example, a field of the header can indicate a value of N. To illustrate, in some embodiments, the value of N can change from time-to-time as channel conditions change, depending on the audio data 146, or both. In such embodiments, one or more of the packets 212 can include information indicating the value of N. Alternatively, the bundle frame count 142 can be known in advance (e.g., due to data exchanged during channel set up) by the transmitting device 102 and the receiving device 252, in which case the bundle frame count 142 does not need to be indicated in each packet 212.

[0110] The modem 170A is configured to modulate a baseband, according to a particular communication protocol, to generate signals representing the packets 212. The RFE 172A is configured to send the signals representing the packets 212 via the channel 176. In an example, the channel 176 may include a wireline medium, an optical medium, or a wireless medium. To illustrate, the RFE 172A may include or correspond to a wireless transmitter configured to send the signals via free-space propagation of electromagnetic waves.[OHl] In the example of FIG. 2, the receiving device 252 is configured to receive the packets 212 from the transmitting device 102. As noted above, the channel 176 may be lossy. For example, one or more of the packets 212 may be delayed during transmissionor never received at the receiving device 252. The receiving device 252 includes a plurality of components arranged to process the packets 212 that are received and to generate output based on the received packets 212.

[0112] In FIG. 2, the components of the receiving device 252 include a RFE 172B, a modem 170B, a depacketizer 220, one or more buffers 222, a decoder controller 230, a codec 120B, a Tenderer 244, and a user interface device 246. In other examples, the receiving device 252 may include more, fewer, or different components. To illustrate, in some examples, the receiving device 252 includes more than one user interface device 246, such as one or more displays, one or more speakers, one or more haptic output devices, etc.

[0113] The RFE 172B is configured to receive the signals representative of packets 212 and to provide the signals (after initial signal processing, such as amplification, filtering, etc.) to the modem 170B. As noted above, the receiving device 252 may not receive all of the packets 212 sent by the transmitting device 102. Additionally, or in the alternative, the packets 212 may be received in a different order than they are transmitted by the transmitting device 102.

[0114] The modem 170B is configured to demodulate the signals to generate bits representing the received packets 212 and to provide the bits representing the received packets 212 to the depacketizer 220. The depacketizer 220 is configured to extract one or more bundle encodings 226, and optionally additional data 228, from the payload of each received packet 212 and to store the extracted data at the buffer(s) 222. The bundle encoding(s) 226 and the additional data 228 correspond to the bundle encodings 208 and the additional data 205, respectively, as received at the receiving device 252. For example, some or all of the bundle encoding(s) 226 can be identical to the corresponding bundle encoding(s) 208, some or all of the bundle encoding(s) 226 can differ from the corresponding bundle encoding(s) 208 due to the presence of one or more flipped bits, and / or one or more of the bundle encoding(s) 208 can be lost during transmission and not represented among the bundle encoding(s) 226. Similar examples apply to the additional data 228.

[0115] In FIG. 2, the buffer(s) 222 include one or more jitter buffers 224 configured to store the bundle encodings 226 and optionally to store the additional data 228. The buffer(s) 222 store the bundle encodings 226 (and optionally the additional data 228) corresponding to audio frames to enable reordering of the bundle encodings 226 (and any corresponding additional data 228), to allow time for delayed bundle encodings 226 (and any corresponding additional data 228) to arrive, etc.

[0116] In the example illustrated in FIG. 2, a decoder controller 230 retrieves data from the buffer(s) 222 to generate input data 232 for the codec 120B. In some embodiments, the decoder controller 230 also performs buffer management operations, such as managing a depth of the jitter buffer(s) 224, a depth of one or more play out buffer(s) 240, or both.

[0117] To decode a particular audio frame, the decoder controller 230 generates the input data 232 for the bundle decoder 128 ("Bundle Dec'r" in FIG. 2) based on available bundle encodings 226 (and any corresponding additional data 228) associated with a set of N audio frame 118. To generate the input data 232, the decoder controller 230 determines, based on playout sequence information (e.g., a playout time or a playout sequence) associated with the bundle encodings 226, a next set of audio frames that is to be decoded. The decoder controller 230 determines whether a bundle encoding 226 representing the next set of audio frames is stored in the buffer(s) 222. If the bundle encoding 226 associated with the next set of audio frames is available (e.g., stored in the buffer(s) 222), the decoder controller 230 generates the input data 232 based on the bundle encoding 226 and any corresponding additional data stored in the buffer(s) 222.

[0118] The decoder controller 230 provides the input data 232 as input to the codec 120B, and based on the input data 232, the codec 120B generates output data representing a set of audio frames 242 (e.g., N audio frames per bundle encoding 226), which may be stored at the buffer(s) 222 (e.g., at one or more playout buffers 240) as representations of audio frames 242. For example, the input data 232 can be provided to the bundle decoder 128, which in some embodiments, is selected from among a plurality of bundle decoders available at the receiving device 252 based on the value of N. The bundle decoder 128 processes the input data 232 to generate N feature dataarrays 234 (e.g., one feature array per audio frame 118 represented by the bundle encoding 226). The N feature arrays 234 are provided as input to the neural synthesizer 130, which processes the N feature arrays 234 to generate corresponding audio frame data 236. The audio frame data 236 can be further processed by the postprocessor s) 132. The audio frame data 236 (optionally, after processing by the postprocessor s) 132) is stored at the buffer(s) 222 as audio frames 242.

[0119] Each of the audio frames 242 approximates one of the audio frames 118. Generally, the audio frames 242 are not exact replicas of the original audio frames 118 due to losses associated with encoding, quantizing, transmitting, and decoding.However, during normal operation (e.g., when the channel 176 is not too lossy), the audio frames 242 match the audio frame 118 sufficiently that differences during rendering may be below human perceptual limits.

[0120] At a playback time associated with a particular audio frame 118, the Tenderer 244 retrieves a corresponding representation of the audio frame 242 from the buffer(s) 222 and processes the representation of the audio frame 242 to generate an audio signal. The Tenderer 244 provides the audio signal to a user interface device 246 to generate a user perceivable output (e.g., sound) based on the representation of the audio frame 242.

[0121] Thus, the system 200 is configured to communicate audio data using bundle encoding (e.g., by jointly encoding data representing audio frames for a bundle). A technical advantage of jointly encoding data representing all of the audio frames of a bundle is that the bundle encoding so generated can be transmitted via CBR transmissions at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0122] FIG. 3 is a block diagram of a particular illustrative aspect of a system 300 operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure. The system 300 is an example of the system 100 of FIG. 1 according to a particular embodiment. In FIG. 3, the device 102 is configured to receive audio frames 118 and to generate a bundle encoding 208 representing N of theaudio frames 118. A codec 120A used to generate the bundle encoding 208 can be selected from among a plurality of codecs 120 based on the bundle frame count, N. The bundle encoding 208 can be transmitted, via the channel 176, to the remote device 182. The remote device 182 of FIG. 3 includes the codec 120B, which is configured to decode the bundle encoding 208 received from the device 102 to generate output audio 338.

[0123] In the example illustrated in FIG. 3, the device 102 includes examples of the preprocessor(s) 122, the feature extractor(s) 124, and the bundle encoder(s) 126 described with reference to FIG. 1. In FIG. 3, the preprocessor(s) 122, the feature extractor(s) 124, and the bundle encoder(s) 126 are each illustrated as aspects of the codec 120 A; however, in other embodiments, the preprocessor s) 122, the feature extractor(s) 124, or both, are distinct from the codec 120A.

[0124] The preprocessor s) 122 are optional and are omitted in some embodiments. The preprocessor(s) 122 are configured to perform one or more operations to prepare the audio frames 118 for encoding. For example, in FIG. 3, the preprocessor s) 122 include a denoiser 302 configured to reduce noise in the audio frames 118. In other examples, the preprocessor s) 122 include components configured to perform other preprocessing operations, such as audio classification, speech enhancement, voice detection, etc. In some embodiments, one or more of the preprocessor(s) 122 are ML-based. To illustrate, the preprocessor s) 122 can include an ML-based audio enhancement component.

[0125] The feature extractor(s) 124 are configured to generate feature arrays 202 for the bundle encoder(s) 126, where the feature arrays 202 include features extracted from or generated based on the audio frames 118. In some embodiments, the feature arrays 202 include one feature array per audio frame 118. Thus, in such embodiments, N feature arrays represent N audio frames 118. One or more of the feature extractor(s) 124 can be ML-based.

[0126] In FIG. 3, the feature extractor(s) 124 include a spectral envelope feature extractor 304 that is configured to generate feature data (e.g., feature arrays 202) representing a spectral envelope of one of the audio frames 118. For example, thespectral envelope feature extractor 304 may generate cepstrum data, cepstral coefficients, liftered cepstrum (e.g., determined using discrete cosine transform truncation to exclude pitch information and only capture spectral envelope information), companded (e.g., log) filterbank energies (e.g., determined by applying inverse discrete cosine transform on liftered cepstrum data), filterbank energies computed by uncompanding (e.g., exp) the companded filterbank energies, a full resolution spectrum, linear prediction (LP) coefficients (e.g., determined from speech frames using an autocorrelation method (e.g., Levinson-Durbin) or a covariance method), line spectral frequencies (LSF), line spectral pairs (LSP), or other data representing the spectral envelope of the audio frame 118.

[0127] In FIG. 3, the feature extractor(s) 124 also include a pitch feature extractor 306 configured to generate additional features 204 (e.g., additional feature data) representing pitch characteristics of one of the audio frames 118. The pitch feature extractor 306 can generate fundamental frequency (fO) data, pitch correlation data, pitch lag data, or other frequency-domain data or time-domain data. In some embodiments, the pitch feature extractor 306 can also generate other data, such as an indication of whether speech in an audio frame 118 is voiced or unvoiced. The pitch feature extractor 306 is optional and is omitted in some embodiments.

[0128] In FIG. 3, the codec 120 A is illustrated as including two feature extractors 124 including the spectral envelope feature extractor 304 configured to provide the feature arrays 202 to the bundle encoder 126 and the pitch feature extractor 306 configured to provide the additional features 204 to the additional data generator(s) 206, where the bundle encoder 126 and the additional data generator(s) 206 each represent encoders or quantizers 310 (also referred to herein as "encoder / quantizers"). In other embodiments, two or more feature extractors 124 generate input data for one of the encoder / quantizers 310. For example, two or more feature extractors 124 can generate feature arrays 202 for each audio frame 118. In this example, the two or more feature arrays 202 for a particular audio frame 118 can be combined as input to the bundle encoder 126. In some embodiments, input data from a single feature extractor 124 is provided to two or more of the encoder / quantizers 310. For example, in some embodiments, the pitch featureextractor 306 is omitted and the input data to the additional data generator(s) 206 is provided by the spectral envelope feature extractor 304.

[0129] In some embodiments, one or more of the encoder / quantizer(s) 310 is an ML- based encoder / quantizer. For example, in FIG. 3, the bundle encoder 126 is illustrated as a feedback recurrent autoencoder that includes a joint encoder 312, a bottleneck 316, and a joint decoder 318. Optionally, the feedback recurrent autoencoder can be configured to use multiple description coding techniques (or other forward error correction techniques) to reduce the effects of packet loss or delay during transmission. The joint encoder 312 includes one or more layers of a machine-learning model (e.g., one or more gated recurrent unit layers, one or more self-attention layers, etc.) that are configured to dimensionally reduce input data (e.g., a set of N feature arrays 202) to generate a dimensionally reduced representation of the input data, which is provided to the bottleneck 316. The bottleneck 316 is configured to generate the bundle encoding 208 based on the dimensionally reduced representation of the N feature arrays 202. The bottleneck 316 may include, for example, one or more fully connected layers, a quantizer, a codebook, other components, or a combination thereof. The bottleneck 316 is configured to provide output data (e.g., the bundle encoding 208) to the joint decoder 318.

[0130] The joint decoder 318 is configured to dimensionally expand the output data from the bottleneck 316 to generate an approximate reproduction of the input data (e.g., the N feature arrays 202) provided to the joint encoder 312. Generally, the joint decoder 318 is configured to perform operations approximately inverse with respect to operations performed by the joint encoder 312. Thus, the joint decoder 318 can include similar layers as the joint encoder 312 (e.g., one or more gated recurrent unit layers and one or more self-attention layers, etc.) arranged in the opposite order relative to the joint encoder 312.

[0131] The joint decoder 318 is also configured to provide feedback (based on the reproduced input data or internal states of the joint decoder 318) to earlier stages of the joint decoder 318, to the joint encoder 312, or both. Feedback from thejoint decoder 318 to an earlier stage of thejoint decoder 318, to thejoint encoder 312, or both,improves encoding and / or decoding of the audio frames 118 by enabling the bundle encoder 126 to account for temporal relationships in the audio frames 118. For example, the feature arrays 202 can include a time sequence of feature arrays, with a first set of N feature arrays representing a first set of N audio frames 118, and a second set of N feature arrays representing a second (e.g., a subsequent) set of N audio frames 118. In this example, the feedback representing internal states of the joint decoder 318 associated with decoding associated with the first set of N feature arrays can be provided to the joint encoder 312 and / or to an earlier stage of the joint decoder 318 to facilitate encoding and / or decoding of the second set of N feature arrays that is subsequent to the first set of N feature arrays. In some embodiments, the joint decoder 318 is enabled during training of the autoencoder and is optional or disabled during inference.

[0132] The additional data generator(s) 206 can include one or more additional feedback recurrent autoencoders, other ML-based encoder(s) and / or quantizer(s) that are configured to generate encoded data 322 based on the additional features 204. To illustrate, the additional data generator(s) 206 can include a feedforward neural network, a convolutional neural network, a self-attention network, or combinations or variants thereof. Alternatively, the additional data generator(s) 206 can be configured to use procedural operations to generate the encoded data 322. Additionally, or alternatively, one or more of additional data generator(s) 206 can be configured to perform forward error correction operations to reduce the effect of lost or delayed packets. For example, the forward error correction operations can include using multiple description coding or full redundancy.

[0133] The device 102 is configured to transmit the encoded data 150, via the channel 176, to the remote device 182. The encoded data 150 includes the bundle encoding 208 and optionally the encoded data 322. Optionally, the device 102 is also configured to transmit other data, such as information indicating the bundle frame count 142.

[0134] The remote device 182 includes the codec 120B. In FIG. 3, the codec 120B includes one or more bundle decoders 128 and a neural synthesizer 130. The bundle decoder(s) 128 are configured to decode the bundle encoding 208. In someembodiments, the codec 120B includes multiple bundle decoders 128 configured to decode bundle encodings 208, each related to a specific value of N. Each of the bundle decoder(s) 128 includes a joint decoder 324 of a feedback recurrent autoencoder. In some aspects, the joint decoder 324 is based on (e.g., is a copy of) the joint decoder 318. Optionally, the codec 120B also includes one or more decoder / dequantizers 332 configured to process a portion of the encoded data 150 (e.g., the encoded data 322) to generate additional input data 334 for neural synthesizer 130.

[0135] The neural synthesizer 130 is configured to process the N feature arrays 234 from the bundle decoders 128 (and optionally the additional input data 334) to generate reconstructed audio data (e.g., audio frame data 236) approximating the audio frames 118 used to generate the bundle encoding 208. The neural synthesizer 130 can include or correspond to any machine-learning based audio or speech synthesizer, such as a neural homomorphic vocoder (NHV), an LPCNet network, a WaveNet network, a WaveRNN network, etc. The additional data generator(s) 206 and the decoder / dequantizers 332 are optional and are omitted in some embodiments. In some of these embodiments, the device 102 sends the additional features 204 (e.g., without encoding) to the remote device 182, and the neural synthesizer 130 generates the audio frame data 236 based on the N feature arrays 234 and the additional features 204. In other embodiments, the device 102 does not generate or send the additional features 204, and the neural synthesizer 130 generates the audio frame data 236 independently of the additional features 204.

[0136] Optionally, the audio frame data 236 can be further processed, such as by a linear prediction (LP) synthesis filter 336 to generate output audio 338. The LP synthesis filter 336 is optional and is omitted in some embodiments. In such embodiments, the remote device 182 can use the audio frame data 236 directly, or after other processing operations, to generate the output audio 338.

[0137] FIG. 4 is a block diagram of a particular illustrative aspect of a system 400 operable to decode audio data of a bundle encoding, in accordance with some examples of the present disclosure. In particular, FIG. 4 illustrates aspects of operation of a particular example of the remote device 182 of any of FIGS. 1-3. In FIG. 4, the remotedevice 182 is configured to receive the encoded data 150 (e.g., the bundle encoding 208 and optionally encoded data 322), via transmissions over the channel 176 from the device 102 of any of FIGS. 1-3.

[0138] In FIG. 4, the neural synthesizer 130 of the codec 120B includes a neural homomorphic vocoder (NHV). An NHV is based on a two-state excitation model of the human vocal tract, which enables the NHV to generate audio data representing speech with high-fidelity based on low-bit rate data (e.g., the encoded data 150). In FIG. 4, the neural synthesizer 130 includes a neural network filter estimator 402 configured to process the N feature arrays 234 from the bundle decoder 128. In some embodiments, the N feature arrays 234 includes N sets of 80 log-Mel features. The neural network filter estimator 402 is configured to generate filter parameters 404, 406 for one or more filters. In FIG. 4, the one or more filters include a harmonic linear time variant (LTV) filter 418, a noise LTV filter 424, or both. In some embodiments, the harmonic LTV filter 418, the noise LTV filter 424, or both, are implemented in the time domain, as a convolution operation between an input signal and the filter impulse response (e.g., the filter parameters 404, 406). In other embodiments, the harmonic LTV filter 418, the noise LTV filter 424, or both, are implemented in the frequency domain by multiplying the filter frequency response (e.g., spectrum or fast Fourier transform (FFT) of the impulse response) with the spectrum (e.g., FFT) of the input signal. Those skilled in the art will appreciate that other possibilities exist for the filter implementations, such as cepstral domain and other representations. All such possibilities are encompassed within the scope of the present disclosure. The filter parameters 404, 406 can include cepstrum data, frequency response data, impulse response data, difference equation coefficients, poles and zeros, etc., depending on the specific embodiment.

[0139] The neural synthesizer 130 of FIG. 4 includes a pulse train generator 414 coupled to the harmonic LTV filter 418. The pulse train generator 414 is configured to provide a pulse train 416 to the harmonic LTV filter 418. The pulse train 416 can be determined based on, for example, fundamental frequency data, pitch lag, a voiced / unvoiced classification, or other data derived from the encoded data 150. The harmonic LTV filter 418 is configured to process the pulse train 416 using harmonic filter parameters 406 to generate a harmonic component (HC) 432.

[0140] The neural synthesizer 130 of FIG. 4 also includes a noise generator 420 coupled to the noise LTV filter 424. The noise generator 420 is configured to generate a random noise signal 422. The noise LTV filter 424 is configured to process the random noise signal 422 using noise filter parameters 404 to generate a noise component (NC) 434.

[0141] A combiner 430 is configured to combine the noise component 434 and the harmonic component 432 to generate audio frame data 236 (e.g. a synthesized speech signal). In some embodiments, the neural synthesizer 130 is configured to convert the noise component 434, the harmonic component 432, or both, to a common domain for combining by the combiner 430. For example, if the harmonic LTV filter 418 operates in the frequency domain and the noise LTV filter 424 operates in the time domain, the neural synthesizer 130 may convert frequency domain output of the harmonic LTV filter 418 to the time domain to generate the harmonic component 432. As another example, the neural synthesizer 130 may convert the noise component 434, the harmonic component 432, or both, to a domain other than the time domain.

[0142] Optionally, the audio frame data 236 can be further processed to generate the output audio 338. For example, in FIG. 4, the audio frame data 236 (e.g., the synthesized speech signal) is provided to a post-filter 436 (e.g., a linear time invariant filter), the LP synthesis filter 336, or both. In some embodiments, the audio frame data 236 is used directly as the output audio 338. In such embodiments, the post-filter 436 and the LP synthesis filter 336 are omitted.

[0143] FIG. 5 is a block diagram of a particular illustrative aspect of a system 500 operable to decode audio data of a bundle encoding, in accordance with some examples of the present disclosure. In particular, FIG. 5 illustrates aspects of operation of another particular example of the remote device 182 of any of FIGS. 1-3. In FIG. 5, the remote device 182 is configured to receive the encoded data 150 (e.g., the bundle encoding 208 and optionally the encoded data 322), via transmissions over the channel 176 from the device 102 of FIGS. 1 or 2. In some embodiments, the remote device 182 includes the codec 120B, which is configured to decode the encoded data 150.

[0144] In FIG. 5, the codec 120B includes the bundle decoder 128, an optional decoder / dequantizer 332, and an example of the neural synthesizer 130. In FIG. 5, the neural synthesizer 130 of the codec 120B includes an LPCNet decoder. The LPCNet decoder is a low bitrate speech synthesis model that separates excitation modeling and spectral envelope modeling to improve performance. In particular, spectral envelope modeling of speech is performed using linear prediction, which enables use of much of the capacity of the neural synthesizer 130 for excitation modeling. This arrangement enables the LPCNet to generate audio data representing speech with high-fidelity based on low-bit rate data (e.g., the encoded data 150). The neural synthesizer 130 includes a frame rate network 510, a sample rate network 514, a linear prediction coefficient (LPC) estimator 502, a linear predictor 506, a sampler 518, and a combiner 524.

[0145] The LPC estimator 502 is configured to process output data (e.g., N feature arrays 234) from the bundle decoder 128 to generate LP coefficients 504. The LP coefficients 504 are provided to the linear predictor 506. The linear predictor 506 is configured to model speech as the output of a linear filter that predicts a current sample (corresponding to LP data 508) based on previous samples (represented by a feedback signal 526) and the LP coefficients 504. The feedback signal 526 represents a previous output audio frame 236 after a delay z’1.

[0146] The frame rate network 510 is configured to process output data (e.g., the N feature arrays 234) from the bundle decoder 128 (and optionally the input data 334 from the decoder / dequantizer 332) to generate conditioning features 512 that are held constant for the duration of each frame. The frame rate network 510 can include various ML layers, such as two or more convolution layers coupled in a residual connection arrangement to one or more fully connected layers.

[0147] The sample rate network 514 is configured to generate a probability distribution 516 of an excitation signal (e.g., an LP residual signal) based on the conditioning features 512, the LP data 508, the feedback signal 526, and a feedback signal 522 (representing a previously sampled excitation signal, e.g., the excitation signal 520 after a delay Z'1). In a particular embodiment, the sample rate network 514 is configured to combine various input vectors to generate an embedding vector, which is processed byrecurrent layers (e.g., gated recurrent units), one or more fully connected (FC) layers (e.g. dual FC layers), and a softmax layer to generate the probability distribution 516.

[0148] The probability distribution 516 of the excitation signal is sampled by the sampler 518 to generate an excitation signal 520. The excitation signal 520 and the LP data 508 are combined by the combiner 524 to generate the audio frame data 236. Optionally, the audio frame data 236 can be provided to the postprocessor s) 132 to generate output audio 338. In other embodiments, the postprocessor(s) 132 are omitted (or bypassed in some operational modes) and the audio frame data 236 is used as the output audio 338.

[0149] FIG. 6 is a block diagram of a particular illustrative aspect of a system 600 operable to communicate audio data using bundle encoding, in accordance with some examples of the present disclosure. The system 600 is an example of any of the systems of FIGS. 1-3 according to a particular embodiment. The system 600 includes many of the same features and aspects as described with reference to FIG. 3. In FIG. 6, the bundle encoder 126 includes a plurality of joint encoders, such as a joint encoder 604, a joint encoder 606, and a joint encoder 608. Similarly, the bundle decoder 128 includes a plurality of joint decoders, such as a joint decoder 612, a joint decoder 614, and a joint decoder 616.

[0150] Each of the joint encoders 604-608 is associated with a particular value N of the bundle frame count 142. For example, the joint encoder 604 is associated with a value of N = 2 (e.g., two audio frames per bundle), the joint encoder 606 is associated with a value of N = 3 (e.g., three audio frames per bundle), and the joint encoder 608 is associated with a value of N = 4 (e.g., four audio frames per bundle). Thus, the joint encoder 604 is configured to generate a bundle encoding 208 based on two (N=2) feature arrays 202 representing two audio frames 118, the joint encoder 606 is configured to generate a bundle encoding 208 based on three (N=3) feature arrays 202 representing three audio frames 118, and the joint encoder 608 is configured to generate a bundle encoding 208 based on four (N=4) feature arrays 202 representing four audio frames 118.

[0151] Likewise, each of the joint decoders 612-616 is associated with a particular value N of the bundle frame count 142, and each of the joint decoders 612-616 is configured to decode a bundle encoding 208 generated by one of the joint encoders 604- 608. For example, the joint decoder 612 is associated with a value of N = 2 (e.g., two audio frames per bundle) to decode a bundle encoding 208 generated by the joint encoder 604, the joint decoder 614 is associated with a value of N = 3 (e.g., three audio frames per bundle) to decode a bundle encoding 208 generated by the joint encoder 606, and the joint decoder 616 is associated with a value of N = 4 (e.g., four audio frames per bundle) to decode a bundle encoding 208 generated by the joint encoder 608. Thus, the joint decoder 612 is configured to decode a bundle encoding 208 to generate two feature arrays 234 for two frames of the audio frame data 236, the joint decoder 614 is configured to decode a bundle encoding 208 to generate three feature arrays 234 for three frames of the audio frame data 236, and the joint decoder 616 is configured to decode a bundle encoding 208 to generate four feature arrays 234 for four frames of the audio frame data 236.

[0152] The bundle encoder 126 including three joint encoders 604-608 is merely one example. In other examples, the bundle encoder 126 includes more than three joint encoders or fewer than three joint encoders (e.g., two joint encoders). Likewise, the joint encoders 604-608 corresponding to values of N = 2, N = 3, and N = 4 is merely one example. In other examples, the bundle encoder 126 includes joint encoders for more, fewer, or different values of N. Further, since the joint decoders 612-616 are configured to decode bundle encodings 208 from the joint encoders 604-608, in examples in which the joint encoders 604-608 include more, fewer, or different joint encoders (e.g., joint encoders for different values of N), the bundle decoder 128 can include corresponding joint decoders.

[0153] In FIG. 6, the codec 120 A includes a switch 602 that is configured to select a particular one of the joint encoders 604-608 to be used based on the bundle frame count 142 (i.e., based on the value of N), and the codec 120B includes a switch 610 that is configured to select a particular one of the joint decoders 612-616 to be used based on the bundle frame count 142. The bundle frame count 142 can be indicated in a packet that includes the encoded data 150 (e.g., in a header of the packet or in a payload of thepacket), or can be communicated between the devices 102, 182 via out of band communication. Alternatively, the device 102 can determine the bundle frame count 142 used to generate the encoded data 150, and the remote device 182 can determine the bundle frame count 142 based on characteristics of the encoded data 150. For example, a first number of bits can be used to encode two audio frames 118, and a second number of bits can be used to encode three audio frames 118. In this example, the remote device 182 can determine the bundle frame count 142 (and therefore which joint decoder 612- 616 to use) based on a number of bits per bundle encoding 208. In this example, packets communicating the encoded data 150 do not include an explicit indication of the bundle frame count 142.

[0154] As another example, the bundle frame count 142 can be determined predetermined, such as by a service provider associated with the channel 176 (e.g., a mobile data provider). To illustrate, a first geographic area can be associated with a bundle frame count 142 of N = 2, and a second geographic area can be associated with a bundle frame count 142 of N = 3. In this illustrative example, if the device 102, the device 182, or both, are in the first geographic area, a bundle frame count 142 of N = 2 can be used as prescribed by the service provider.

[0155] As another example, the bundle frame count 142 can be determined during set up of a communication session between the device 102 and the remote device 182. To illustrate, during set up of the communication session, the device 102 and the remote device 182 can exchange data indicating capabilities of the devices 102, 182 (e.g., which values of N are supported), channel conditions associated with the channel 176, etc., and one or both of the devices 102, 182 can select a bundle frame count 142 to be used during the communication session.

[0156] As another example, the bundle frame count 142 can be determined (e.g., modified or updated) during a communication session between the device 102 and the remote device 182. To illustrate, during the communication session, the channel conditions associated with the channel 176, geographic locations of the device 102, 182, or both, can change. In this illustrative example, one or both of the devices 102, 182 can select a new (e.g., updated) bundle frame count 142 to be used.

[0157] In some embodiments, combinations of the above examples (or others) can be used to determine the bundle frame count 142 used at a particular time. For example, the devices 102, 182 can begin a communication session by using a bundle frame count 142 specified by a service provider, can subsequently exchange data to select a different bundle frame count 142, and can later change the bundle frame count 142 again due to channel conditions or other factors.

[0158] In the example illustrated in FIG. 6, the same feature extractor(s) 124 are shown providing feature arrays 202 to the bundle encoder 126 irrespective of which joint encoder 604-608 is selected. In other examples, the switch 602 can be disposed before the feature extractor(s) 124 such that different feature extractor(s) 124 can be used for different bundle frame counts 142.

[0159] FIG. 7 is a diagram of an illustrative aspect of operations associated with bundle encoding, in accordance with some examples of the present disclosure. In particular, the diagram in FIG. 7 shows an example of a system 700 configured to jointly encode a set of N feature arrays 202 to generate a bundle encoding 208. The system 700 is configured to use various machine-learning operations to generate the bundle encoding 208. In particular, the system 700 is arranged as an autoencoder, e.g., a feedback recurrent autoencoder including an encoder portion 730, a decoder portion 740, and the bottleneck 316 between the encoder portion 730 and the decoder portion 740. In some embodiments, the bundle encoder 126 of any of FIGS. 1-3 or 6 includes the encoder portion 730 and the bottleneck 316. Optionally, the bundle encoder 126 of any of FIGS. 1-3 or 6 also includes the decoder portion 740. In some embodiments, the bundle decoder 128 of any of FIGS. 1-6 includes the decoder portion 740 of the system 700.

[0160] The system 700 is configured to generate a bundle encoding 208 representing N feature arrays 202. In general, an architecture of the system 700 is configured for a specific value of N. To illustrate, in the example illustrated in FIG. 7, the system 700 is configured to generate a bundle encoding 208 representing a set of five (N = 5) feature arrays, including a feature array 702A, a feature array 702B, a feature array 702C, a feature array 702D, a feature array 702E. In this example, the feature arrays 702A-702Erepresent features extracted from five audio frames of a sequence of audio frames 118 of audio data 146 of FIG. 1.

[0161] In the example illustrated in FIG. 7, the encoder portion 730 includes an input 732, one or more temporal intermingling layers 734, a combiner 708, and a joint encoder 710. The input 732 of the encoder portion 730 is configured to receive as input the five feature arrays 702A-702E. In FIG. 7, the input 732 include one or more sets of convolution layers 704. For example, the input 732 can include a set of convolution layers 704 for each feature array 702A-702E, such as a set of convolution layers 704A for the feature array 702A (e.g., a feature array representing a first audio frame for the bundle encoding 208), a set of convolution layers 704B for the feature array 702b (e.g., a feature array representing a second audio frame for the bundle encoding 208), a set of convolution layers 704C for the feature array 702C (e.g., a feature array representing a third audio frame for the bundle encoding 208), a set of convolution layers 704D for the feature array 702D (e.g., a feature array representing a fourth audio frame for the bundle encoding 208), and a set of convolution layers 704E for the feature array 702E (e.g., a feature array representing a fifth audio frame for the bundle encoding 208). For embodiments in which the system 700 is configured for a different value of N, the input 732 includes a correspondingly different number of sets (e.g., N sets) of convolution layers 704.

[0162] N intermediate feature arrays 733 from the sets of convolution layers 704 can be provided to the temporal intermingling layer(s) 734. The temporal intermingling layer(s) 734 are configured to perform temporal intermingling of the N intermediate feature arrays 733 to generate N temporal context arrays 735. A "temporal context array" refers to data that represents a feature array 702 and a corresponding temporal context of the feature array 702. For example, in FIG. 7, the temporal intermingling layer(s) 734 include bi-directional gated recurrent unit (GRU) layers 706, which are configured to encode, in a temporal context array 735 for a particular feature array 702, at least a portion of the particular feature array 702 and additionally information about one or more features arrays that precede and / or follow the feature array 702. To illustrate, a temporal context array 735B for the feature array 702B includes at least a first portion of the intermediate feature array 733B, and additionally include at least a second portionof one or more feature arrays (e.g., the intermediate feature array 733A) that precede the intermediate feature array 733B, at least a third portion of one or more features arrays (e.g., the intermediate feature array 733 C) that follow the intermediate feature array 733B, or both. In a particular aspect, the first portion is larger than each of the second portion and the third portion. In this example, the N temporal context arrays 735 represent hidden states of the bidirectional GRU layers. Although the temporal intermingling layer(s) 734 include bidirectional GRU layers 706 in FIG. 7, in other examples, the temporal intermingling layer(s) 734 can include other types of machinelearning layers, such as unidirectional GRU layers, LTSM layers, self-attention layers, etc.

[0163] The combiner 708 is configured to combine the temporal context arrays 735 to generate an intra-bundle context array 736. In a particular aspect, a single intra-bundle context array 736 is formed from N temporal context arrays 735 and thus represents the N feature arrays 202. The combiner 708 can be configured to perform any of various combination operations, such as concatenation, pooling, or other mathematical or logical combination operations to combine the N temporal context arrays 735.

[0164] The joint encoder 710 and the bottleneck 316 together are configured to generate the bundle encoding 208 based on the intra-bundle context array 736. For example, the joint encoder 710 can include one or more machine-learning layers, which can include, for example, GRU layer(s), LSTM layer(s), other recurrent layers, fully connected layer(s), convolution layer(s), self-attention layer(s), pooling layer(s), other machinelearning layer(s), or a combination thereof, and the bottleneck 316 can include one or more fully connected layers, a quantizer, other machine-learning layer(s), or a combination thereof. As one example, the joint encoder 710 can include GRU layer(s) that are configured to receive a feedback signal 317A from a decoder portion 740 (e.g., a local decoder portion 740 or a decoder portion 740 of a remote device) in addition to the intra-bundle context array 736 as input, and to provide output representing hidden states of the GRU layer(s) to the bottleneck 316. The bottleneck 316 is configured to quantize or otherwise prepare the output of the joint encoder 710 for transmission to another device as the bundle encoding 208.

[0165] In a particular aspect, the bundle encoding 208 includes a particular number of bits, such M bits, which together represent N audio frames (e.g., audio frames 118 of FIG. 1). The number of bits (e.g., M) used to represent a set of audio frames (e.g., N frames) can depend on the number of audio frames represented. In the example illustrated in FIG. 7, the system 700 is configured to encode data representing five audio frames (based on input of five feature arrays), resulting in an M-bit bundle encoding 208. However, in an example of the system 700 that is configured to encode data representing three audio frames (e.g., N = 3), the bundle encoding 208 may include a different number of bits.

[0166] In some embodiments, associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder (e.g., the system 700 or the encoder portion 730 of the system 700). In such embodiments, it is generally the case that no specific bit of the bundle encoding 208 represents content of just one of the audio frames. Rather, the value of each of the M bits is usually associated, at least to some extent, with content of each of the audio frames (though due to training of the ML encoder, more direct relationships between specific bits and corresponding audio frames can exist due simply to arrangements of ML weights assigned during training).

[0167] During training of the system 700 (and optionally during inference at a transmitting device using the system 700), the bundle encoding 208 is provided to the decoder portion 740 of the system 700. The decoder portion 740 is configured to perform operations substantially inverse with respect to operations performed by the encoder portion 730. For example, in FIG. 7, the decoder portion 740 includes a joint decoder 720, a splitter 722, one or more temporal de-intermingling layers 744, and an output 746.

[0168] The joint decoder 720 is configured to perform inverse operations with respect to the joint encoder 710 to generate an intra-bundle context array 741 based on the bundle encoding 208. The joint decoder 720 can include, for example, one or more machine-learning layers, such as, but not limited to, GRU layer(s), LSTM layer(s), other recurrent layers, fully connected layer(s), convolution layer(s), self-attention layer(s),pooling layer(s), other machine-learning layer(s), or a combination thereof. The joint decoder 720 can also be configured to generate a feedback signal 317B, which is provided to the encoder portion 730 for use during encoding of a subsequent set of N feature arrays 202.

[0169] The splitter 722 is configured to generate a set of N temporal context arrays 742 based on the intra-bundle context array 741. For example, the splitter 722 can perform inverse operations with respect to the combiner 708 to separate the N temporal context arrays 742.

[0170] The temporal de-intermingling layer(s) 744 are configured to generate N feature arrays 743 based on the N temporal context arrays 742. For example, the temporal deintermingling layer(s) 744 can perform inverse operations with respect to the temporal intermingling layer(s) 734. To illustrate, the temporal de-intermingling layer(s) 744 in FIG. 7 include bidirectional GRU layers 724 configured to perform operations inverse with respect to the bidirectional GRU layers 706 of the temporal intermingling layer(s) 734. It is noted that the operations performed by the temporal de-intermingling layer(s) 744 are conceptually inverse relative to the operations performed by the temporal intermingling layer(s) 734, rather than literally inverse. That is, it is a purpose of the temporal de-intermingling layer(s) 744 to generate N feature arrays 743 based on the N temporal context arrays 742; however, the specific operations performed by the temporal de-intermingling layer(s) 744 are based on training of the entire system 700. Such training is an optimization type process, which does not enforce application of inverse weights among any two layers of the system 700. Accordingly, while it may be conceptually helpful to think of the temporal de-intermingling layer(s) 744 as inverse with respect to the temporal intermingling layer(s) 734, the operations performed by the temporal de-intermingling layer(s) 744 and the temporal intermingling layer(s) 734 are trained, and unlikely to literally apply inverse weights.

[0171] The output 746 includes one or more layers configured to perform further processing of the N intermediate feature arrays 743 to generate N feature arrays 234. For example, the output 746 of FIG. 7 includes one or more sets of convolution layers 726, such as a set of convolution layers 726A to generate a feature array 728A (e.g., afeature array representing a reproduction of a first audio frame), a set of convolution layers 726B to generate a feature array 728B (e.g., a feature array representing a reproduction of a second audio frame), a set of convolution layers 726C to generate a feature array 728C (e.g., a feature array representing a reproduction of a third audio frame), a set of convolution layers 726D to generate a feature array 728D (e.g., a feature array representing a reproduction of a fourth audio frame), and a set of convolution layers 726E to generate a feature array 728E (e.g., a feature array representing a reproduction of a fifth audio frame). In some embodiments, each of the sets of convolution layers 726 is configured to perform transpose convolution to generate the respective feature array 728 of the N feature arrays 234.

[0172] FIG. 8 is a diagram of an illustrative aspect of operations associated with bundle encoding, in accordance with some examples of the present disclosure. In particular, the diagram in FIG. 8 shows an example of a system 800 configured to jointly encode a set of N feature arrays 702 to generate a bundle encoding 208. The system 800 is configured to use various machine-learning operations to generate the bundle encoding 208. In particular, the system 800 is arranged as an autoencoder, e.g., a feedback recurrent autoencoder including an encoder portion 830, a decoder portion 840, and the bottleneck 316 between the encoder portion 830 and the decoder portion 840. In some embodiments, the bundle encoder 126 of any of FIGS. 1-3 or 6 includes the encoder portion 830 and the bottleneck 316. Optionally, the bundle encoder 126 of any of FIGS. 1-3 or 6 also includes the decoder portion 840. In some embodiments, the bundle decoder 128 of any of FIGS. 1-6 includes the decoder portion 840 of the system 800.

[0173] The system 800 is configured to generate the bundle encoding 208 representing N feature arrays 202. In general, an architecture of the system 800 is configured for a specific value of N. To illustrate, in the example illustrated in FIG. 8, the system 800 is configured to generate a bundle encoding 208 representing a set of five (N = 5) feature arrays, including the feature arrays 702A-702E described with reference to FIG. 7. In this example, the feature arrays 702A-702E represent features extracted from five audio frames of a sequence of audio frames 118 of audio data 146 of FIG. 1.

[0174] In the example illustrated in FIG. 8, the encoder portion 830 includes one or more temporal intermingling layers 802, one or more convolution layers 804, and a joint encoder 806. In the system 800, the temporal intermingling layer(s) 802 are configured to receive input including the N feature arrays 202 (e.g., the five feature arrays 702A- 702E in FIG. 8) and to generate an intermediate feature array 833 (e.g., a temporally convolved feature array) based on the N feature arrays 202. In FIG. 8, the temporal intermingling layers 802 are configured to perform temporal unshuffling convolution operations (e.g., intra-bundle context modeling) using the feature arrays 702 to generate the temporally convolved feature array 833. For example, in FIG. 8, the feature arrays 702 are temporally related to one another. To illustrate, the feature array 702A represents a first audio frame, the feature array 702B represents a second audio frame subsequent to the first audio frame, the feature array 702C represents a third audio frame subsequent to the second audio frame, and so forth. Thus, the temporal unshuffling convolution operations applied to the N feature arrays 202 intermingle content of temporally related input data from the feature arrays 702 to generate a single temporally convolved (e.g., intermingled) feature array 833.

[0175] The temporal intermingling layers 802 can operate in any of several different manners, depending on the specific embodiment. For example, in some embodiments, the temporal intermingling layers 802 can include stacked convolution layers arranged such that each feature array 702 is convolved individually (e.g., as a ID convolution), and portions of the intermediate feature arrays are subsequently intermingled and subjected to one or more additional convolution operations.

[0176] In some embodiments, the temporal intermingling layers 802 can use channel shuffle / channel unshuffle operations, in which case the temporal intermingling layers 802 can include one or more channel shuffle and / or channel unshuffle units. For example, the temporal intermingling layers 802 can include one or more pointwise group convolution layers, a channel shuffle layer, one or more depthwise convolution layers, and one or more pointwise convolution layers. In this example, the pointwise group convolution layer(s) operate on individual channels (e.g., individual feature arrays 702) to generate convolved channel input (e.g., N intermediate feature arrays). The channel shuffle layer mixes the N intermediate feature arrays with one another. Thechannel shuffle layer divides each of N intermediate feature arrays into several subgroups and recombines the sub-groups to form N shuffled feature arrays, where each shuffled feature array includes sub-groups from each of the N intermediate feature arrays. Channel unshuffling operations reverse channel shuffle operations. The depthwise convolution layer(s) and pointwise convolution layer(s) operate to combine the N shuffled feature arrays to form the temporally intermediate feature array 833. In some embodiments, the channel shuffle / channel unshuffle operations can include a residual connection and a combiner following the final pointwise convolution layer.

[0177] In some embodiments, the convolution layer(s) 804 generate an intra-bundle context array 835 based on the temporally convolved feature array 833. Alternatively, in some embodiments, the convolution layer(s) 804 are omitted or incorporated into the temporal intermingling layer(s) 802, in which case the temporally convolved feature array 833 is used as the intra-bundle context array 835.

[0178] The joint encoder 806 and the bottleneck 316 together are configured to generate the bundle encoding 208 based on the intra-bundle context array 835. For example, the joint encoder 806 can include one or more machine-learning layers, which can include, for example, GRU layer(s), LSTM layer(s), other recurrent layers, fully connected layer(s), convolution layer(s), self-attention layer(s), pooling layer(s), other machinelearning layer(s), or a combination thereof, and the bottleneck 316 can include one or more fully connected layers, a quantizer, other machine-learning layer(s), or a combination thereof. As one example, the joint encoder 806 can include GRU layer(s) that are configured to receive a feedback signal 317A from the decoder portion 840 and the intra-bundle context array 835 as input, and to provide output representing hidden states of the GRU layer(s) to the bottleneck 316. The bottleneck 316 is configured to quantize or otherwise prepare the output of the joint encoder 806 for transmission to another device as the bundle encoding 208.

[0179] As described with reference to FIG. 7, the bundle encoding 208 includes a particular number of bits, such M bits, which together represent N audio frames (e.g., audio frames 118 of FIG. 1). The number of bits (e.g., M) used to represent a set of audio frames (e.g., N frames) can depend on the number of audio frames represented. Inthe example illustrated in FIG. 8, the system 800 is configured to encode data representing five audio frames (based on input of five feature arrays), resulting in an M- bit bundle encoding 208. However, in an example of the system 800 that is configured to encode data representing three audio frames (e.g., N = 3), the bundle encoding 208 may include a different number of bits.

[0180] In some embodiments, associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder (e.g., the system 800 or the encoder portion 830 of the system 800). In such embodiments, it is generally the case that no specific bit of the bundle encoding 208 represents content of just one of the audio frames. Rather, the value of each of the M bits is usually associated, at least to some extent, with content of each of the audio frames (though due to training of the ML encoder, more direct relationships between specific bits and corresponding audio frames can exist due simply to arrangements of ML weights assigned during training).

[0181] During training of the system 800, and optionally during inference using the system 800, the bundle encoding 208 is provided to the decoder portion 840 of the system 800. The decoder portion 840 is configured to perform operations substantially inverse with respect to operations performed by the encoder portion 830. For example, in FIG. 8, the decoder portion 840 includes a joint decoder 810, one or more convolution layers 812, and one or more temporal de-intermingling layers 814.

[0182] The joint decoder 810 is configured to perform inverse operations with respect to the joint encoder 806 to generate an intra-bundle context array 841 based on the bundle encoding 208. The joint decoder 810 can include, for example, one or more machine-learning layers, such as, but not limited to, GRU layer(s), LSTM layer(s), other recurrent layers, fully connected layer(s), convolution layer(s), self-attention layer(s), pooling layer(s), other machine-learning layer(s), or a combination thereof. The joint decoder 810 can also be configured to generate the feedback signal 317B, which is provided to the encoder portion 830 for use during encoding of a subsequent set of N feature arrays 202.

[0183] In some embodiments, the convolution layer(s) 812 generate a temporally convolved (e.g., intermingled) feature array 843 based on the intra-bundle context array 841. Alternatively, in some embodiments, the convolution layer(s) 812 are omitted or incorporated into the temporal de-intermingling layer(s) 814, in which case the intra- bundle context array 841 is used as the temporally convolved feature array 843.

[0184] The temporal de-intermingling layer(s) 814 are configured to generate N feature arrays 234 (e.g., the feature arrays 728A-728E) based on the temporally convolved feature array 843. For example, the temporal de-intermingling layer(s) 814 can perform inverse operations with respect to the temporal intermingling layer(s) 802. For example, the temporal de-intermingling layers 814 can be configured to perform channel shuffle / channel unshuffle operations (whichever is inverse to the operations performed by the temporal intermingling layers 802 for the particular embodiment). To illustrate, the temporal de-intermingling layers 814 can include one or more pointwise convolution layers, one or more depthwise convolution layers, a channel unshuffle layer, and one or more pointwise group convolution layers. In this illustrative example, each of the convolution layers can perform transpose convolution with respect to respective similar convolution operations of the temporal intermingling layers 802.

[0185] It is noted that the operations performed by the temporal de-intermingling layer(s) 814 are conceptually inverse relative to the operations performed by the temporal intermingling layer(s) 802, rather than literally inverse. That is, it is a purpose of the temporal de-intermingling layer(s) 814 is to generate N feature arrays 234 based on the temporally convolved feature array 843; however, the specific operations performed by the temporal de-intermingling layer(s) 814 are based on training of the entire system 800. Such training is an optimization type process, which does not enforce application of inverse weights among any two layers of the system 800. Accordingly, while it may be conceptually helpful to think of the temporal de-intermingling layer(s) 814 as inverse with respect to the temporal intermingling layer(s) 802, the operations performed by the temporal de-intermingling layer(s) 814 and the temporal intermingling layer(s) 802 are trained, and unlikely to literally apply inverse weights.

[0186] FIG. 9 is a flowchart of an example of a method 900 of generating a bundle encoding, in accordance with some examples of the present disclosure. The method 900 can be performed by one or more processors using various encoding operations described with reference to FIGS. 1-8.

[0187] The method 900 includes, at block 902, obtaining N feature arrays, each feature array representing a frame of audio data, where N is an integer corresponding to a bundle frame count. For example, the processor(s) 190 of FIG. 1 can use the feature extractor(s) 124 to generate a set of N feature arrays based on a set of N audio frames 118.

[0188] The method 900 includes, at block 904, performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. For example, the processor(s) 190 of FIG. 1 can use the bundle encoder(s) 126 to generate an intra-bundle context array. As one example, the bundle encoder(s) 126 can generate the intra-bundle context array 736 using the temporal intermingling operations described with reference to system 700 of FIG. 7. To illustrate, the temporal intermingling operations can include performing convolution, at the convolution layer(s) 704, of each feature array 702 to generate N intermediate feature arrays 733 and providing the N intermediate feature arrays 733 as input to one or more recurrent layers (e.g., the bidirectional GRU layer(s) 706) to generate N temporal context arrays 735. In this illustrative example, the combiner 708 can combine the N temporal context arrays to generate the intra-bundle context array 736. For example, the combiner 708 can perform concatenation, pooling, or another combination operation, to combine the N temporal context arrays 735 to generate the intra-bundle context array 736.

[0189] As another example, the bundle encoder(s) 126 can generate the intra-bundle context array 835 using the temporal intermingling operations described with reference to system 800 of FIG. 8. To illustrate, the temporal intermingling operations performed by the temporal intermingling layer(s) 802 can include performing temporal unshuffling convolution of the N feature arrays 202 (e.g., the feature arrays 702) to generate a temporally convolved feature array 833 and performing convolution of the temporallyconvolved feature array 833 to generate the intra-bundle context array 835. In some embodiments, the temporal intermingling layer(s) 802 are configured to use a channel unshuffle operation to perform the temporal unshuffling convolution.

[0190] The method 900 includes, at block 906, encoding the intra-bundle context array to generate a bundle encoding. For example, the processor(s) 190 of FIG. 1 can use the bundle encoder(s) 126 to generate the bundle encoding. As one example, the joint encoder 710 and the bottleneck 316 of the system 700 of FIG. 7 can generate the bundle encoding 208 based on the intra-bundle context array 736. As another example, the joint encoder 806 and the bottleneck 316 of the system 800 of FIG. 8 can generate the bundle encoding 208 based on the intra-bundle context array 835.

[0191] In some embodiments, multiple codecs and / or multiple bundle encoders 126 are available, and the method 900 includes selecting the specific codec 120 or the specific bundle encoder 126 to generate the bundle encoding 208 based on the value of N indicated by the bundle frame count 142. For example, the method 900 can include obtaining data indicating a value of N and selecting a first machine-learning (ML) encoder (e.g., a first bundle encoder 126) to generate and encode the intra-bundle context array from among a plurality of ML encoders based on a value of N. In this example, the method 900 can also include determining a value of N based on channel condition data associated with communication of the bundle encoding.

[0192] FIG. 10 is a flowchart of an example of a method 1000 of generating audio data based on a bundle encoding, in accordance with some examples of the present disclosure. The method 1000 can be performed by one or more processors using various decoding operations described with reference to FIGS. 1-8.

[0193] The method 1000 includes, at block 1002, decoding a bundle encoding to generate an intra-bundle context array. In some embodiments, the bundle encoding can be received from another device via a communication channel. For example, the processor(s) 190 of FIG. 1 can use the bundle decoder(s) 128 to generate an intra-bundle context array based on a bundle encoding received as encoded data 150 from another device (e.g., one of the other device(s) 182). To illustrate, the bundle decoder(s) 128 caninclude the joint decoder 720 of the system 700 of FIG. 7, which can decode the bundle encoding 208 to generate the intra-bundle context array 741. As another example, the bundle decoder(s) 128 can include the joint decoder 810 of the system 800 of FIG. 8, which can decode the bundle encoding 208 to generate the intra-bundle context array 841.

[0194] The method 1000 includes, at block 1004, performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays, where N is an integer corresponding to a bundle frame count. For example, the processor(s) 190 of FIG. 1 can use the bundle decoder(s) 128 to perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays.

[0195] To illustrate, in some embodiments, the bundle decoder(s) 128 can generate the N feature arrays 234 using the temporal de-intermingling operations described with reference to the system 700 of FIG. 7. In such embodiments, the splitter 722 can split the intra-bundle context array 741 to generate N temporal context arrays 742 and provide the N temporal context arrays 742 as input to the temporal de-intermingling layer(s) 744 (e.g., the bidirectional GRU layers 724) to generate N intermediate feature arrays 743. The output 746 can perform convolution (e.g., transposed convolution) of each of the N intermediate feature arrays 743 to generate the N feature arrays 728.

[0196] As another example, the bundle decoder(s) 128 can generate the N feature arrays 234 using the temporal intermingling operations described with reference to the system 800 of FIG. 8. To illustrate, in some embodiments, the temporal de-intermingling layer(s) 814 of FIG. 8 perform temporal de-intermingling via a channel shuffle operation using the temporally intermediate feature arrays to generate the N feature arrays.

[0197] The method 1000 includes, at block 1006, generating N audio frames based on the N feature arrays. For example, the processor(s) 190 of FIG. 1 can use the neural synthesizer(s) 130 and optionally the postprocessor(s) 132 to generate audio frames based on the feature arrays. The neural synthesizer(s) 130 can include any of the neural synthesizer(s) 130 described with reference to FIGS. 1-6.

[0198] In some embodiments, multiple codecs and / or multiple bundle decoders 128 are available, and the method 1000 includes selecting the specific codec 120 or the specific bundle decoder 128 to generate the N feature arrays based on the value of N indicated by the bundle frame count 142. For example, the method 1000 can include obtaining data indicating a value of N. To illustrate, the value of N can be received with a data packet that includes the bundle encoding, can be known in advance of receipt of the packet, or can be determined based on the bundle encoding. In this example, the method 1000 can include selecting a first machine-learning (ML) decoder (e.g., a particular bundle decoder 128) from among a plurality of ML decoders based on a value of N and using the first ML decoder to decode the bundle encoding to generate the intra-bundle context array.

[0199] The method 900 of FIG. 9, the method 1000 of FIG. 10, or both, can be implemented by a field-programmable gate array (FPGA) device, an applicationspecific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a controller, another hardware device, firmware device, or any combination thereof. As an example, the method 900 of FIG. 9, the method 1000 of FIG. 10, or both, may be performed by a processor that executes instructions, such as described with reference to FIG. 23.

[0200] FIG. 11 depicts an implementation 1100 of the device 102 as an integrated circuit 1102 that includes the one or more processors 190. In FIG. 11, the processor(s) 190 include the codec(s) 120. As described with reference to FIG. 1, the codec(s) 120 include feature extractor(s) 124 and the bundle encoder(s) 126 for encoding data for transmission. Additionally, or alternatively, the codec(s) 120 include the bundle decoder(s) 128 and the neural synthesizer(s) 130 for decoding encoded data. Optionally, the codec(s) 120 can include the preprocessor(s) 122, the postprocessor s) 132, or both.

[0201] The integrated circuit 1102 in FIG. 11 also includes input circuitry 1104, such as one or more bus interfaces, to enable the integrated circuit 1102 to receive signals representing input data 1124 for processing. The input data 1124 can include, for example, the audio data 146 of FIG. 1 or a portion thereof (e.g., one or more audioframes 118), the bundle frame count 142, the encoded data 150, or a combination thereof.

[0202] The integrated circuit 1102 also includes output circuitry 1106, such as a bus interface, to enable the integrated circuit 1102 to output signals representing output data 1114. For example, the output data 1114 can correspond to or include the encoded data 150 of FIG. 1, the output audio 338 of FIG. 3, or a combination thereof.

[0203] When the input data 1124 includes the audio data 146, the codec(s) 120 enable the integrated circuit 1102 to jointly encode data representing a set of audio frames (e.g., N frames) for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. When the input data 1124 includes the encoded data 150, the codec(s) 120 enable the integrated circuit 1102 to decode a bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the integrated circuit 1102 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0204] FIG. 12 depicts an example 1200 in which the device 102 includes or is included within a mobile device 1202, such as a phone or tablet, as illustrative, non-limiting examples. The mobile device 1202 includes one or more microphones 1206, one or more speakers 1208, and a display screen 1204. Components of the integrated circuit 1102 (illustrated using dashed lines to indicate internal components that are not generally visible to a user), such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the mobile device 1202.

[0205] In a particular example, during use of the mobile device 1202, the microphone(s) 1206 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointlyencode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the mobile device 1202, the mobile device 1202 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the mobile device 1202 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0206] FIG. 13 illustrates a hearing aid device 1300 that incorporates aspects of the device 102 of FIG. 1. The example illustrated in FIG. 13 is an in-ear hearing aid device 1300 that includes a housing 1302 with a molded portion configured to fit within a contour of one ear of a user. In other examples, the hearing aid device 1300 can include an over-ear portion configured to be worn over the ear of the user. An earpiece 1310 is coupled to the housing 1302 and includes one or more speakers 1308. The hearing aid device 1300 also includes one or more microphones 1306 disposed on the housing 1302. In FIG. 13, components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor(s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the hearing aid device 1300.

[0207] In a particular example, during use of the hearing aid device 1300, the microphone(s) 1306 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the hearing aid device 1300, the hearing aid device 1300 can receive encoded data that include a bundle encoding representing multiple audioframes (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the hearing aid device 1300 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0208] FIG. 14 depicts an implementation 1400 in which the device 102 includes or is included within a headset device 1402. The headset device 1402 includes one or more microphones 1406 and one or more speakers 1408. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor(s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor s) 132, or a combination thereof, are integrated in the headset device 1402.

[0209] In a particular example, during use of the headset device 1402, the microphone(s) 1406 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the headset device 1402, the headset device 1402 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the headset device 1402 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0210] FIG. 15 depicts an implementation 1500 in which the device 102 includes or is included within a wearable electronic device 1502, illustrated as a "smart watch." The wearable electronic device 1502 includes a display screen 1504, one or more microphones 1506, and one or more speakers 1508. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor(s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the wearable electronic device 1502.

[0211] In a particular example, during use of the wearable electronic device 1502, the microphone(s) 1506 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the wearable electronic device 1502, the wearable electronic device 1502 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the wearable electronic device 1502 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0212] In some embodiments, the wearable electronic device 1502 is configured to generate a notification based on content of the encoded data. For example, the display screen 1504 can generate visual information based on the content of the data. As another example, the wearable electronic device 1502 can include a haptic device that provides a haptic notification (e.g., vibrates) based on content of data encoded by or decoded by the codec(s) 120. As another example, the display screen 1504 can be configured to display images or video received via one or more transmissions.

[0213] FIG. 16 depicts an implementation 1600 in which the device 102 includes or is included within a portable electronic device that corresponds to augmented reality or mixed reality glasses 1602. The glasses 1602 include a holographic projection unit 1610 configured to project visual data onto a surface of a lens 1612 or to reflect the visual data off of a surface of the lens 1612 and onto the wearer’s retina. The glasses 1602 also include one or more microphones 1606 and one or more speakers 1608. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the glasses 1602.

[0214] In a particular example, during use of the glasses 1602, the microphone(s) 1606 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the glasses 1602, the glasses 1602 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the glasses 1602 are able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0215] In a particular example, the holographic projection unit 1610 is configured to display a notification based on content of data encoded by or decoded by the codec(s) 120. As another example, the holographic projection unit 1610 can be configured to display images or video received as encoded data via one or more transmissions.

[0216] FIG. 17 depicts an implementation 1700 in which the device 102 includes or is included within a portable electronic device that corresponds to a pair of earbuds 1706that includes a first earbud 1702 and a second earbud 1704. Although earbuds are described, it should be understood that the present technology can be applied to other in- ear or over-ear audio devices.

[0217] The first earbud 1702 includes a first microphone 1720, such as a high signal -to- noise microphone positioned to capture the voice of a wearer of the first earbud 1702, an array of one or more other microphones configured to detect ambient sounds and spatially distributed to support beamforming, illustrated as microphones 1722A, 1722B, and 1722C, an "inner" microphone 1724 proximate to the wearer’s ear canal (e.g., to assist with active noise cancelling), and a self-speech microphone 1726, such as a bone conduction microphone configured to convert sound vibrations of the wearer’s ear bone or skull into an audio signal.

[0218] The second earbud 1704 can be configured in a substantially similar manner as the first earbud 1702. In some implementations, the first earbud 1702 is also configured to receive one or more audio signals generated by one or more microphones of the second earbud 1704, such as via wireless transmission between the earbuds 1702, 1704, or via wired transmission in implementations in which the earbuds 1702, 1704 are coupled via a transmission line.

[0219] In some implementations, the earbuds 1702, 1704 are configured to automatically switch between various operating modes, such as a passthrough mode in which ambient sound is played via a speaker 1730, a playback mode in which nonambient sound (e.g., streaming audio corresponding to a phone conversation, media playback, video game, etc.) is played back through the speaker 1730, and an audio zoom mode or beamforming mode in which one or more ambient sounds are emphasized and / or other ambient sounds are suppressed for playback at the speaker 1730. In other implementations, the earbuds 1702, 1704 may support fewer modes or may support one or more other modes in place of, or in addition to, the described modes.

[0220] In an illustrative example, the earbuds 1702, 1704 can automatically transition from the playback mode to the passthrough mode in response to detecting the wearer’s voice and may automatically transition back to the playback mode after the wearer hasceased speaking. In some examples, the earbuds 1702, 1704 can operate in two or more of the modes concurrently, such as by performing audio zoom on a particular ambient sound (e.g., a dog barking) and playing out the audio zoomed sound superimposed on the sound being played out while the wearer is listening to music (which can be reduced in volume while the audio zoomed sound is being played). In this example, the wearer can be alerted to the ambient sound associated with the audio event without halting playback of the music.

[0221] In FIG. 17, components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in one or both of the earbuds 1702, 1704. In a particular example, during use of the earbuds 1702, 1704, one or more of the microphone(s) 1720, 1722, 1724, 1726 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device, to the other earbud, or both. As another example, during use of the earbuds 1702, 1704, the earbuds 1702, 1704 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the earbuds 1702, 1704 are able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0222] FIG. 18 is an implementation 1800 in which the device 102 includes a wireless speaker and voice activated device 1802. The wireless speaker and voice activated device 1802 can have wireless network connectivity and is configured to execute an assistant operation. The wireless speaker and voice activated device 1802 includes one or more microphones 1806 and one or more speakers 1808. Components of theintegrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the wireless speaker and voice activated device 1802.

[0223] In a particular example, during use of the wireless speaker and voice activated device 1802, the microphone(s) 1806 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the wireless speaker and voice activated device 1802, the wireless speaker and voice activated device 1802 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the wireless speaker and voice activated device 1802 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0224] FIG. 19 depicts an implementation 1900 in which the device 102 includes or is included within a portable electronic device that corresponds to a camera device 1902. The camera device 1902 includes one or more microphones 1906, one or more speakers 1908, and an image sensor 1910. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor s) 132, or a combination thereof, are integrated in the camera device 1902.

[0225] In a particular example, during use of the camera device 1902, the microphone(s) 1906 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the camera device 1902, the camera device 1902 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the camera device 1902 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0226] FIG. 20 depicts an implementation 2000 in which the device 102 includes a portable electronic device that corresponds to a virtual reality, mixed reality, or augmented reality headset 2002. A visual interface device is positioned in front of the user's eyes to enable display of augmented reality, mixed reality, or virtual reality images or scenes to the user while the headset 2002 is worn. The headset 2002 also includes one or more microphones 2006, one or more speakers 2008. The headset 2002 also includes a display screen 2010 disposed to be positioned in front of a user's eyes when the headset 2002 is worn. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor s) 132, or a combination thereof, are integrated in the headset 2002.

[0227] In a particular example, during use of the headset 2002, the microphone(s) 2006 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As anotherexample, during use of the headset 2002, the headset 2002 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the headset 2002 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0228] FIG. 21 depicts an implementation 2100 in which the device 102 corresponds to, or is integrated within, a vehicle 2102, illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). The vehicle 2102 includes one or more microphones 2106, and one or more speakers 2108. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor(s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the vehicle 2102.

[0229] In a particular example, during use of the vehicle 2102, the microphone(s) 2106 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the vehicle 2102, the vehicle 2102 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the vehicle 2102 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0230] FIG. 22 depicts another implementation 2200 in which the device 102 corresponds to, or is integrated within, a vehicle 2202, illustrated as a car. The vehicle 2202 includes a display screen 2210, one or more microphones 2206, and one or more speakers 2208. Components of the integrated circuit 1102, such as the processor(s) 190, the memory 140, the input circuitry 1104, the output circuitry 1106, the codec(s) 120, the preprocessor s) 122, the feature extractor(s) 124, the bundle encoder(s) 126, the bundle decoder(s) 128, the neural synthesizer(s) 130, the postprocessor(s) 132, or a combination thereof, are integrated in the vehicle 2202.

[0231] In a particular example, during use of the vehicle 2202, the microphone(s) 2206 can capture audio (e.g., the audio data 146), and the codec(s) 120 can jointly encode data representing a set of audio frames (e.g., N frames) of the audio for a bundle to generate a bundle encoding that represents all of the audio frames of the bundle. The bundle encoding can subsequently be transmitted to another device. As another example, during use of the vehicle 2202, the vehicle 2202 can receive encoded data that include a bundle encoding representing multiple audio frames (e.g., N frames). In this example, the codec(s) 120 can decode the bundle encoding to generate data representing multiple audio frames that were jointly encoded to form the bundle encoding. Accordingly, the vehicle 2202 is able to transmit the audio data 146 via CBR transmissions (e.g., transmissions 178) at a low bitrate and yet provide high quality reproduction of the audio frames 118 due, in part, to the ability of the bundle encoder(s) 126 to account for temporal redundancies among the audio frames 118 of the bundle.

[0232] Referring to FIG. 23, a block diagram of a particular illustrative implementation of a device is depicted and generally designated 2300. In various implementations, the device 2300 may have more or fewer components than illustrated in FIG. 23. In an illustrative implementation, the device 2300 may correspond to the device 102. In an illustrative implementation, the device 2300 may perform one or more operations described with reference to FIGS. 1-22.

[0233] In a particular implementation, the device 2300 includes a processor 2306 (e.g., a CPU). The device 2300 of FIG. 23 also includes one or more additional processors 2310 (e.g., one or more DSPs). In a particular aspect, the processor(s) 190 of FIG. 1correspond to the processor 2306, the processors 2310, or a combination thereof. The processors 2310 may include a speech and music coder-decoder (CODEC) 2308 that includes a voice coder ("vocoder") encoder 2336, a vocoder decoder 2338, or a combination thereof. In some embodiments, the vocoder encoder 2336, the vocoder decoder 2338, or both, includes or corresponds to examples of the codec(s) 120 of FIG. 1. For example, in FIG. 23, the vocoder encoder 2336 includes the feature extractor(s) 124 and the bundle encoder(s) 126. As another example, in FIG. 23, the vocoder decoder 2338 includes the bundle decoder(s) 128 and the neural synthesizer(s) 130. Optionally, the vocoder encoder 2336 can also include the preprocessor(s) 122. Additionally, or alternatively, the vocoder decoder 2338 can optionally include the postprocessor s) 132.

[0234] In this context, the term "processor" refers to an integrated circuit consisting of logic cells, interconnects, input / output blocks, clock management components, memory, and optionally other special purpose hardware components, designed to execute instructions and perform various computational tasks. Examples of processors include, without limitation, central processing units (CPUs), digital signal processors (DSPs), neural processing units (NPU), graphics processing units (GPUs), field programmable gate arrays (FPGAs), microcontrollers, quantum processors, coprocessors, vector processors, other similar circuits, and variants and combinations thereof. In some cases, a processor can be integrated with other components, such as communication components, input / output components, etc. to form a system on a chip (SOC) device or a packaged electronic device.

[0235] Taking CPUs as a starting point, a CPU typically includes one or more processor cores, each of which includes a complex, interconnected network of transistors and other circuit components defining logic gates, memory elements, etc. A core is responsible for executing instructions to, for example, perform arithmetic and logical operations. Typically, a CPU includes an Arithmetic Logic Unit (ALU) that handles mathematical operations and a Control Unit that generates signals to coordinate the operation of other CPU components, such as to manage operations of a fetch-decode- execute cycle.

[0236] CPUs and / or individual processor cores generally include local memory circuits, such as registers and cache to temporarily store data during operations. Registers include high-speed, small-sized memory units intimately connected to the logic cells of a CPU. Often registers include transistors arranged as groups of flip-flops, which are configured to store binary data. Caches include fast, on-chip memory circuits used to store frequently accessed data. Caches can be implemented, for example, using Static Random-Access Memory (SRAM) circuits.

[0237] Operations of a CPU (e.g., arithmetic operations, logic operations, and flow control operations) are directed by software and firmware. At the lowest level, the CPU includes an instruction set architecture (ISA) that specifies how individual operations are performed using hardware resources (e.g., registers, arithmetic units, etc.). Higher level software and firmware is translated into various combinations of ISA operations to cause the CPU to perform specific higher-level operations. For example, an ISA typically specifies how the hardware components of the CPU move and modify data to perform operations such as addition, multiplication, and subtraction, and high-level software is translated into sets of such operations to accomplish larger tasks, such as adding two columns in a spreadsheet. Generally, a CPU operates on various levels of software, including a kernel, an operating system, applications, and so forth, with each higher level of software generally being more abstracted from the ISA and usually more readily understandable by human users.

[0238] GPUs, NPUs, DSPs, microcontrollers, coprocessors, FPGAs, ASICS, and vector processors include components similar to those described above for CPUs. The differences among these various types of processors are generally related to the use of specialized interconnection schemes and ISAs to improve a processor's ability to perform particular types of operations. For example, the logic gates, local memory circuits, and the interconnects therebetween of a GPU are specifically designed to improve parallel processing, sharing of data between processor cores, and vector operations, and the ISA of the GPU may define operations that take advantage of these structures. As another example, ASICs are highly specialized processors that include similar circuitry arranged and interconnected for a particular task, such as encryption or signal processing. As yet another example, FPGAs are programmable devices thatinclude an array of configurable logic blocks (e.g., interconnect sets of transistors and memory elements) that can be configured (often on the fly) to perform customizable logic functions.

[0239] A processor can be configured to perform a specific task by including, within the processor, specialized hardware to perform the task. Additionally, or alternatively, the processor can be configured to perform a specific task by loading and / or executing instructions (e.g., computer code) that, when executed, cause the processor to perform the specific task. Loading executable instructions to perform the task causes an internal configuration change in the processor that transforms what may otherwise be a general- purpose processor into a special purpose processor for performing the task.

[0240] In FIG. 23, the device 2300 includes the memory 140. The memory 140 may include instructions 2356, that are executable by the one or more additional processors 2310 (or the processor 2306) to implement the functionality described with reference to the codec(s) 120. In FIG. 23, the memory 140 also includes the audio data 146. The device 2300 also includes the modem 170 coupled, via the RFE 172 to one or more antennas 174.

[0241] The device 2300 may include a display 2328 coupled to a display controller 2326. One or more speakers 2392 and one or more microphones 2394 may be coupled to a CODEC 2334. The CODEC 2334 may include a digital-to-analog converter (DAC) 2302, an analog-to-digital converter (ADC) 2304, or both. In a particular implementation, the CODEC 2334 may receive analog signals from the microphone(s) 2394, convert the analog signals to digital signals representing the audio data 146 using the analog-to-digital converter 2304, and provide the digital signals (e.g., the audio frames 242) to the speech and music codec 2308. The speech and music codec 2308 may process the digital signals (e.g., using the codec(s) 120). In a particular implementation, the speech and music codec 2308 may provide digital signals to the CODEC 2334. The CODEC 2334 may convert the digital signals to analog signals using the digital-to-analog converter 2302 and may provide the analog signals to the speaker(s) 2392.

[0242] In a particular implementation, the device 2300 may be included in a system -in- package or system-on-chip device 2322. In a particular implementation, the memory 140, the processor 2306, the processors 2310, the display controller 2326, the CODEC 2334, the modem 170, the RFE 172, or a combination thereof, are included in the system-in-package or system-on-chip device 2322. In a particular implementation, an input device 2330 and a power supply 2344 are coupled to the system-in-package or the system-on-chip device 2322. Moreover, in a particular implementation, as illustrated in FIG. 23, the display 2328, the input device 2330, the speaker(s) 2392, the microphone(s) 2394, the antenna 174, and the power supply 2344 are external to the system-in-package or the system-on-chip device 2322. In a particular implementation, each of the display 2328, the input device 2330, the speaker(s) 2392, the microphone(s) 2394, the antenna 174, and the power supply 2344 may be coupled to a component of the system-in-package or the system-on-chip device 2322, such as an interface or a controller.

[0243] The device 2300 may include a smart speaker, a speaker bar, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet, a personal digital assistant, a display device, a television, a gaming console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a vehicle, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, an aerial vehicle, a home automation system, a voice-activated device, a wireless speaker and voice activated device, a portable electronic device, a car, a computing device, a communication device, an internet-of- things (loT) device, a virtual reality (VR) device, a base station, a mobile device, or any combination thereof.

[0244] In conjunction with the described implementations, an apparatus includes means for obtaining N feature arrays, each feature array representing a frame of audio data, where N is an integer corresponding to a bundle frame count. For example, the means for obtaining the N feature arrays can include the device 102, the modem 170, the RFE 172, the antenna 174, the processor(s) 190, the codec(s) 120, the feature extractor(s) 124, the bundle encoder(s) 126, the system 100, the integrated circuit 1102, the input circuitry 1104, the processor 2306, the processor(s) 2310, the system-in-package or thesystem-on-chip device 2322, other circuitry configured to obtain feature arrays, or a combination thereof.

[0245] The apparatus also includes means for performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays. For example, the means for performing the temporal intermingling operations on the N feature arrays to generate the intra-bundle context array representing the N feature arrays can include the device 102, the processor(s) 190, the codec(s) 120, the bundle encoder(s) 126, the system 100, the integrated circuit 1102, the processor 2306, the processor(s) 2310, the system-in-package or the system-on-chip device 2322, other circuitry configured to perform temporal intermingling operations, or a combination thereof.

[0246] The apparatus also includes means for encoding the intra-bundle context array to generate a bundle encoding. For example, the means for encoding the intra-bundle context array to generate the bundle encoding can include the device 102, the modem 170, the RFE 172, the antenna 174, the processor(s) 190, the codec(s) 120, the bundle encoder(s) 126, the system 100, the integrated circuit 1102, the processor 2306, the processor(s) 2310, the system-in-package or the system-on-chip device 2322, other circuitry configured to encode an intra-bundle context array to generate a bundle encoding, or a combination thereof.

[0247] In conjunction with the described implementations, an apparatus includes means for decoding a bundle encoding to generate an intra-bundle context array. For example, the means for decoding the bundle encoding can include the device 102, the processor(s) 190, the codec(s) 120, the bundle decoder(s) 128, the system 100, the integrated circuit 1102, the processor 2306, the processor(s) 2310, the system-in- package or the system-on-chip device 2322, other circuitry configured to decoding a bundle encoding to generate an intra-bundle context array, or a combination thereof.

[0248] The apparatus also includes means for performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays, where N is an integer corresponding to a bundle frame count. For example, the means for performingtemporal de-intermingling can include the device 102, the processor(s) 190, the codec(s) 120, the bundle decoder(s) 128, the system 100, the integrated circuit 1102, the processor 2306, the processor(s) 2310, the system-in-package or the system-on-chip device 2322, other circuitry configured to means for perform temporal de-intermingling on an intra-bundle context array to generate N feature arrays, or a combination thereof.

[0249] The apparatus also includes means for generating N audio frames based on the N feature arrays. For example, the means for generating N audio frames based on the N feature arrays can include the device 102, the modem 170, the RFE 172, the antenna 174, the processor(s) 190, the codec(s) 120, the bundle decoder(s) 128, the system 100, the neural synthesizer(s) 130, the postprocessor s) 132, the integrated circuit 1102, the processor 2306, the processor(s) 2310, the system-in-package or the system-on-chip device 2322, other circuitry configured to encode an intra-bundle context array to generate a bundle encoding, or a combination thereof.

[0250] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 140) includes instructions (e.g., the instructions 1122, or the instructions 2356) that, when executed by one or more processors (e.g., the processor(s) 190, the processor(s) 2310, or the processor 2306), cause the one or more processors to obtain N feature arrays (each feature array representing a frame of audio data, where N is an integer corresponding to a bundle frame count), perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays, and encode the intra-bundle context array to generate a bundle encoding.

[0251] In some implementations, a non-transitory computer-readable medium (e.g., a computer-readable storage device, such as the memory 140) includes instructions (e.g., the instructions 1122, or the instructions 2356) that, when executed by one or more processors (e.g., the processor(s) 190, the processor(s) 2310, or the processor 2306), cause the one or more processors to decode a bundle encoding to generate an intra- bundle context array and perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing audio data, where N is an integer corresponding to a bundle frame count.

[0252] Particular aspects of the disclosure are described below in sets of interrelated Examples:

[0253] According to Example 1, a device includes a memory configured to store audio data; and one or more processors configured to obtain N feature arrays, each feature array representing a frame of the audio data, where N is an integer corresponding to a bundle frame count; perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; and encode the intra-bundle context array to generate a bundle encoding.

[0254] Example 2 includes the device of Example 1, wherein, to perform the temporal intermingling operations, the one or more processors are configured to perform temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array and perform convolution of the temporally convolved feature array to generate the intra-bundle context array.

[0255] Example 3 includes the device of Example 2, wherein the one or more processors are configured to use a channel unshuffle operation to perform the temporal unshuffling convolution.

[0256] Example 4 includes the device of Example 1, wherein, to perform the temporal intermingling operations, the one or more processors are configured to perform convolution of each feature array to generate N intermediate feature arrays; provide the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays; and generate the intra-bundle context array based on the N temporal context arrays.

[0257] Example 5 includes the device of Example 4, wherein, to generate the intra- bundle context array, the one or more processors are configured to concatenate the N temporal context arrays.

[0258] Example 6 includes the device of Example 4, wherein, to generate the intra- bundle context array, the one or more processors are configured to perform one or more pooling operations using the N temporal context arrays.

[0259] Example 7 includes the device of any of Examples 4 to 6, wherein the one or more recurrent layers comprise bidirectional recurrent layers.

[0260] Example 8 includes the device of Example 7, wherein the N temporal context arrays represent hidden states of the bidirectional recurrent layers.

[0261] Example 9 includes the device of any of Examples 1 to 8, wherein the one or more processors are configured to determine a value of N based on channel condition data associated with communication of the bundle encoding.

[0262] Example 10 includes the device of any of Examples 1 to 9, wherein the one or more processors are configured to use a first machine-learning (ML) encoder to encode the intra-bundle context array to generate the bundle encoding, and wherein the first ML encoder is selected from among a plurality of ML encoders based on a value of N.

[0263] Example 11 includes the device of Example 10, wherein the first ML encoder is configured as a feedback recurrent autoencoder.

[0264] Example 12 includes the device of any of Examples 1 to 11, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0265] Example 13 includes the device of any of Examples 1 to 12 and further includes a modem coupled to the one or more processors and configured to facilitate transmission of the bundle encoding to another device.

[0266] Example 14 includes the device of any of Examples 1 to 13, wherein the one or more processors are integrated in a mobile device.

[0267] Example 15 includes the device of any of Examples 1 to 13, wherein the one or more processors are integrated in a wearable device.

[0268] Example 16 includes the device of any of Examples 1 to 13, wherein the one or more processors are integrated in a vehicle.

[0269] According to Example 17, a method includes obtaining N feature arrays, each feature array representing a frame of audio data, where N is an integer corresponding to a bundle frame count; performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; and encoding the intra-bundle context array to generate a bundle encoding.

[0270] Example 18 includes the method of Example 17, wherein performing the temporal intermingling operations includes: performing temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array and performing convolution of the temporally convolved feature arrays to generate the intra- bundle context array.

[0271] Example 19 includes the method of Example 18, further comprising using a channel unshuffle operation to perform the temporal unshuffling convolution.

[0272] Example 20 includes the method of Example 17, wherein performing the temporal intermingling operations includes: performing convolution of each feature array to generate N intermediate feature arrays; providing the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays; and generating the intra-bundle context array based on the N temporal context arrays.

[0273] Example 21 includes the method of Example 20, wherein generating the intra- bundle context array includes concatenating the N temporal context arrays.

[0274] Example 22 includes the method of Example 20, wherein generating the intra- bundle context array includes performing one or more pooling operations using the N temporal context arrays.

[0275] Example 23 includes the method of any of Examples 20 to 22, wherein the one or more recurrent layers comprise bidirectional recurrent layers.

[0276] Example 24 includes the method of Example 23, wherein the N temporal context arrays represent hidden states of the bidirectional recurrent layers.

[0277] Example 25 includes the method of any of Examples 17 to 24 and further includes determining a value of N based on channel condition data associated with communication of the bundle encoding.

[0278] Example 26 includes the method of any of Examples 17 to 25, further includes obtaining data indicating a value of N; and selecting a first machine-learning (ML) encoder to encode the intra-bundle context array from among a plurality of ML encoder based on the value of N.

[0279] Example 27 includes the method of Example 26, wherein the first ML encoder is configured as a feedback recurrent autoencoder.

[0280] Example 28 includes the method of any of Examples 17 to 27, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0281] According to Example 29, a non-transitory computer-readable device stores instructions that are executable by one or more processors to cause the one or more processors to obtain N feature arrays, each feature array representing a frame of audio data, where N is an integer corresponding to a bundle frame count; perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; and encode the intra-bundle context array to generate a bundle encoding.

[0282] Example 30 includes the non-transitory computer-readable device of Example 29, wherein, to perform the temporal intermingling operations, the temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array and perform convolution of the temporally convolved feature arrays to generate the intra- bundle context array.

[0283] Example 31 includes the non -transitory computer-readable device of Example 30, wherein the instructions cause the one or more processors to use a channel unshuffle operation to perform the temporal unshuffling convolution.

[0284] Example 32 includes the non-transitory computer-readable device of Example 29, wherein, to perform the temporal intermingling operations, the instructions cause the one or more processors to perform convolution of each feature array to generate N intermediate feature arrays; provide the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays; and generate the intrabundle context array based on the N temporal context arrays.

[0285] Example 33 includes the non-transitory computer-readable device of Example 32, wherein, to generate the intra-bundle context array, the instructions cause the one or more processors to concatenate the N temporal context arrays.

[0286] Example 34 includes the non-transitory computer-readable device of Example 32, wherein, to generate the intra-bundle context array, the instructions cause the one or more processors to perform one or more pooling operations using the N temporal context arrays.

[0287] Example 35 includes the non-transitory computer-readable device of any of Examples 32 to 34, wherein the one or more recurrent layers comprise bidirectional recurrent layers.

[0288] Example 36 includes the non-transitory computer-readable device of Example 35, wherein the N temporal context arrays represent hidden states of the bidirectional recurrent layers.

[0289] Example 37 includes the non-transitory computer-readable device of any of Examples 29 to 36, wherein the instructions cause the one or more processors to determine a value of N based on channel condition data associated with communication of the bundle encoding.

[0290] Example 38 includes the non-transitory computer-readable device of any of Examples 29 to 37, wherein the instructions cause the one or more processors to use afirst machine-learning (ML) encoder to encode the intra-bundle context array to generate the bundle encoding, and wherein the first ML encoder is selected from among a plurality of ML encoders based on a value of N.

[0291] Example 39 includes the non-transitory computer-readable device of Example 38, wherein the first ML encoder is configured as a feedback recurrent autoencoder.

[0292] Example 40 includes the non-transitory computer-readable device of any of Examples 29 to 39, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0293] According to Example 41, an apparatus includes means for obtaining N feature arrays, each feature array representing a frame of audio data, wherein N is an integer corresponding to a bundle frame count; means for performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; and means for encoding the intra-bundle context array to generate a bundle encoding.

[0294] Example 42 includes the apparatus of Example 41, wherein the means for performing the temporal intermingling includes: means for performing temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array and means for performing convolution of the temporally convolved feature arrays to generate the intra-bundle context array.

[0295] Example 43 includes the apparatus of Example 42, further comprising means for using a channel unshuffle operation to perform the temporal unshuffling convolution.

[0296] Example 44 includes the apparatus of Example 41, wherein the means for performing the temporal intermingling includes: means for performing convolution of each feature array to generate N intermediate feature arrays; means for providing the N intermediate feature arrays as input to one or more recurrent layers to generate Ntemporal context arrays; and means for generating the intra-bundle context array based on the N temporal context arrays.

[0297] Example 45 includes the apparatus of Example 44, wherein the means for generating the intra-bundle context array includes means for concatenating the N temporal context arrays.

[0298] Example 46 includes the apparatus of Example 44, wherein the means for generating the intra-bundle context array includes means for performing one or more pooling operations using the N temporal context arrays.

[0299] Example 47 includes the apparatus of any of Examples 44 to 46, wherein the one or more recurrent layers comprise bidirectional recurrent layers.

[0300] Example 48 includes the apparatus of Example 47, wherein the N temporal context arrays represent hidden states of the bidirectional recurrent layers.

[0301] Example 49 includes the apparatus of any of Examples 41 to 48 and further includes means for determining a value of N based on channel condition data associated with communication of the bundle encoding.

[0302] Example 50 includes the apparatus of any of Examples 41 to 49, further includes means for obtaining data indicating a value of N; and means for selecting a first machine-learning (ML) encoder to encode the intra-bundle context array from among a plurality of ML encoders based on the value of N.

[0303] Example 51 includes the apparatus of Example 50, wherein the first ML encoder is configured as a feedback recurrent autoencoder.

[0304] Example 52 includes the apparatus of any of Examples 41 to 51, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0305] According to Example 53, a device includes a memory configured to store audio data; and one or more processors configured to decode a bundle encoding to generate an intra-bundle context array; and perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing the audio data, wherein N is an integer corresponding to a bundle frame count.

[0306] Example 54 includes the device of Example 53, wherein, to perform the temporal de-intermingling, the one or more processors are configured to perform a channel shuffle operation on the feature array to generate the N feature arrays.

[0307] Example 55 includes the device of Example 53, wherein, to perform the temporal de-intermingling, the one or more processors are configured to split the intra- bundle context array to generate N temporal context arrays; provide the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and perform further processing of each feature array to generate the N feature arrays.

[0308] Example 56 includes the device of Example 55, wherein the one or more recurrent layers comprise one or more bidirectional recurrent layers.

[0309] Example 57 includes the device of any of Examples 53 to 56, wherein the one or more processors are configured to use a first machine-learning (ML) decoder to decode the bundle encoding to generate the intra-bundle context array, and wherein the first ML decoder is selected from among a plurality of ML decoders based on a value of N.

[0310] Example 58 includes the device of Example 57, wherein the first ML decoder is configured as a decoder portion of a feedback recurrent autoencoder.

[0311] Example 59 includes the device of any of Examples 53 to 58, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0312] Example 60 includes the device of any of Examples 53 to 59 and further includes a modem coupled to the one or more processors and configured to facilitate receipt of the bundle encoding from another device.

[0313] Example 61 includes the device of any of Examples 53 to 60, wherein the one or more processors are integrated in a mobile device.

[0314] Example 62 includes the device of any of Examples 53 to 60, wherein the one or more processors are integrated in a wearable device.

[0315] Example 63 includes the device of any of Examples 53 to 60, wherein the one or more processors are integrated in a vehicle.

[0316] According to Example 64, a method includes decoding a bundle encoding to generate an intra-bundle context array; performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays, wherein N is an integer corresponding to a bundle frame count; and generating N audio frames based on the N feature arrays.

[0317] Example 68 includes the method of Example 64, wherein performing the temporal de-intermingling includes performing a channel shuffle operation on the feature array to generate the N feature arrays.

[0318] Example 69 includes the method of Example 64, wherein performing the temporal de-intermingling comprises: splitting the intra-bundle context array to generate N temporal context arrays; providing the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and performing further processing of each feature array to generate the N feature arrays.

[0319] Example 70 includes the method of Example 69, wherein the one or more recurrent layers comprise one or more bidirectional recurrent layers.

[0320] Example 71 includes the method of any of Examples 64 to 70, further includes selecting a first machine-learning (ML) decoder from among a plurality of ML decodersbased on a value of N; and using the first ML decoder to decode the bundle encoding to generate the intra-bundle context array.

[0321] Example 72 includes the method of Example 71, wherein the first ML decoder is configured as a decoder portion of a feedback recurrent autoencoder.

[0322] Example 73 includes the method of any of Examples 64 to 72, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0323] Example 74 includes the method of any of Examples 64 to 73 and further includes receiving the bundle encoding from another device via a communication channel.

[0324] According to Example 75, an apparatus includes means for decoding a bundle encoding to generate an intra-bundle context array; means for performing temporal deintermingling on the intra-bundle context array to generate N feature arrays, wherein N is an integer corresponding to a bundle frame count; and means for generating N audio frames based on the N feature arrays.

[0325] Example 76 includes the apparatus of Example 75, wherein the means for performing the temporal de-intermingling includes: means for performing a channel shuffle operation on the feature array to generate the N feature arrays.

[0326] Example 77 includes the apparatus of Example 75, wherein the means for performing the temporal de-intermingling comprises: means for splitting the intra- bundle context array to generate N temporal context arrays; means for providing the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and means for performing further processing of each feature array to generate the N feature arrays.

[0327] Example 78 includes the apparatus of Example 77, wherein the one or more recurrent layers comprise one or more bidirectional recurrent layers.

[0328] Example 79 includes the apparatus of any of Examples 75 to 78, further includes means for selecting a first machine-learning (ML) decoder from among a plurality of ML decoders based on a value of N, and wherein the means for decoding the bundle encoding to generate the intra-bundle context array corresponds to the first ML decoder.

[0329] Example 80 includes the apparatus of Example 79, wherein the first ML decoder is configured as a decoder portion of a feedback recurrent autoencoder.

[0330] Example 81 includes the apparatus of any of Examples 75 to 80, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0331] Example 82 includes the apparatus of any of Examples 75 to 81 and further includes means for receiving the bundle encoding from another device via a communication channel.

[0332] According to Example 83, a non-transitory computer-readable device stores instructions that are executable by one or more processors to cause the one or more processors to decode a bundle encoding to generate an intra-bundle context array; and perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing audio data, wherein N is an integer corresponding to a bundle frame count.

[0333] Example 84 includes the non-transitory computer-readable device of Example 83, wherein, to perform the temporal de-intermingling, the instructions cause one or more processors to perform a channel shuffle operation on the feature array to generate the N feature arrays.

[0334] Example 85 includes the non-transitory computer-readable device of Example 83, wherein, to perform the temporal de-intermingling, the instructions cause the one or more processors to split the intra-bundle context array to generate N temporal context arrays; provide the N temporal context arrays as input to one or more recurrent layers togenerate N intermediate feature arrays; and perform further processing of each feature array to generate the N feature arrays.

[0335] Example 86 includes the non -transitory computer-readable device of Example 85, wherein the one or more recurrent layers comprise one or more bidirectional recurrent layers.

[0336] Example 87 includes the non-transitory computer-readable device of any of Examples 83 to 86, wherein the instructions cause the one or more processors to use a first machine-learning (ML) decoder to decode the bundle encoding to generate the intra-bundle context array, and wherein the first ML decoder is selected from among a plurality of ML decoders based on a value of N.

[0337] Example 88 includes the non-transitory computer-readable device of Example 87, wherein the first ML decoder is configured as a decoder portion of a feedback recurrent autoencoder.

[0338] Example 89 includes the non-transitory computer-readable device of any of Examples 83 to 88, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

[0339] Those of skill would further appreciate that the various illustrative logical blocks, configurations, modules, circuits, and algorithm steps described in connection with the implementations disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or combinations of both. Various illustrative components, blocks, configurations, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends upon the particular application and design constraints imposed on the overall system. Skilled artisans may implement the described functionality in varying ways for each particular application, such implementation decisions are not to be interpreted as causing a departure from the scope of the present disclosure.

[0340] The steps of a method or algorithm described in connection with the implementations disclosed herein may be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module may reside in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disk, a removable disk, a compact disc read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor may read information from, and write information to, the storage medium. In the alternative, the storage medium may be integral to the processor. The processor and the storage medium may reside in an application-specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In the alternative, the processor and the storage medium may reside as discrete components in a computing device or user terminal.

[0341] The previous description of the disclosed aspects is provided to enable a person skilled in the art to make or use the disclosed aspects. Various modifications to these aspects will be readily apparent to those skilled in the art, and the principles defined herein may be applied to other aspects without departing from the scope of the disclosure. Thus, the present disclosure is not intended to be limited to the aspects shown herein but is to be accorded the widest scope possible consistent with the principles and novel features as defined by the following claims.

Claims

WHAT IS CLAIMED IS:

1. A device comprising: a memory configured to store audio data; and one or more processors configured to: obtain N feature arrays, each feature array representing a frame of the audio data, wherein N is an integer corresponding to a bundle frame count; perform temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; and encode the intra-bundle context array to generate a bundle encoding.

2. The device of claim 1, wherein, to perform the temporal intermingling operations, the one or more processors are configured to: perform temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array; and perform convolution of the temporally convolved feature array to generate the intra- bundle context array.

3. The device of claim 1, wherein, to perform the temporal intermingling operations, the one or more processors are configured to: perform convolution of each feature array to generate N intermediate feature arrays; provide the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays; and generate the intra-bundle context array based on the N temporal context arrays.

4. The device of claim 3, wherein, to generate the intra-bundle context array, the one or more processors are configured to concatenate the N temporal context arrays.

5. The device of claim 3, wherein, to generate the intra-bundle context array, the one or more processors are configured to perform one or more pooling operations using the N temporal context arrays.

6. The device of claim 1, wherein the one or more processors are configured to determine a value of N based on channel condition data associated with communication of the bundle encoding.

7. The device of claim 1, wherein the one or more processors are configured to use a first machine-learning (ML) encoder to encode the intra-bundle context array to generate the bundle encoding, and wherein the first ML encoder is selected from among a plurality of ML encoders based on a value of N.

8. The device of claim 1, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

9. The device of claim 1, further comprising a modem coupled to the one or more processors and configured to facilitate transmission of the bundle encoding to another device.

10. The device of claim 1, wherein the one or more processors are integrated in a mobile device.

11. The device of claim 1, wherein the one or more processors are integrated in a wearable device.

12. The device of claim 1, wherein the one or more processors are integrated in a vehicle.

13. A method compri sing : obtaining N feature arrays, each feature array representing a frame of audio data, wherein N is an integer corresponding to a bundle frame count; performing temporal intermingling operations on the N feature arrays to generate an intra-bundle context array representing the N feature arrays; andencoding the intra-bundle context array to generate a bundle encoding.

14. The method of claim 13, wherein performing the temporal intermingling operations includes: performing temporal unshuffling convolution of the N feature arrays to generate a temporally convolved feature array; and performing convolution of the temporally convolved feature arrays to generate the intra- bundle context array.

15. The method of claim 13, wherein performing the temporal intermingling operations includes: performing convolution of each feature array to generate N intermediate feature arrays; providing the N intermediate feature arrays as input to one or more recurrent layers to generate N temporal context arrays; and generating the intra-bundle context array based on the N temporal context arrays.

16. The method of claim 15, wherein generating the intra-bundle context array includes concatenating the N temporal context arrays.

17. The method of claim 15, wherein generating the intra-bundle context array includes performing one or more pooling operations using the N temporal context arrays.

18. The method of claim 13, further comprising determining a value of N based on channel condition data associated with communication of the bundle encoding.

19. The method of claim 13, further comprising: obtaining data indicating a value of N; and selecting a first machine-learning (ML) encoder to encode the intra-bundle context array from among a plurality of ML encoder based on the value of N.

20. The method of claim 13, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and wherein associations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

21. A device comprising: a memory configured to store audio data; and one or more processors configured to: decode a bundle encoding to generate an intra-bundle context array; and perform temporal de-intermingling on the intra-bundle context array to generate N feature arrays representing the audio data, wherein N is an integer corresponding to a bundle frame count.

22. The device of claim 21, wherein, to perform the temporal de-intermingling, the one or more processors are configured to perform a channel shuffle operation on the feature array to generate the N feature arrays.

23. The device of claim 21, wherein, to perform the temporal de-intermingling, the one or more processors are configured to: split the intra-bundle context array to generate N temporal context arrays; provide the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and perform further processing of each feature array to generate the N feature arrays.

24. The device of claim 21, wherein the one or more processors are configured to use a first machine-learning (ML) decoder to decode the bundle encoding to generate the intra-bundle context array, and wherein the first ML decoder is selected from among a plurality of ML decoders based on a value of N.

25. The device of claim 21, wherein the bundle encoding represents N frames using M bits, wherein M is an integer representing a fixed bundle size, and whereinassociations between specific frames of the N frames and specific bits of the M bits are due to trained parameters of an ML encoder that generates the bundle encoding.

26. The device of claim 21, further comprising a modem coupled to the one or more processors and configured to facilitate receipt of the bundle encoding from another device.

27. A method comprising: decoding a bundle encoding to generate an intra-bundle context array; performing temporal de-intermingling on the intra-bundle context array to generate N feature arrays, wherein N is an integer corresponding to a bundle frame count; and generating N audio frames based on the N feature arrays.

28. The method of claim 27, wherein performing the temporal de-intermingling includes performing a channel shuffle operation on the feature array to generate the N feature arrays.

29. The method of claim 27, wherein performing the temporal de-intermingling comprises: splitting the intra-bundle context array to generate N temporal context arrays; providing the N temporal context arrays as input to one or more recurrent layers to generate N intermediate feature arrays; and performing further processing of each feature array to generate the N feature arrays.

30. The method of claim 27, further comprising: selecting a first machine-learning (ML) decoder from among a plurality of ML decoders based on a value of N; and using the first ML decoder to decode the bundle encoding to generate the intra-bundle context array.

Images