Data reconstruction using machine learning predictive coding

Abstract

A method includes generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time sequence of data samples. The method includes generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples. The method includes providing the first reconstructed data sample and the second reconstructed data sample as inputs to a neural network. The neural network is configured to generate network predicted data samples using machine learning prediction coding. The network predicted data sample corresponds to a predicted version of a particular data sample in the time series of data samples that is located between the first data sample and the second data sample.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to commonly owned Greek Patent Application No. 20220100725 filed on September 2, 2022, the entire contents of which are expressly incorporated herein by reference. Technical Field

[0003] The present disclosure relates generally to encoding and / or decoding data, and in particular, to using machine learning predictive coding to generate network predicted data samples. Background Art

[0004] Advances in technology have led to smaller and more powerful computing devices. For example, there are currently various portable personal computing devices, including wireless phones, such as mobile and smart phones, tablets and laptop computers, which are small, lightweight and easy to carry by users. These devices can transmit voice packets, data packets or both through wired or wireless networks. In addition, many such devices incorporate additional functions, such as digital still cameras, digital video cameras, digital recorders and audio file players. In addition, such devices can process executable instructions, including software applications that can be used to access the Internet, such as web browser applications. Therefore, these devices can include significant computing power.

[0005] A common use of such wireless devices is communication (e.g., voice, video, and / or data communication). In wireless communication, a device with data to be sent generates a signal representing the data as a set of bits. Typically, the signal also includes other information, such as a packet header. Because wireless devices are usually power-limited (e.g., battery-powered) and because wireless communication resources (e.g., radio frequency channels) may be crowded, it may be desirable to use as few bits as possible to send specific data. However, many techniques for representing data using fewer bits are lossy. That is, encoding the data to be sent using fewer bits results in a less accurate representation of the data. Therefore, there may be a conflict between the goal of sending an accurate (e.g., high-fidelity) representation of the data to be sent (e.g., using more bits) and the goal of efficiently sending data (e.g., using fewer bits). Summary of the invention

[0006] According to certain aspects, a device includes a memory and one or more processors coupled to the memory. The one or more processors are operably configured to generate a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream. The one or more processors are also operably configured to generate a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples. The one or more processors are also operably configured to provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of data samples, and the specific data sample is located between the first data sample and the second data sample.

[0007] According to another specific aspect, a method includes generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time sequence of data samples of a portion of a data stream. The method also includes generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time sequence of data samples. The method also includes providing the first reconstructed data sample and the second reconstructed data sample as input to a neural network. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a specific data sample in the time sequence of data samples, and the specific data sample is located between the first data sample and the second data sample.

[0008] According to another specific aspect, an apparatus includes a component for generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream. The apparatus also includes a component for generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples. The apparatus also includes a component for providing the first reconstructed data sample and the second reconstructed data sample as input to a neural network. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of data samples, and the specific data sample is located between the first data sample and the second data sample.

[0009] According to another specific aspect, a non-transitory computer-readable medium stores instructions that are executable by one or more processors to generate a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream. Execution of the instructions also causes the one or more processors to generate a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples. Execution of the instructions also causes the one or more processors to provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of data samples, and the specific data sample is located between the first data sample and the second data sample. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 is a diagram of a specific illustrative example of a system configured to use machine learning predictive coding to reconstruct a data sample using nearby reconstructed data samples.

[0011] Figure 2 is a diagram of a specific illustrative example of a system configured to determine a residual vector for reconstructed data samples generated using machine learning predictive coding.

[0012] Figure 3 is a diagram of a specific illustrative example of a system configured to bundle data representing data samples into a single packet.

[0013] Figure 4 is a diagram of a specific illustrative example of a system including two or more devices configured to communicate via the transmission of encoded data.

[0014] Figure 5 is a diagram of a specific illustrative example of a neural network architecture configured to use machine learning predictive coding to reconstruct a data sample using nearby reconstruction data.

[0015] Figure 6 is a diagram of a specific illustrative example of a system configured to reconstruct multiple data samples at different locations using machine learning predictive coding.

[0016] Figure 7 is a diagram of a Feedback Recurrent Autoencoder (FRAE) architecture configured to generate an auxiliary vector based on multiple data samples.

[0017] Figure 8 is a diagram of another specific illustrative example of a system configured to reconstruct multiple data samples at different locations using machine learning predictive coding.

[0018] Fig. 9 is a diagram of a specific example of components of an encoding and decoding device in an integrated circuit.

[0019] Fig.10 is a diagram of a mobile device according to some examples of the present disclosure, the mobile device comprising circuitry configured to use machine learning predictive coding to reconstruct a data sample using nearby reconstructed data samples.

[0020] Fig.11 is a diagram of a head-mounted device according to some examples of the present disclosure, the head-mounted device including circuitry configured to use machine learning predictive decoding to reconstruct data samples using nearby reconstructed data samples.

[0021] Fig.12 is a diagram of a wearable electronic device according to some examples of the present disclosure, the wearable electronic device including circuitry configured to use machine learning predictive decoding to reconstruct data samples using nearby reconstructed data samples.

[0022] Fig.13 is a diagram of a voice-controlled speaker system according to some examples of the present disclosure, the voice-controlled speaker system including circuitry configured to use machine learning predictive coding to reconstruct data samples using nearby reconstructed data samples.

[0023] Fig.14 is a diagram of a camera according to some examples of the present disclosure, the camera including circuitry configured to use machine learning predictive coding to reconstruct data samples using nearby reconstructed data samples.

[0024] Fig.15 is a diagram of a head-mounted device (such as a virtual reality, mixed reality, or augmented reality head-mounted device) according to some examples of the present disclosure, the head-mounted device including circuits configured to use machine learning predictive decoding to reconstruct data samples using nearby reconstructed data samples.

[0025] Fig.16 is a diagram of a first example of a vehicle according to some examples of the present disclosure, the vehicle comprising circuitry configured to use machine learning predictive decoding to reconstruct a data sample using nearby reconstructed data samples.

[0026] Fig.17 is a diagram of a second example of a vehicle according to some examples of the present disclosure, the vehicle comprising circuitry configured to use machine learning predictive decoding to reconstruct data samples using nearby reconstructed data samples.

[0027] Fig.18 is a flowchart of a specific example of an operating method of a communication device.

[0028] Fig.19is a flow chart of another specific example of an operating method of a communication device.

[0029] Fig. 20 is a diagram of a specific example of components of a transmitting device in an integrated circuit.

[0030] Fig.21 is a diagram of a specific example of components of a receiving device in an integrated circuit.

[0031] Fig. 22 is a block diagram of a specific illustrative example of a device operable to perform encoding, decoding, or both. DETAILED DESCRIPTION

[0032] A feedback redundant autoencoder (FRAE) or a different type of encoder can be used to encode data samples of a data stream (e.g., an audio data stream, a video data stream, etc.) to generate information that is sent to a receiving device. For example, each data sample can be processed by a FRAE to generate a latent vector. For many applications, the latent vector is quantized to form a latent code, which is included in a packet sent to the receiving device. This data encoding and transmission scheme is a relatively efficient way to communicate data; however, additional efficiency can be obtained by using a machine learning predictive decoding algorithm at the receiving device to (autonomously) generate network-predicted data samples for a particular data sample of the data stream, even when no data bits representing the particular data sample (e.g., no residual vector) are transmitted to the receiving device.

[0033] Various aspects disclosed herein save transmission bandwidth by not sending data representing certain data samples. For example, no data bits (e.g., zero data bits) may be allocated for transmission of data representing some data samples of a data stream. Despite the allocation of zero data bits to certain data samples, a receiving device is still able to generate a network-predicted data sample that is a perceptibly accurate representation of the certain data sample.

[0034] For example, a first device (eg, a transmitting device) may encode reference data samples to generate a latent vector. Unlike the encoded reference data samples, the first device bypasses encoding of intermediate data samples (eg, data samples between reference data samples).

[0035] The first device provides reference data samples to the neural network instead of encoding the intermediate data samples. The neural network uses machine learning predictive decoding to generate predicted data samples of the intermediate data samples. At least one predicted data sample corresponds to a relative timing between the first reference data sample and the second reference data sample. In some cases, the predicted data samples can be perceptually accurate representations of the intermediate data samples, making allocation of data bits for representing data of the predicted data samples unnecessary.

[0036] As used herein, a "reference data sample" refers to a data sample of a data stream used to predict or reconstruct a predicted data sample. In certain aspects, the designation of a particular data sample as a reference data sample or a predicted data sample is independent of the content of the data sample. Instead, the designation can be based on the perceived quality of the machine learning prediction and / or based on a general or temporary need to reduce transmission bandwidth when communicating with a receiving device. There may be no substantial difference between the content of the reference data sample and the predicted data sample. A latent code corresponding to a latent vector encoding the corresponding reference data sample is sent to a second device (e.g., a receiving device).

[0037] The second device may reconstruct the reference data samples by performing a decoding operation on the potential code received from the first device. After decoding the reference data samples, the second device provides the reconstructed reference data samples to the neural network. The neural network uses machine learning predictive decoding to generate predicted data samples. At least one predicted data sample corresponds to a time position between the first reference data sample and the second reference data sample. In some cases, the predicted data sample may be an accurate representation of the intermediate data sample to significantly improve the quality of the reconstructed reference data sample.

[0038] As described herein, the predicted data samples may be referred to as “network-predicted data samples.” Thus, instead of using interpolation techniques to generate interpolated data samples based on reconstructed reference data samples, the second device implements a nonlinear and data-driven neural network model to reconstruct features of the predicted data samples. Thus, the network-predicted data samples are more accurate representations of the predicted data samples than data samples generated based solely on interpolation.

[0039] Unless expressly limited by its context, the term "generating" is used herein to indicate any of its ordinary meanings, such as calculating or otherwise generating. Unless expressly limited by its context, the term "computing" is used herein to indicate any of its ordinary meanings, such as calculating, evaluating, smoothing, and / or selecting from a plurality of values. Unless expressly limited by its context, the term "obtaining" is used herein to indicate any of its ordinary meanings, such as calculating, deriving, receiving (e.g., from another component, block, or device), and / or retrieving (e.g., from a memory register or storage element array).

[0040] Unless expressly limited by its context, the term "generating" is used to indicate any of its ordinary meanings, such as calculating, generating, and / or providing. Unless expressly limited by its context, the term "providing" is used to indicate any of its ordinary meanings, such as calculating, generating, and / or producing. Unless expressly limited by its context, the term "coupling" is used to indicate a direct or indirect electrical or physical connection. If the connection is indirect, there may be other blocks or components between the structures being "coupled." For example, a speaker can be acoustically coupled to a nearby wall via an intermediate medium (e.g., air), which enables waves (e.g., sound) to propagate from the speaker to the wall (or vice versa).

[0041] The term "configuration" may be used with reference to a method, an apparatus, a device, a system, or any combination thereof, as indicated by its specific context. Where the term "comprising" is used in this specification and claims, other elements or operations are not excluded. The term "based on" (as in "A is based on B") is used to indicate any of its ordinary meanings, including the following: (i) "based at least on" (e.g., "A is based at least on B") and, where appropriate in a particular context, (ii) "equal to" (e.g., "A is equal to B"). In the case where A is based on B (i), at least based on is included, which may include a configuration in which A is coupled to B. Similarly, the term "responsive to" is used to indicate any of its ordinary meanings, including "at least responsive to". The term "at least one" is used to indicate any of its ordinary meanings, including "one or more". The term "at least two" is used to indicate any of its ordinary meanings, including "two or more".

[0042] Unless otherwise specified in a particular context, the terms "device" and "equipment" are used generally and interchangeably. Unless otherwise specified, any disclosure of the operation of a device having specific features is also explicitly intended to disclose methods having similar features (and vice versa), and any disclosure of the operation of a device according to a specific configuration is also explicitly intended to disclose methods according to a similar configuration (and vice versa). Unless otherwise specified in a particular context, the terms "method", "process", "procedure" and "technique" are used generally and interchangeably. The terms "element" and "module" may be used to indicate a portion of a larger configuration. The term "packet" may correspond to a data unit comprising a header portion and a payload portion. Any incorporation by reference of a portion of a document shall also be understood to incorporate definitions of terms or variables referenced within that portion, where such definitions appear elsewhere in the document, as well as any drawings referenced in the incorporation portion.

[0043] As used herein, the term "communication device" refers to an electronic device that can be used for voice and / or data communications over a wireless communication network. Examples of communication devices include speaker poles, smart speakers, cellular phones, personal digital assistants (PDAs), handheld devices, head-mounted devices, wireless modems, laptop computers, personal computers, etc.

[0044] Certain aspects are described herein with reference to the accompanying drawings. In the specification, common features are represented by common reference numerals in all drawings. In some drawings, multiple instances of a particular type of feature are used. Although the features are physically and / or logically different, each feature uses the same reference numeral, and different instances are distinguished by adding letters to the reference numerals. When referring to features as a group or type herein (for example, when no reference is made to a specific one of the features), the reference numeral is used without a distinguishing letter. However, when referring to a specific feature among multiple features of the same type, the reference numeral is used with a distinguishing letter. For example, referring to Figure 1 , a plurality of reconstructed data samples are shown and are associated with reference numerals 126A and 126B. When referring to a particular one of the reconstructed data samples, such as reconstructed data sample 126A, a distinguishing letter "A" is used. However, when referring to any one of the reconstructed data samples or a group of the reconstructed data samples, reference numeral 126 is used without a distinguishing letter.

[0045] Figure 1 is a diagram of a specific illustrative example of a system 100 configured to use machine learning predictive coding to reconstruct a data sample using nearby reconstructed data samples. The system 100 includes a transmitting device 102 and a receiving device 104. The transmitting device 102 is configured to send one or more encoded data packets to the receiving device 104.

[0046] The transmission device 102 includes a feedback recurrent autoencoder (FRAE) 110. Although a FRAE architecture is shown, the techniques described herein may operate using any architecture or circuit operable to encode data, reconstruct data based on an encoded version of the data, or both. The FRAE 110 is configured to receive a data stream comprising data arranged in a time series. For example, the data stream may include a time series of data samples 120, where each data sample 120 represents a time window portion of the data. Figure 1 As shown, data sample 120 includes data sample 120A, data sample 120B, and data sample 120C. Figure 1, but in other embodiments, additional data samples 120 may be included in the time series of data samples 120. As a non-limiting example, one or more data samples 120 may be disposed between data sample 120A and data sample 120B, one or more data samples 120 may be disposed between data sample 120B and data sample 120C, and so on. Data sample 120A includes data (e.g., extracted features) generated at an earlier time instance than data included in data sample 120B, and data sample 120B includes data generated at an earlier time instance than data included in data sample 120C. According to some embodiments, adjacent data samples 120 may include overlapping data (e.g., temporal redundancy). For example, a portion of the data in data sample 120A may also be included in data sample 120B. In some examples, the data included in data sample 120 includes media data, such as voice data, audio data, video data, game data, augmented reality data, other media data, or a combination thereof.

[0047] exist Figure 1 In the example of , FRAE 110 includes an encoder portion 113 and a decoder portion 115. Encoder portion 113 is configured to encode data samples 120A, 120C to generate corresponding latent vectors 124A, 124C. Each latent vector 124 represents output state data (also referred to as "output state") of the latent space of FRAE 110, which encodes the corresponding data sample 120A, 120C. As used herein, the term "vector" is not intended to limit the output state data to a specific data structure. For example, a latent vector may include any ordered arrangement of data values, such as but not limited to one or more vectors, one or more arrays, a set of index values, etc. For ease of illustration and description, data samples 120A, 120C encoded by encoder portion 113 of FRAE 110 are shaded gray and are referred to as "reference data samples". Data including or representing latent vectors 124A, 124C is sent to receiving device 104 as part of transmission 106. For example, latent codes representing latent vectors 124A, 124C may be sent via transmission 106 in one or more packets.

[0048] To reduce the amount of data bits sent to the receiving device, the encoder portion 113 of the FRAE 110 may bypass the encoding operation on certain data samples (such as data sample 120B) and send latent vectors 124A, 124C associated with reference data samples 120A, 120C. As described below, in these scenarios, machine learning predictive decoding (e.g., neural network 114) may be used to reconstruct data samples at the receiving device 104 that did not undergo the encoding operation at the FRAE 110. Figure 2 As described, in some embodiments, machine learning predictive decoding can also be used to reconstruct data samples that have not undergone encoding operations at FRAE 110 at the transmission device 102. For ease of illustration and description, data samples 120B that have not undergone encoding operations are unshaded and are referred to as "predicted data samples".

[0049] To encode the data sample 120A for transmission, the data sample 120A is provided to an encoder portion 113 of the FRAE 110 at a first time instance. The encoder portion 113 of the FRAE 110 is configured to generate a latent vector 124A for the data sample 120A. Data representing the latent vector 124A (e.g., a latent code corresponding to a quantized version of the latent vector 124A) is transmitted to the receiving device 104 in a data packet. The encoder portion 113 of the FRAE 110 may include a plurality of layers, such as one or more fully connected layers, one or more recurrent layers (e.g., one or more gated recurrent unit (GRU) layers), a bottleneck layer, or other layers. The one or more fully connected layers correspond to a feed-forward neural network architecture that includes a plurality of input nodes and generates one or more outputs based on different weighting and mapping functions. According to some embodiments, the fully connected layer may include a plurality of node levels (e.g., input level nodes, intermediate level nodes, and output level nodes) having unique weighting and mapping patterns. For ease of explanation, the fully connected layer is described as receiving one or more inputs (e.g., data samples 120A) and generating one or more outputs based on a neural network operation. However, it should be understood that the architecture of each fully connected layer described herein can be unique and can have a unique weighting and mapping pattern to generate a simple or complex neural network. The GRU layer is configured to generate input data (from one or more outputs of the fully connected layer) that is provided to the bottleneck layer. The GRU layer can generate input data using data associated with a previous time step. The bottleneck layer can generate a potential vector 124A based on the input data of the GRU layer.

[0050] To encode data sample 120C for transmission, data sample 120C is provided to encoder portion 113 of FRAE 110 at a second time instance after the first time instance. In particular embodiments, encoding of data sample 120C may be based in part on feedback (e.g., loop state) from decoder portion 115 of FRAE 110, wherein the feedback is related to decoding of a previous data sample (e.g., data sample 120A). According to one embodiment, data sample 120B is not provided to FRAE 110. According to another embodiment, data sample 120B is provided to FRAE 110, and FRAE 110 bypasses performance of encoding operations for data sample 120B.

[0051] The receiving device 104 includes a decoder portion 117. The decoder portion 117 of the receiving device 104 is a copy (e.g., another instance) of the decoder portion 115 of the FRAE 110 of the transmitting device 102. Upon receiving the latent vector 124A, the decoder portion 117 is configured to generate a reconstructed data sample 126A based on decoding the latent vector 124A. The reconstructed data sample 126A corresponds to a reconstructed version of the data sample 120A. For illustration, the decoder portion 117 of the receiving device 104 may include a GRU layer and one or more fully connected layers. The GRU layer is configured to be initialized using feedback (e.g., a loop state from a previous data sample) and to perform a decoding operation on the latent vector 124A to generate an output, which is provided to one or more fully connected layers for processing. One or more fully connected layers are configured to generate a reconstructed data sample 126A based on the output of the GRU layer to reconstruct the data sample 120A. The decoder portion 117 may operate in a substantially similar manner to generate a reconstructed data sample 126C based on the latent vector 124C. Reconstructed data sample 126C corresponds to a reconstructed version of data sample 120C. Although the above examples describe FRAE 110 as including one or more GRU layers and one or more fully connected layers, in other embodiments, FRAE 110 includes more layers, fewer layers, or different layers. For example, FRAE 110 may include one or more convolutional layers, one or more self-attention layers, one or more other types of recurrent or autoregressive layers, or a combination thereof.

[0052] The reconstructed data samples 126A and the reconstructed data samples 126C are provided as inputs to the neural network 114 to generate network predicted data samples 150. The network predicted data samples 150 correspond to predicted versions of the data samples 120B, which are disposed between the data samples 120A and the data samples 120C in the time series. Therefore, instead of using pure interpolation techniques to generate interpolated data samples based on the reconstructed data samples 126A, 126C, the system 100 implements a nonlinear and data-driven neural network model to predict the features of the data samples 120B and generate the network predicted data samples 150 based on the predictions. Therefore, the network predicted data samples 150 are more accurate representations of the data samples 120B than data samples generated based solely on interpolation. Figure 5 The operation and architecture of neural network 114 is described in more detail.

[0053] Figure 1The system enables the use of relatively few bits to accurately represent the data to be transmitted. For example, by reconstructing data sample 120B based on reconstructions of nearby data samples 120A, 120C (e.g., reconstructed data samples 126A, 126C) using machine learning predictive decoding (e.g., neural network 114), a network predicted data sample 150 of data sample 120B can be generated independently of data sample 120B. As a result, the encoding and transmission of data sample 120B can be bypassed at the transmitting device 102 to reduce the amount of data bits transmitted, and the receiving device 104 can reconstruct an accurate representation of data sample 120B based on the reconstructions of nearby data samples 120A, 120C. Therefore, despite the transmission of the encoded representation of data sample 120B being bypassed or in the absence of receiving the encoded representation of data sample 120B, the receiving device 104 can generate a relatively accurate representation of data sample 120B. For example, the neural network 114 can be used to generate a network predicted data sample associated with a data sample that is lost (e.g., unintentionally lost) or omitted (e.g., intentionally omitted).

[0054] Figure 2 2 is a diagram of a specific illustrative example of a system 200 configured to determine a residual vector of a reconstructed data sample generated using machine learning predictive coding. The system 200 includes a transmitting device 102 and a receiving device 104.

[0055] exist Figure 2 In the example shown, the transmission device 102 includes a reference Figure 1 100, each of which operates in a substantially similar manner as described above. For example, the transmission device 102 includes an encoder portion 113 of the FRAE 110 and a decoder portion 115 of the FRAE 110. In a manner similar to that described above, the encoder portion 113 of the FRAE 110 is configured to generate latent vectors 124A, 124C for data samples 120A, 120C, respectively.

[0056] exist Figure 2 In the example of FRAE 110, decoder portion 115 of FRAE 110 is configured to generate reconstructed data samples 226 based on latent vector 124. For example, decoder portion 115 of FRAE 110 may operate in a substantially similar manner as decoder portion 117 of receiving device 104 to generate reconstructed data samples 226 based on latent vector 124. Thus, reconstructed data samples 226 are substantially similar to reconstructed data samples 126A, 126C generated by decoder portion 117 of receiving device 104.

[0057] In addition, the transmitting device 102 includes a neural network 214 and a residual determination unit 202. The neural network 214 of the transmitting device 102 is a copy (e.g., another instance) of the neural network 114 of the receiving device 104, and is therefore configured to use machine learning predictive decoding to generate a network predicted data sample 250. The network predicted data sample 250 is substantially similar to the network predicted data sample 150.

[0058] In some scenarios, there may be slight differences between the network predicted data sample 250 and the data sample 120B. To account for the differences, the residual determination unit 202 is configured to determine a residual between the network predicted data sample 250 and the data sample 120B. Data describing the residual may be provided to the receiving device 104 to enable the receiving device 104 to adjust the network predicted sample (if appropriate) to better match the data sample 120B. For illustration, the data sample 120B and the network predicted data sample 250 are provided to the residual determination unit 202. The residual determination unit 202 is configured to determine a residual vector 280 associated with the network predicted data sample 250. The residual vector 280 may be based on a comparison of the data sample 120B and the network predicted data sample 250 (e.g., the difference between them).

[0059] The codebook 204 may be used to quantize the residual vector 280 to generate a residual code 282. For example, the processor (or quantizer) may map the value (eg, floating point value) of the residual vector 280 to a representative value of the codebook 204 to generate the residual code 282. Figures 3 to 5 As further described, the residual codes 282 may be packetized and sent to the receiving device to improve reconstruction of the data samples 120B at the receiving device 104 .

[0060] exist Figure 2 In the example of , data representing the residual vector 280 and data representing the output latent vectors 124A, 124C are included in the transmission 106 to the receiving device 104. For illustration, the residual code 282 (representing a quantized version of the residual vector 280) and the latent code (representing a quantized version of the latent vector 124) may be sent via the transmission 106 in one or more packets.

[0061] exist Figure 2In the example of , the receiving device 104 includes a residual reconstruction unit 251. The residual reconstruction unit 251 is configured to receive a residual code 282 from the transmitting device 102. The residual reconstruction unit 251 is further configured to modify the network predicted data sample 150 based on the residual code 282 to generate a modified network predicted data sample 152. Because the residual code 282 takes into account the residual between the data sample 120B and the network predicted data sample 250, modifying the network predicted data sample 150 based on the residual code 282 results in a more accurate representation of the data sample 120B (than the network predicted data sample 150) at the receiving device 104. Therefore, by modifying the network predicted data sample 150 based on the residual code 282, the receiving device 104 can further improve the reconstruction of the data sample 120B.

[0062] The system 200 enables a more accurate representation of the data to be transmitted using relatively fewer bits. For example, an accurate representation of the data sample 120B can be generated by reconstructing the predicted data sample 152 based on the reconstruction of the reference data samples 120A, 120C (e.g., the reconstructed data samples 126A, 126C) and the residual code 282 using machine learning predictive decoding (e.g., the neural network 114). As a result, the encoding and transmission of the data sample 120B can be bypassed at the transmitting device 102 to reduce the amount of data bits transmitted, and the receiving device 104 can use the neural network 114 to reconstruct a more accurate representation of the data sample 120B (e.g., the modified network predicted data sample 152) based on the reconstruction of the nearby data samples 120A, 120C (e.g., the reconstructed data samples 126A, 126C). Therefore, even if the transmission of the encoded representation of the data sample 120B is bypassed or if the encoded representation of the data sample 120B is not received, the receiving device 104 can generate a relatively accurate representation of the data sample 120B. For example, neural network 114 may be used to reconstruct data samples associated with lost packets.

[0063] Figure 3 3 is a diagram of a specific illustrative example of a system 300 operable to bundle data representing one or more latent vectors and data representing one or more residual vectors into a single packet. The system 300 includes a packet generator 304. The components of the system 300 can be integrated into the transmission device 102.

[0064] The packet generator 304 generates a first packet 340 during a first time instance. The data associated with the first time instance is Figure 3 In gray. Figure 3In the example shown, packet generator 304 receives a latent code 324A representing latent vector 124A. For example, latent code 324A may correspond to an encoded (e.g., quantized) version of latent vector 124A representing data sample 120A. Additionally, packet generator 304 receives residual code 282, which corresponds to an encoded (e.g., quantized) version of residual vector 280.

[0065] exist Figure 3 In the example shown, packet generator 304 includes latent code 324A and residual code 282 in first packet 340. Packet generator 304 is also configured to generate a header 342 for first packet 340. Header 342 may indicate a destination for first packet 340 and may indicate other properties of first packet 340. As a non-limiting example, header 342 may indicate that first packet 340 includes latent code 324A, residual code 282, and possibly other data (e.g., data representing additional data samples or residual vectors). In addition, header 342 may indicate that receiving device 104 may predict (e.g., reconstruct) one or more missing or omitted data samples (e.g., data sample 120B) from first packet 340 based on latent code 324A, latent code 324C from a later packet (e.g., second packet 350), and residual code 282. In some implementations, first packet 340 may include two or more latent codes 324 representing two or more data samples.

[0066] The packet generator 304 generates a second packet 350 during a second time instance after the first time instance. Figure 3 In the example shown, packet generator 304 receives a latent code 324C representing latent vector 124C. For example, latent code 324C may correspond to an encoded (e.g., quantized) version of latent vector 124C representing data sample 120C. Additionally, packet generator 304 receives a residual code 382 corresponding to an encoded (e.g., quantized) version of another residual vector (e.g., a residual vector subsequent to residual vector 280).

[0067] exist Figure 3In the example shown, packet generator 304 includes latent code 324C and residual code 382 in second packet 350. Packet generator 304 is also configured to generate a header 352 for second packet 350. Header 352 may indicate a destination for second packet 350 and may indicate other properties of second packet 350. As a non-limiting example, header 352 may indicate that second packet 350 includes latent code 324C, residual code 382, ​​and possibly other data. In addition, header 342 may indicate that receiving device 104 may predict (e.g., reconstruct) lost or omitted data samples based on latent code 324C, data from a later packet, and residual code 382.

[0068] Although potential code 324A and potential code 324C are depicted in different groups 340, 350, in some embodiments, potential code 324A and potential code 324C may be included in the same group. In addition, as described above, in other embodiments, potential codes representing more than two data samples may be included in a group. As a non-limiting example, in some embodiments, a group may include data representing five data samples. Although residual codes 282, 382 are included in Figure 3 340, 350, but the inclusion of residual codes 282, 382 is optional. In some embodiments, the number of data samples represented in the packet and the determination of whether to include a residual code in the packet can be based on network conditions. For example, if the network conditions cannot meet the threshold, the residual code 282, 382 can be included in the packet 340, 350.

[0069] Figure 3 The system enables the use of relatively fewer bits to accurately represent the data to be transmitted. For example, the encoding of data sample 120B can be bypassed, and the data representing nearby data samples 120A, 120C can be grouped and transmitted as described above. Therefore, the encoding and transmission of data sample 120B can be bypassed at the transmitting device 102 to reduce the amount of data bits transmitted, and the receiving device 104 can reconstruct an accurate representation of data sample 120B based on the reconstruction of nearby data samples 120A, 120C using machine learning predictive decoding described herein. As a result, the receiving device 104 can generate a more accurate representation of data sample 120B, despite bypassing the transmission of the encoded representation of data sample 120B.

[0070] Figure 4 is a diagram of a specific illustrative example of a system 400 including two or more devices configured to communicate via the transmission of encoded data. Figure 4The example shows a transmitting device 102 configured to encode and send data and a receiving device 104 configured to receive, decode, and use the data. Although the system 400 shows one transmitting device 102, the system 400 may include more than one transmitting device 102. For example, a two-way communication system may include two devices (e.g., mobile phones), and each device can send data to and receive data from the other device. That is, each device can act as both a transmitting device 102 and a receiving device 104. In another example, a single receiving device 104 can receive data from more than one transmitting device 102. Additionally or alternatively, the system 400 may include more than one receiving device 104. For example, a single transmitting device 102 can send (e.g., multicast or broadcast) data to multiple receiving devices 104. Thus, Figure 4 The one-to-one pairing of transmitting device 102 and receiving device 104 shown in FIG. 1 is merely illustrative of one configuration and is not intended to be limiting.

[0071] exist Figure 4 In the example of , the transmission device 102 includes multiple components arranged to obtain data from the data stream 404 and process the data to generate data packets (e.g., the first packet 340 and the second packet 350) that are sent via the transmission medium 432. Figure 4 , the components of the transmission device 102 include a feature extractor 406, a subsystem 410, a packet generator 304, a modem 428, and a transmitter 430. The subsystem 410 includes an encoder portion 113 of the FRAE 110 and a decoder portion 115 of the FRAE 110. Optionally, for example, Figure 4 In the example illustrated in , the subsystem 410 also includes one or more of the neural network 214, the residual determination unit 202, and the codebook 204. In other examples, the transmission device 102 may include more, fewer, or different components. For illustration, in some examples, the transmission device 102 includes one or more data generation devices configured to generate the data stream 404. Examples of such data generation devices include, for example, but not limited to, a microphone, a camera, a game engine, a media processor (e.g., a computer-generated graphics engine), an augmented reality engine, a sensor, or other devices and / or instructions configured to output the data stream 404. For further explanation, in some examples, the transmission device 102 includes a transceiver instead of the transmitter 430 (or the transmitter 430 is arranged therein).

[0072] Figure 4The data stream 404 in the example includes data arranged in a time series. For example, the data stream 404 may include a sequence of time window portions of the data. In some examples, the data packets include media data, such as voice data, audio data, video data, game data, augmented reality data, other media data, or a combination thereof.

[0073] The feature extractor 406 is configured to generate data samples 120 based on the data stream 404. The data samples 120 include data representing a portion of the data stream 404. The feature extraction techniques used by the feature extractor 406 may include, for example, data aggregation, interpolation, compression, windowing, domain transformation, sampling, smoothing, statistical analysis, etc. For illustration, when the data stream 404 includes speech data or other audio data, the feature extractor 406 may be configured to determine time domain or frequency domain spectral information describing the time window portion of the data stream 404. In this example, the data samples 120 may include spectral information. As a non-limiting example, the data samples 120 may include data describing the cepstrum of the speech data of the data stream 404, data describing the pitch associated with the speech data, other data indicating the characteristics of the speech data, or a combination thereof. As another illustrative example, when the data stream 404 includes video data, game data, or both, the feature extractor 406 may be configured to determine pixel information associated with an image frame of the data stream 404. In the same or other examples, data samples 120 may include other information, such as metadata associated with data stream 404 , compression data (eg, keyframe identifiers), or other information used by encoder portion 113 of FRAE 110 .

[0074] The encoder portion 113 of the FRAE 110 in the subsystem 410 is configured to encode the data samples 120 to generate a latent vector (eg, Figure 1 In certain embodiments, codebook 204 is used to encode each latent vector to generate a corresponding latent code 324. Latent code 324 is provided to packet generator 304. Optionally, in some embodiments, decoder portion 115 of FRAE 110 in the subsystem is configured to generate reconstructed data samples (e.g., Figure 2 In such an embodiment, the neural network 214 may generate network-predicted data samples (e.g., Figure 2 The network predicted data sample 250), and the residual determination unit 202 can generate a residual vector (eg, Figure 2 In this embodiment, the residual determination unit 202 provides the residual vector to the codebook 204 to generate a corresponding residual code, and the corresponding residual code is provided to the packet generator 304.

[0075] The packet generator 304 is configured to generate packets based on the potential code 324 and possibly other data. For example, the packet generator 304 may generate a first packet 340 based on the potential code 324A and the residual code, as described with respect to Figure 3 As described. In this example, the packet generator 304 may also generate a second packet 350 based on the latent code 324C. It should be understood that each packet 340, 350 may be generated based on data representing multiple data samples 120 rather than based on data representing a single data sample 120. As a non-limiting example, each packet 340, 350 may include data representing eight (8) data samples 120 (e.g., latent codes and optional residual codes). As another non-limiting example, each packet 340, 350 may include data representing four (4) data samples 120. Additionally, in some embodiments, a packet may include more than one residual code.

[0076] The modem 428 is configured to modulate the baseband according to a particular communication protocol to generate a signal representing the first packet 340 and the second packet 350. The transmitter 430 is configured to transmit the signal representing the packets 340, 350 via a transmission medium 432. The transmission medium 432 may include a wired medium, an optical medium, or a wireless medium. For illustration, the transmitter 430 may include or correspond to a wireless transmitter configured to transmit a signal via free space propagation of electromagnetic waves.

[0077] exist Figure 4 , the components of the receiving device 104 include a receiver 454, a modem 456, a depacketizer 458, one or more buffers 460, a decoder controller 465, one or more decoder networks 470, a renderer 478, and a user interface device 480. In other examples, the receiving device 104 may include more, fewer, or different components. For illustration, in some examples, the receiving device 104 includes more than one user interface device 480, such as one or more displays, one or more speakers, one or more tactile output devices, etc. For further explanation, in some examples, the receiving device 104 includes a transceiver instead of the receiver 454 (or the receiver 454 is arranged therein).

[0078] The receiver 454 is configured to receive signals representing the packets 340, 350 and provide the signals (after initial signal processing, such as amplification, filtering, etc.) to the modem 456. In some cases, the receiving device 104 may not receive all of the packets 340, 350 sent by the transmitting device 102. For example, one or more of the packets 340, 350 may be lost during transmission. Additionally or alternatively, the packets 340, 350 may be received in an order different from the order in which the transmitting device 102 sent them.

[0079] The modem 456 is configured to demodulate the signal to generate bits representing the received packets 340, 350 and provide the bits representing the received data packets to the depacketizer 458. The depacketizer 458 is configured to extract the latent code 324 from the payload of each received packet 340, 350 and store the latent code 324 at the buffer 460. For example, in Figure 4 , buffer 460 includes a jitter buffer 462 configured to store latent code 324 .

[0080] exist Figure 4 In the example shown, decoder controller 465 retrieves data from buffer 460 for decoder network 470. In some implementations, decoder controller 465 also performs buffer management operations, such as managing the depth of jitter buffer 462, the depth of playout buffer 474, or both. If decoder network 470 includes multiple decoders, decoder controller 465 can also determine which decoder to use at a particular time.

[0081] To generate the reconstructed data samples 126A, 126C, the decoder controller 465 provides the latent vector 124 to the decoder portion 117 of the decoder network 470. Figure 1In a manner similar to that described, the decoder portion 117 may generate reconstructed data samples 126A, 126C based on the latent vectors 124A, 124C. In addition, the decoder network 470 may include a neural network 114 configured to generate network-predicted data samples 150 using machine learning predictive decoding. Therefore, instead of using a pure interpolation technique to generate interpolated data samples based on the reconstructed data samples 126A, 126C, the decoder network 470 implements a nonlinear and data-driven neural network model to predict the features of the data sample 120B and generate the network-predicted data sample 150 based on the prediction. Optionally, the decoder network 470 may include a residual reconstruction unit 251. The residual reconstruction unit 251 may modify the network-predicted data sample 150 based on the residual code 282 to generate a modified network-predicted data sample 152. Therefore, the network-predicted data sample 150 (or the modified network-predicted data sample 152) may be a more accurate representation of the data sample 120B than the data sample generated based on interpolation. As described above, network predicted data sample 150 may be generated if encoding data associated with data sample 120B is lost during transmission, or if transmitting device 102 bypasses encoding of data sample 120B.

[0082] The reconstructed data samples 126A, 126C and the network predicted data samples 499 may be stored at the buffer 460 (e.g., at one or more play buffers 474). The network predicted data samples 499 stored in the buffer 460 may correspond to the network predicted data samples 150 or the modified network predicted data samples 152. During playback, the renderer 478 retrieves the data samples 126A, 499, 126C from the buffer 460 and processes the data samples 126A, 499, 126C to generate output signals, such as audio signals, video signals, game update signals, etc. The renderer 478 provides a signal to the user interface device 480 to generate a user-perceivable output based on the data samples 126A, 150, 126C. For example, the user-perceivable output may include one or more of a sound, an image, or a vibration. In some embodiments, the renderer 478 includes or corresponds to a game engine that generates a user-perceivable output in response to modifying the game state based on the data samples 126A, 150, 126C.

[0083] Figure 5 is a diagram of a specific illustrative example of a neural network architecture 500 configured to use machine learning predictive decoding to reconstruct a data sample using nearby reconstruction data. The neural network architecture 500 can be integrated into Figure 6 neural network 114, neural network 214, one or more neural networks 614A-614C, Figure 7One or more of the neural networks 714A-714D, or a combination thereof.

[0084] The neural network architecture 500 includes a convolutional layer 502, a fully connected layer 504, a GRU layer 506, a fully connected layer 508, and a deconvolutional layer 510. During operation, a reconstructed data sample 526A and a reconstructed data sample 526C are provided as input to the convolutional layer 502. According to one embodiment, the reconstructed data samples 526A, 526C correspond to the reconstructed data samples 126A, 126C or the reconstructed data sample 226.

[0085] The convolution layer 502 is configured to apply a convolution operation to an input (e.g., reconstructed data samples 526A, 526C) and provide the output vector of the convolution operation to the fully connected layer 504. The fully connected layer 504 is a feedforward neural network that includes multiple input nodes and generates one or more outputs based on different weighting and mapping functions. According to some embodiments, the fully connected layer may include multiple node levels (e.g., input level nodes, intermediate level nodes, and output level nodes) with unique weighting and mapping patterns. For ease of explanation, the fully connected layer 504 is described as receiving one or more inputs (e.g., the output vector of the convolution layer 502) and generating one or more outputs based on the neural network operation. However, it should be understood that the architecture of each fully connected layer described herein may be unique and may have a unique weighting and mapping pattern to generate a simple or complex neural network. The fully connected layer 504 is configured to generate one or more outputs based on the output vector of the convolution layer 502 and provide one or more outputs to the GRU layer 506.

[0086] The GRU layer 506 is configured to generate data states that are provided to the fully connected layer 508. The GRU layer 506 may also receive feedback (e.g., a recurrent state) associated with a previous time step. For example, the GRU layer 506 may access recurrent states from nearby time steps to generate data states. The data states generated by the GRU layer 506 are provided to the fully connected layer 508. The fully connected layer 508 is configured to generate outputs based on the data states generated by the GRU layer 506, and the deconvolution layer 510 is configured to generate network-predicted data samples 550 based on the outputs of the fully connected layer 508. According to one embodiment, the network-predicted data samples 550 correspond to the network-predicted data samples 150 or the network-predicted data samples 250.

[0087] Figure 6 is a diagram of a specific illustrative example of a system 600 configured to reconstruct multiple data samples for varying temporal locations using machine learning predictive coding. Figure 6 In the example shown, the system 600 includes Figure 6614A, neural network 614B, and neural network 614C. In some embodiments, neural networks 614A-614C are instances of a single neural network (e.g., a set of code corresponding to a neural network is executed multiple times, including a first execution of operations associated with neural network 614A, a second execution of operations associated with neural network 614B, and a third execution of operations associated with neural network 614C). In other embodiments, one or more of neural networks 614A-614C are different from the other neural networks. For illustration, neural network 614A can be different from neural networks 614B and 614C. System 600 can be integrated into transmitting device 102, receiving device 104, or both.

[0088] exist Figure 6 In the example of , five data samples 620 are shown. For example, Figure 6 Data sample 620A, data sample 620B, data sample 620C, data sample 620D, and data sample 620E are depicted. Data sample 620A includes data (e.g., extracted features) generated at an earlier time instance than the data included in data sample 620B, data sample 620B includes data generated at an earlier time instance than the data included in data sample 620C, etc. According to some embodiments, adjacent data samples 620 may include overlapping data (e.g., temporal redundancy). For example, a portion of the data in data sample 620A may also be included in data sample 620B. In some examples, the data included in data sample 620 includes media data, such as voice data, audio data, video data, game data, augmented reality data, other media data, or a combination thereof.

[0089] With Figure 1 In a similar manner to data samples 120A and 120C, Figure 6 In the example of , data samples 620A, 620E may be encoded by an encoder, such as encoder portion 113 of FRAE 110. After data samples 620A, 620E are encoded, a decoder (e.g., decoder portion 115 of transmitting device 102, decoder portion 117 of receiving device 104, or both) may reconstruct data samples 620A, 620E in a manner similar to that described above to generate reconstructed data samples 626A, 626E.

[0090] In order to reduce the amount of data bits sent to the receiving device 104, the system 600 can bypass the encoding operation on certain data samples (such as data samples 620B-620D). As described below, in these scenarios, machine learning predictive decoding (e.g., neural networks 614A-614C) can be used to reconstruct data samples 620B-620D that have not undergone the encoding operation. For ease of illustration and description, data samples 620B-620D that have not undergone the encoding operation are unshaded and are referred to as "predicted data samples."

[0091] To predict and reconstruct data sample 620C, reconstructed data sample 626A and reconstructed data sample 626E are provided as inputs to neural network 614A. In some embodiments, temporal position input 680A is provided to neural network 614A to indicate the temporal position of data sample 620C relative to data samples 620A, 620E. Figure 6 In the example of FIG. 6A , the time position input 680A has a value of one-half (1 / 2). That is, because the data sample 620C to be predicted is halfway between the data samples 620A, 620E used to generate the data input to the neural network 614A, providing a value of one-half as the time position input 680A instructs the neural network 614A to predict a data sample halfway between the two input data samples. In other embodiments, the time position input 680A is omitted. For example, the neural network 614A can be configured to predict a data sample at a particular time position relative to one or both of the reconstructed data samples 626A and 626E used as reference samples. For illustration, the neural network 614A can be selected for use Figure 4 The decoder controller 465 is used because the neural network 614A is configured to predict the data sample halfway between the two input data samples, and the data sample to be predicted is halfway between the reconstructed data samples 626A and 626E. In such an embodiment, other neural networks can be configured for predicting other time positions of the data samples.

[0092] The neural network 614A is configured to generate a network-predicted data sample 650A using machine learning predictive decoding. The network-predicted data sample 650A corresponds to a predicted version of the data sample 620C disposed between the data sample 620A and the data sample 620E. Therefore, instead of using a pure interpolation technique to generate an interpolated data sample based on the reconstructed data samples 626A, 626E, the system 600 implements a nonlinear and data-driven neural network model to predict the features of the data sample 620C and generate the network-predicted data sample 650A based on the prediction. Therefore, the network-predicted data sample 650A is a more accurate representation of the data sample 620C than a data sample generated based solely on interpolation.

[0093] Optionally, the neural network 614A may be provided with an input indicating whether the input data samples 626A, 626E are based on reference data samples. In the above example, the input data samples 626A, 626E are based on the reference data samples 620A, 620E (e.g., not predicted using a neural network). Thus, in this case, the neural network 614A may weight each input equally.

[0094] To predict and reconstruct data sample 620B, reconstructed data sample 626A and network predicted data sample 650A are provided as inputs to neural network 614B. A temporal position input 680B may also be provided to neural network 614B to indicate the temporal position of data sample 620B relative to data samples 620A, 620C. Figure 6 In the example of , the time position input 680B has a value of one-half (1 / 2). That is, because the data sample 620B to be predicted is halfway between the data samples 620A, 620C associated with the data input to the neural network 614B, providing a value of one-half as the time position input 680B indicates that the neural network 614B predicts a data sample halfway between the two input data samples. The neural network 614B is configured to generate a network-predicted data sample 650B using machine learning predictive decoding. The network-predicted data sample 650B corresponds to a predicted version of the data sample 620B disposed between the data sample 620A and the data sample 620C. Therefore, instead of using a pure interpolation technique to generate an interpolated data sample based on the data samples 626A, 650A, the system 600 implements a nonlinear and data-driven neural network model to predict the features of the data sample 620B and generate the network-predicted data sample 650B based on the prediction. Therefore, the network-predicted data sample 650B is a more accurate representation of the data sample 620B than the data sample generated based on interpolation alone.

[0095] Optionally, an input may be provided to the neural network 614B indicating whether the input data sample 626A, 650A is based on a reference data sample. In the above example, the reconstructed data sample 626A is based on the reference data sample 620A (e.g., not predicted using the neural network); however, the network-predicted data sample 650A is based on the predicted data sample. Therefore, in this case, the neural network 614B may assign more value (e.g., a heavier weight) to the reconstructed data sample 626A.

[0096] To predict and reconstruct data sample 620D, reconstructed data sample 626E and network predicted data sample 650A are provided as inputs to neural network 614C. A temporal position input 680C may also be provided to neural network 614C to indicate the temporal position of data sample 620D relative to data samples 620C, 620E. Figure 6 In the example of , the time position input 680C has a value of one-half (1 / 2). That is, because the data sample 620D to be predicted is halfway between the data samples 620C, 620E associated with the data input to the neural network 614C, providing a value of one-half as the time position input 680C instructs the neural network 614C to predict a data sample halfway between the two input data samples. The neural network 614C is configured to generate a network-predicted data sample 650C using machine learning predictive decoding. The network-predicted data sample 650C corresponds to a predicted version of the data sample 620D timing between the data sample 620C and the data sample 620E. Therefore, instead of using pure interpolation techniques to generate interpolated data samples based on data samples 650A, 626E, the system 600 implements a nonlinear and data-driven neural network model to predict the features of the data sample 620D and generate the network-predicted data sample 650C based on the prediction. As a result, the network-predicted data sample 650C is a more accurate representation of the data sample 620D than the data sample generated based on interpolation alone.

[0097] Optionally, the neural network 614C may be provided with an input indicating whether the input data sample 626E, 650A is based on a reference data sample. In the above example, the reconstructed data sample 626E is based on the reference data sample 620E (e.g., not predicted using the neural network); however, the network-predicted data sample 650A is based on the predicted data sample. Therefore, in this case, the neural network 614C may assign more value (e.g., a heavier weight) to the reconstructed data sample 626E.

[0098] Despite Figure 6 In the example of , each time position input 680 has a value of half, but in other embodiments, a particular time position input 680 may have a different value based on the time position of the data sample to be predicted 620 relative to the input data sample of the neural network 614. As a non-limiting example, if the neural network 614B is to predict and reconstruct the data sample 620B based on the reconstructed data samples 626A, 626E, the time position input 680B will have a value of four fifths (4 / 5). According to another embodiment, the time position input 680 may indicate the number of data samples between the data samples associated with the input of the neural network 614 and whether the data sample associated with the input of the neural network 614 is a reference data sample.

[0099] Figure 7 is a diagram of a FRAE architecture 700 configured to generate a single latent vector and auxiliary vectors 890 for multiple data samples. Figure 7 In the example shown, two time steps of FRAE are shown in an expanded manner (e.g., side by side) to facilitate describing the timewise interactions within FRAE. According to one embodiment, FRAE architecture 700 can be integrated into FRAE, such as Figure 1 FRAE 110.

[0100] The FRAE architecture 700 includes a convolutional layer 702, a linear layer 704, a GRU layer 706, a GRU layer 708, a linear layer 710, and a deconvolutional layer 712. Figure 7 In the example of , multiple instances of each layer 702-712 are shown. For example, a first instance of convolutional layer 702A and a second instance of convolutional layer 702B are shown. Each instance of convolutional layers 702A, 702B can indicate common circuitry that performs operations at different times. For example, the first instance of convolutional layer 702A can correspond to convolutional layer 702 performing operations at a first time, and the second instance of convolutional layer 702B can correspond to convolutional layer 702 performing operations at a second time. In a similar manner, in Figure 7 In the example of , there are multiple instances of linear layers 704A, 704B, multiple instances of GRU layers 706A, 706B, multiple instances of GRU layers 708A, 708B, multiple instances of linear layers 710A, 710B, and multiple instances of deconvolution layers 712A, 712B. The FRAE architecture 700 enables machine learning predictive decoding to be used for larger size packets. To illustrate, in Figure 7 In the example of FIG. 8 , five data samples 820E- 820A may be associated with a current grouping and five data samples 820F- 820J may be associated with a previous grouping.

[0101] During a first time instance, each data sample 820J-820F associated with a previous grouping is input into the FRAE architecture 700. For illustration, the data samples 820J-820F are provided to the convolution layer 702A at the same time. The data samples 820J-820F may undergo processing by the convolution layer 702A, the linear layer 704A, and the GRU layer 706A. Based on the processing, the data samples 820J-820F may be encoded to generate a single latent vector 724A. Therefore, instead of generating a latent vector for each data sample 820J-820F, the FRAE architecture 700 generates a single latent vector 724A representing the data samples 820J-820F. In addition, the FRAE architecture 700 is configured to generate an auxiliary vector 890A. The auxiliary vector 890A indicates a transition characteristic between two groups of data samples. According to one embodiment, the auxiliary vector 890A indicates a transition characteristic of the sound associated with the current grouping based on the data samples 820J-820F and the sound associated with the previous grouping. For example, auxiliary vector 890A may indicate whether the transition is smooth, abrupt, a vowel pronunciation transition, etc. Based on latent vector 724A, GRU layer 708A, linear layer 710A, and deconvolution layer 712A may generate representative data samples 826F (e.g., inverse frequency spectrograms or inverse frequency spectra of data samples 820J-820F). Figure 8 As described, representative data samples 826F may be used to predict data samples using machine learning predictive coding.

[0102] During the second time instance, each data sample 820E-820A associated with the current grouping is input into the FRAE architecture 700. For illustration, the data sample 820E-820A is provided to the convolution layer 702B at the same time. The data sample 820E-820A can undergo the processing of the convolution layer 702B, the linear layer 704B and the GRU layer 706B. The GRU layer 706B can receive feedback from the GRU layer 708A so that the encoding of the data sample 820E-820A is based on the encoding of the previous grouping. Based on this processing, the data sample 820E-820A can be encoded to generate a single potential vector 724B. Therefore, instead of generating a potential vector for each data sample 820E-820A, the FRAE architecture 700 generates a single potential vector 724B representing the data sample 820E-820A. In addition, the FRAE architecture 700 is configured to generate an auxiliary vector 890B. The auxiliary vector 890B indicates the transition characteristics between the two groups of data samples. According to one embodiment, auxiliary vector 890B indicates the transition characteristics of the sound associated with the current classification based on data samples 820E-820A, and indicates the transition characteristics of the sound associated with the previous classification based on data samples 820J-802F. For example, auxiliary vector 890B can indicate whether the transition is smooth, abrupt, a vowel pronunciation transition, etc. Based on latent vector 724B, GRU layer 708B, linear layer 710B, and deconvolution layer 712B can generate representative data samples 826A (e.g., cepstrograms or inverse spectra of data samples 820E-820A). As described with respect to Figure 8 As described, representative data sample 826A may be used to predict data samples using machine learning predictive coding.

[0103] As about Figure 8 As described, the auxiliary vector 890 can be used by the neural network to predict and reconstruct one or more data samples. For example, because the auxiliary vector 890 indicates the properties (e.g., transition characteristics) of the data sample to be predicted, the neural network can use the auxiliary vector 890 to improve the data sample reconstruction.

[0104] Figure 8 8 is a diagram of another specific illustrative example of a system 800 configured to reconstruct multiple data samples at varying time locations using machine learning predictive coding. The system 800 can be integrated into the transmitting device 102, the receiving device 104, or both. Figure 8 In the example of , six data samples 820 are shown. For example, Figure 8 Depicted are representative data sample 826A, data sample 820B, data sample 820C, data sample 820D, data sample 820E, and representative data sample 826F.

[0105] exist Figure 8In the example shown, the system 800 includes Figure 8 814B, and 814C). In some embodiments, neural networks 814A-814C are instances of a single neural network (e.g., a set of code corresponding to a neural network is executed multiple times, including a first execution of operations associated with neural network 814A, a second execution of operations associated with neural network 814B, and a third execution of operations associated with neural network 814C). In other embodiments, one or more of neural networks 814A-814C are different from the other neural networks. For illustration, neural network 814A can be different from neural networks 814B and 814C.

[0106] As about Figure 7 As described, to reduce the amount of data bits sent to receiving device 104, the individual encoding of data samples 820E-820A is bypassed, and data samples 820E-820A are jointly encoded to generate latent vector 724B, which can be reconstructed at receiving device 104 to generate representative data sample 826A. A similar process can be performed with respect to latent vector 724A to generate representative data sample 826F.

[0107] about Figure 8 The described techniques enable the receiving device 104 to reconstruct data samples 820E-820B using representative data samples 826A, 826F. To predict and reconstruct data sample 820D, representative data sample 826A and representative data sample 826F are provided as inputs to neural network 814A. A temporal position input 880A is provided to neural network 814A to indicate the relative timing of data sample 820D to data samples 820A, 820F. Figure 8In the example of , time position input 880A has a value of two fifths (2 / 5). That is, because data sample 820D to be predicted is two fifths of the way between data samples 820A, 820F used to generate the data input to neural network 814A, providing a value of two fifths as time position input 880A instructs neural network 814A to predict a data sample that is two fifths of the way between two input data samples. To illustrate, data sample 820E is one data sample away from representative data sample 826F, data sample 820D is two data samples away from representative data sample 826F, data sample 820C is three data samples away from representative data sample 826F, data sample 820B is four data samples away from representative data sample 826F, and representative data sample 826A is five data samples away from representative data sample 826F. Because the data sample to be predicted 820D is two data samples away from the representative data sample 826F, and the inputs to the neural network 814A (e.g., representative data samples 826F, 826A) are five data samples away from each other, the time position input 880A has a value of two fifths (2 / 5).

[0108] The neural network 814A is configured to generate a network-predicted data sample 850A using machine learning predictive decoding. The network-predicted data sample 850A corresponds to a predicted version of data sample 820D disposed between data sample 820A and data sample 820F. An auxiliary vector 890B (e.g., an auxiliary vector) is also provided to the neural network 814A. The auxiliary vector 890B indicates the transition characteristics between the sound associated with the current grouping based on data samples 820A-820E and the sound associated with the previous grouping based on at least data sample 820F. For example, the auxiliary vector 890B may indicate whether the transition is smooth, abrupt, a vowel pronunciation transition, etc. As described with respect to FIG. Figure 7 As depicted, auxiliary vector 890B is generated by GRU layer 706B.

[0109] To predict and reconstruct data sample 820E, representative data sample 826F and network predicted data sample 850A are provided as inputs to neural network 814B. A temporal position input 880B is provided to neural network 814B to indicate the relative timing of data sample 820E to data samples 820D, 820F. Figure 8In the example of , the time position input 880B has a value of one-half (1 / 2). That is, because the data sample 820E to be predicted is halfway between the data samples 820D, 820F used to generate the data input to the neural network 814B, providing a value of one-half as the time position input 880B indicates that the neural network 814B predicts a data sample halfway between the two input data samples. The neural network 814B is configured to generate a network-predicted data sample 850B using machine learning predictive decoding. The network-predicted data sample 850B corresponds to a predicted version of the data sample 820E between the data sample 820D and the data sample 820F. The auxiliary vector 890 is also provided to the neural network 814B.

[0110] To predict and reconstruct data sample 820C, representative data sample 826A and network-predicted data sample 850A are provided as inputs to neural network 814C. A temporal position input 880C is provided to neural network 814C to indicate the temporal position of data sample 820C (e.g., relative to data samples 820D, 820A). Figure 8 In the example of , the time position input 880C has a value of one third (1 / 3). That is, because the data sample 820C to be predicted is one third between the data samples 820D, 820A used to generate the data input to the neural network 814C, providing a value of one third as the time position input 880C instructs the neural network 814C to predict a data sample one third between the two input data samples. The neural network 814C is configured to generate a network-predicted data sample 850C using machine learning predictive decoding. The network-predicted data sample 850C corresponds to a predicted version of the data sample 820C between the data sample 820D and the data sample 820A. The auxiliary vector 890 is also provided to the neural network 814C.

[0111] To predict and reconstruct data sample 820B, representative data sample 826A and network-predicted data sample 850C are provided as inputs to neural network 814D. A temporal position input 880D is provided to neural network 814D to indicate the temporal position of data sample 820B (e.g., relative to data samples 820C, 820A). Figure 8In the example of FIG. 8 , the time position input 880D has a value of one-half (1 / 2). That is, because the data sample 820B to be predicted is halfway between the data samples 820C, 820A used to generate the data input to the neural network 814D, providing a value of one-half as the time position input 880D instructs the neural network 814D to predict a data sample halfway between the two input data samples. The neural network 814D is configured to generate a network-predicted data sample 850D using machine learning predictive decoding. The network-predicted data sample 850D corresponds to a predicted version of the data sample 820B between the data sample 820C and the data sample 820A. The auxiliary vector 890 is also provided to the neural network 814D.

[0112] Fig. 9 An embodiment 900 is depicted in which a device 902 includes one or more processors 910 including components for encoding and reconstructing data samples as described herein. For example, the one or more processors 910 include an encoder 913, a decoder 915, a neural network 914, a residual determination unit 202, and a residual reconstruction unit 251. The encoder 913 may correspond to the encoder portion 113 of the FRAE 110, the FRAE architecture 700, or both. The decoder 915 may correspond to the decoder portion 115 of the FRAE 110, the decoder portion 117 of the receiving device 104, or both. The neural network 914 may correspond to the neural network 114, the neural network 214, the neural network architecture 500, the neural network 614A-614C, the neural network 814A-814D, or a combination thereof.

[0113] The device 902 also includes an input interface 904 (e.g., one or more wired or wireless interfaces) configured to receive the data stream 404 and an output interface 906 (e.g., one or more wired or wireless interfaces) configured to provide the reconstructed data samples 926 to another device, such as Figure 4The playback buffer 474 or playback device (e.g., a speaker) of the device 902 may be a playback buffer 474 or a playback device (e.g., a speaker). The data stream 404 may include data samples 120, data samples 620, data samples 820, or a combination thereof. The reconstructed data samples 926 may include reconstructed data samples 126, network predicted data samples 150, reconstructed data samples 226, modified network predicted data samples 152, reconstructed data samples 526, network predicted data samples 550, reconstructed data samples 626, network predicted data samples 650, reconstructed data samples 826, network predicted data samples 850, or a combination thereof. The device 902 may correspond to a system on a chip or other modular device that can be integrated into other systems to provide audio encoding and decoding, such as within a mobile phone, another communication device, an entertainment system, or a vehicle, as illustrative, non-limiting examples. In some embodiments, device 902 may be integrated into a server, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a camera, a navigation device, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, a motor vehicle (e.g., a car), or any combination thereof.

[0114] In the illustrated embodiment 900, the apparatus 902 includes a memory 920 (e.g., one or more memory devices) including instructions 922. The apparatus 902 also includes one or more processors 910 coupled to the memory 920 and configured to execute the instructions 922 from the memory 920. In the embodiment 900, the encoder 913, the decoder 915, the neural network 914, the residual determination unit 202, and / or the residual reconstruction unit 251 may correspond to the instructions 922 or be implemented via the instructions 922. For example, when the instructions 922 are executed by the processor 910, the processor 910 may generate a first reconstructed data sample based on a first latent vector of FRAE. The processor 910 may also generate a second reconstructed data sample based on a second latent vector of FRAE. The processor 910 may also provide the first reconstructed data sample and the second reconstructed data sample as input to the neural network 914. The neural network 914 is configured to generate a network-predicted data sample using machine learning predictive decoding.

[0115] As an illustrative, non-limiting example, Fig.10An embodiment 1000 is depicted in which the device 902 is integrated into a mobile device 1002, such as a phone or tablet computer. The mobile device 1002 includes a microphone 1010 positioned to primarily capture the user's voice, a speaker 1020 configured to output sound, and a display screen 1004. The device 902 can generate a first reconstructed data sample based on a first latent vector of FRAE. The device 902 can also generate a second reconstructed data sample based on a second latent vector of FRAE. The device 902 can also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample can be processed by the speaker 1020 and output as sound.

[0116] Fig.11 An embodiment 1100 is depicted in which the device 902 is integrated into a head mounted device 1102. The head mounted device 1102 includes a microphone 1110 positioned to primarily capture the user's voice and one or more head mounted devices 1120. The device 902 may generate a first reconstructed data sample based on a first latent vector of the FRAE. The device 902 may also generate a second reconstructed data sample based on a second latent vector of the FRAE. The device 902 may also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network predicted data sample using machine learning predictive decoding. The network predicted data sample may be processed by the head mounted device 1120 and output as sound.

[0117] Fig.12 An embodiment 1200 is depicted in which the device 902 is integrated into a wearable electronic device 1202 (illustrated as a "smart watch"). The wearable electronic device 1202 may include a microphone 1210, a speaker 1220, and a display screen 1204. The device 902 may generate a first reconstructed data sample based on a first latent vector of FRAE. The device 902 may also generate a second reconstructed data sample based on a second latent vector of FRAE. The device 902 may also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample may be processed by the speaker 1220 and output as sound.

[0118] Fig.131300 is an embodiment in which the device 902 is integrated into a wireless speaker and voice activated device 1302. The wireless speaker and voice activated device 1302 may have wireless network connectivity and be configured to perform auxiliary operations. The wireless speaker and voice activated device 1302 includes a microphone 1310 and a speaker 1320. The device 902 may generate a first reconstructed data sample based on a first latent vector of FRAE. The device 902 may also generate a second reconstructed data sample based on a second latent vector of FRAE. The device 902 may also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network predicted data sample using machine learning predictive decoding. The network predicted data sample may be processed by the speaker 1320 and output as sound.

[0119] Fig.14 An embodiment 1400 is depicted in which the device 902 is integrated into a portable electronic device corresponding to the camera device 1402. The camera device 1402 includes a microphone 1410 and a speaker 1420. The device 902 can generate a first reconstructed data sample based on a first latent vector of the FRAE. The device 902 can also generate a second reconstructed data sample based on a second latent vector of the FRAE. The device 902 can also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample can be processed by the speaker 1420 and output as sound.

[0120] Fig.15 An embodiment 1500 is depicted in which the device 902 is integrated into a portable electronic device corresponding to an extended reality (“XR”) head mounted device 1502, such as a virtual reality (“VR”), augmented reality (“AR”), or mixed reality (“MR”) head mounted device. The visual interface is located in front of the user's eyes to enable the display of augmented reality or virtual reality images or scenes to the user when the head mounted device 1502 is worn. The device 902 may generate a first reconstructed data sample based on a first latent vector of the FRAE. The device 902 may also generate a second reconstructed data sample based on a second latent vector of the FRAE. The device 902 may also provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network 914. The neural network 914 is configured to generate a network predicted data sample using machine learning predictive decoding. The network predicted data sample may be processed and output by a speaker 1520. In a specific example, the visual interface device is configured to display a notification indicating the user's voice from the microphone 1510 or a notification indicating the user's voice from the sound output by the speaker 1520.

[0121] Fig.16 An embodiment 1600 is depicted in which device 902 corresponds to or is integrated within vehicle 1602, which is illustrated as a manned or unmanned aerial device (e.g., a package delivery drone). Vehicle 1602 includes microphone 1610 and speaker 1620. Device 902 can generate a first reconstructed data sample based on a first latent vector of FRAE. Device 902 can also generate a second reconstructed data sample based on a second latent vector of FRAE. Device 902 can also provide the first reconstructed data sample and the second reconstructed data sample as input to neural network 914. Neural network 914 is configured to generate network-predicted data samples using machine learning predictive decoding. The network-predicted data samples can be processed by speaker 1620 and output as sound.

[0122] Fig.17 Another embodiment 1700 is depicted, in which the device 902 corresponds to or is integrated in the vehicle 1702, which is shown as a car. The vehicle 1702 also includes a microphone 1710 and a speaker 1720. The microphone 1710 is positioned to capture the speech of the operator of the vehicle 1702. The device 902 can generate a first reconstructed data sample based on a first latent vector of the FRAE. The device 902 can also generate a second reconstructed data sample based on a second latent vector of the FRAE. The device 902 can also provide the first reconstructed data sample and the second reconstructed data sample as input to the neural network 914. The neural network 914 is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample can be processed by the speaker 1720 and output as sound. One or more operations of the vehicle 1702 can be initiated based on one or more detected keywords (e.g., "unlock", "start the engine", "play music", "display the weather forecast" or another voice command), such as by providing feedback or information via the display 1722 or the speaker 1720.

[0123] Fig.18 is a flowchart of a specific example of a method 1800 of operation of a communication device. In various embodiments, the method 1800 may be performed by Figure 1 System 100, Figure 2 System 200, Figure 4 system 400, a transmission device 102, a receiving device 104, Figure 5 Neural network architecture 500, Figure 6 system 600, FRAE architecture 700 or Figure 8 One or more of the systems 800 are executed.

[0124] exist Fig.18In the example of , method 1800 includes generating a first reconstructed data sample based on a first latent vector of the FRAE at block 1802. The first reconstructed data sample corresponds to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream. For example, referring to Figure 1 , the decoder portion 117 generates a reconstructed data sample 126A based on the latent vector 124A of the FRAE 110. The reconstructed data sample 126A corresponds to a reconstructed version of the data sample 120A in the time series of data samples 120 of the portion of the data stream 404.

[0125] Method 1800 also includes generating a second reconstructed data sample based on the second latent vector of the FRAE at block 1804. The second reconstructed data sample corresponds to a reconstructed version of the second data sample in the time series of data samples. Figure 1 , the decoder portion 117 generates a reconstructed data sample 126C based on the latent vector 124C of the FRAE 110. The reconstructed data sample 126C corresponds to a reconstructed version of the data sample 120C in the time series of data samples 120.

[0126] The method 1800 also includes providing the first reconstructed data sample and the second reconstructed data sample as input to the neural network at block 1806. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a particular data sample in the time series of data samples, and the particular data sample is arranged between the first data sample and the second data sample. For example, referring to Figure 1 , reconstructed data samples 126A, 126C are provided as inputs to neural network 114. Neural network 114 uses machine learning predictive coding to generate network predicted data sample 150, which corresponds to a predicted version of data sample 120B in the time series of data samples 120. Data sample 120B is disposed between data sample 120A and data sample 120C.

[0127] According to one embodiment in which method 1800 is performed by a transmitting device, method 1800 includes determining a residual vector associated with a data sample predicted by the network. Figure 2 , the residual determination unit 202 determines a residual vector 280 associated with the network predicted data sample 250. The residual vector 280 may be determined based on a comparison of the data sample 120B and the network predicted data sample 250. The method 1800 may also include quantizing the residual vector using a codebook to generate a residual code. For example, referring to Figure 2 , quantizing the residual vector 280 using the codebook 204 to generate a residual code 282. The method 1800 may also include sending the residual code to a receiving device. Figure 3 and Figure 4 , the transmitting device 102 sends the residual code 282 as part of the first packet 340 to the receiving device 104. Because sending the residual code 282 as part of the first packet 340 increases the number of bits sent, in some scenarios, the residual vector 280 is determined and quantized in response to determining that the network conditions fail to meet the threshold. As a non-limiting example, if the network traffic is above a certain threshold so that the network is relatively congested, the residual vector 280 can be determined, quantized, and sent because the possibility of packet loss is relatively high. However, if the network traffic is below a certain threshold, the transmitting device 102 can bypass the determination of the residual vector 280.

[0128] According to one embodiment, method 1800 includes providing a network-predicted data sample and a first reconstructed data sample as input to a neural network. The neural network is configured to generate another network-predicted data sample using machine learning predictive decoding. The other network-predicted data sample corresponds to a predicted version of another specific data sample in the time series of data samples, and the other specific data sample is arranged between the first data sample and the specific data sample. For example, referring to Figure 6 , the network predicted data sample 650A and the reconstructed data sample 626A are provided as inputs to the neural network 614B. The neural network 614B uses machine learning predictive decoding to generate the network predicted data sample 650B. The network predicted data sample corresponds to a predicted version of the data sample 620B disposed between the data sample 620A and the data sample 620C. According to one embodiment, the method 1800 also includes providing a time position input to the neural network. The time position input indicates the time position of another specific data sample (e.g., relative to the first data sample and the specific data sample). For example, with reference to Fig.18 , the time position input 680B is provided to the neural network 614B to indicate the time position of the data sample 620B.

[0129] According to one embodiment in which the method 1800 is performed by a sending device, the method 1800 includes sending data representing the first latent vector as part of a first packet to a receiving device. Figure 4, the transmitting device 102 sends the latent code 324A (which corresponds to data representing the first latent vector) as part of a first packet 340 to the receiving device 104. The first packet 340 has a relatively small number of bits dedicated to the data sample 120B. For example, in some scenarios, the first packet 340 does not include any dedicated bits for the data sample 120B. In these scenarios, as described above, machine learning predictive decoding is used to reconstruct the data sample 120B. In other scenarios, the first packet 340 has a small number of bits dedicated to the data sample 120B. For example, data associated with the residual code 282 is included in the first packet 340 and is used to reconstruct the data sample 120B at the receiving device 104. The method 1800 may also include sending the data representing the second latent vector to the receiving device as part of a second packet. For example, referring to Figure 4 , transmitting device 102 sends latent code 324C (which corresponds to data representing the second latent vector) to receiving device 104 as part of second packet 350.

[0130] According to one embodiment of the method 1800 being performed by a receiving device, the method 1800 includes receiving data representing a first latent vector from a sending device. Figure 4 , receiving device 104 receives latent code 324A from transmitting device 102, wherein latent code 324A includes or corresponds to data representing latent vector 124A. Method 1800 may also include receiving data representing a second latent vector from a transmitting device. Figure 4 , receiving device 104 receives latent code 324C from transmitting device 102, where latent code 324C includes or corresponds to data representing latent vector 124C.

[0131] According to an embodiment in which the method 1800 is performed by a receiving device, the method 1800 includes receiving a residual code from a transmitting device. Figure 3 and 4 , the receiving device 104 receives the residual code 282 from the transmitting device 102 as part of the first packet 340. The method 1800 may also include modifying the network predicted data samples based on the residual code. Figure 2 , the residual reconstruction unit 251 modifies the network predicted data sample 150 based on the residual code 282 to generate a modified network predicted data sample 152.

[0132] Fig.18The method 1800 enables an accurate representation of data to be transmitted using relatively few bits. For example, by using machine learning predictive decoding (e.g., neural network 114) to reconstruct data sample 120B based on reconstructions of nearby data samples 120A, 120C (e.g., based on reconstructed data samples 126A, 126C), a network predicted data sample 150 of data sample 120B can be generated. As a result, encoding and transmission of data sample 120B can be bypassed at transmitting device 102 to reduce the amount of data bits transmitted, and receiving device 104 can use neural network 114 to reconstruct an accurate representation of data sample 120B (e.g., modified network predicted data sample 152) based on reconstructions of nearby data samples 120A, 120C (e.g., reconstructed data samples 126A, 126C). Therefore, receiving device 104 can generate a relatively accurate representation of data sample 120B even if transmission of an encoded representation of data sample 120B is bypassed or if an encoded representation of data sample 120B is not received. For example, neural network 114 may be used to reconstruct data samples associated with lost packets.

[0133] Fig.18 The method 1800 may be implemented by a field programmable gate array (FPGA) device, an application specific integrated circuit (ASIC), a processing unit such as a central processing unit (CPU), a digital signal processor (DSP), a graphics processing unit (GPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.18 The method 1800 may be performed by a processor executing instructions, such as reference Fig. 22 as described by processor 2210.

[0134] Fig.19 is a flowchart of another specific example of a method 1900 of operation of a communication device. In various embodiments, the method 1900 may be performed by Figure 1 System 100, Figure 2 System 200, Figure 4 system 400, a transmission device 102, a receiving device, Figure 5 Neural network architecture 500, Figure 6 system 600, FRAE architecture 700 or Figure 8 One or more of the systems 800 are executed.

[0135] exist Fig.19 In the example of , method 1900 includes generating a first reconstructed data sample based on the first encoding at block 1902. The first reconstructed data sample corresponds to a reconstructed version of a first data sample in a time sequence of data samples of a portion of a data stream. For example, referring to Figure 1, the decoder portion 117 generates a reconstructed data sample 126A based on the latent vector 124A of the FRAE 110. The reconstructed data sample 126A corresponds to a reconstructed version of the data sample 120A in the time series of data samples 120 of the portion of the data stream 404.

[0136] Method 1900 also includes generating a second reconstructed data sample based on the second encoding at block 1904. The second reconstructed data sample corresponds to a reconstructed version of a second data sample in the time series of data samples. Figure 1 , the decoder portion 117 generates a reconstructed data sample 126C based on the latent vector 124C of the FRAE 110. The reconstructed data sample 126C corresponds to a reconstructed version of the data sample 120C in the time series of data samples 120.

[0137] The method 1900 also includes providing the first reconstructed data sample and the second reconstructed data sample as input to the neural network at block 1906. The neural network is configured to generate a network-predicted data sample using machine learning predictive decoding. The network-predicted data sample corresponds to a predicted version of a particular data sample in the time series of data samples, and the particular data sample is arranged between the first data sample and the second data sample. For example, referring to Figure 1 , reconstructed data samples 126A, 126C are provided as inputs to neural network 114. Neural network 114 uses machine learning predictive coding to generate network predicted data sample 150, which corresponds to a predicted version of data sample 120B in the time series of data samples 120. Data sample 120B is disposed between data sample 120A and data sample 120C.

[0138] Fig.19 The method 1900 may be implemented by an FPGA device, an ASIC, a processing unit (such as a CPU, a DSP, a GPU), a controller, another hardware device, a firmware device, or any combination thereof. As an example, Fig.19 The method 1900 may be performed by a processor executing instructions, such as reference Fig. 22 as described by processor 2210.

[0139] Fig. 20An implementation 2000 is depicted in which a device 2002 includes one or more processors 2010 that include components of a transmission device 102. The device 2002 also includes an input interface 2004 (e.g., one or more buses or wireless interfaces) configured to receive input data (such as data stream 404), and an output interface 2006 (e.g., one or more buses or wireless interfaces) configured to output data 2014 (such as packets 340, 350). The device 2002 may correspond to a system-on-chip or other modular device that may be integrated into other systems to provide data encoding, such as within a mobile phone, another communication device, an entertainment system, or a vehicle, as illustrative, non-limiting examples. In some embodiments, device 2002 may be integrated into a server, a mobile communication device, a smart phone, a cellular phone, a laptop computer, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a digital video disc (DVD) player, a tuner, a camera, a navigation device, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, a motor vehicle (e.g., a car), or any combination thereof.

[0140] In the illustrated embodiment 2000, the device 2002 includes a memory 2020 (e.g., one or more memory devices) including instructions 2022 and one or more codebooks 204. The device 2002 also includes one or more processors 2010 coupled to the memory 2020 and configured to execute the instructions 2022 from the memory 2020. In this embodiment 2000, the feature extractor 406, the subsystem 410, the neural network 214, and the packet generator 304 can correspond to the instructions 2022 or be implemented via the instructions 2022. The subsystem 410 includes an encoder portion 113 and a decoder portion 115. When the instructions 2022 are executed by the processor 2010, the processor 2010 can generate a reconstructed data sample 226 based on the latent vector 124. The processor 2010 can also provide the reconstructed data sample 226 as an input to the neural network 214 to generate a network predicted data sample 250.

[0141] Fig.21 An embodiment 2100 is depicted in which a device 2102 includes one or more processors 2110 that include components of a receiving device 104. The device 2102 also includes a Figure 4 The receiver 454 receives the input data 2112 (e.g., packets 340, 350) from the input interface 2104 (e.g., one or more buses or wireless interfaces), and is configured to receive the input data 2112 (e.g., packets 340, 350) based on the input data 2112 (e.g., provided to Figure 4The device 2102 may be configured to receive a signal from a user interface device 480 of the user interface device 480 and provide an output interface 2106 (e.g., one or more buses or wireless interfaces) for output 2114. The device 2102 may correspond to a system on a chip or other modular device that can be integrated into other systems to provide data decoding, such as within a mobile phone, another communication device, an entertainment system, or a vehicle, as illustrative, non-limiting examples. According to some embodiments, the device 2102 may be integrated into a server, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a DVD player, a tuner, a camera, a navigation device, a headset, an augmented reality headset, a mixed reality headset, a virtual reality headset, a motor vehicle (e.g., a car), or any combination thereof.

[0142] In the illustrated embodiment 2100, the device 2102 includes a memory 2120 (e.g., one or more memory devices) including instructions 2122 and one or more buffers 460. The device 2102 also includes one or more processors 2110 coupled to the memory 2120 and configured to execute the instructions 2122 from the memory 2120. In this embodiment 2100, the depacketizer 458, the decoder controller 465, the decoder network 470, the decoder 472, and / or the renderer 478 may correspond to the instructions 2122 or be implemented via the instructions 2122. For example, when the instructions 2122 are executed by the processor 2110, the processor 2110 may generate a reconstructed data sample 126A based on the latent vector 124A. The processor 2110 may also generate a reconstructed data sample 126C based on the latent vector 124C. The processor 2110 may also provide the reconstructed data samples 126A, 126C as input to the neural network 114 to generate a network-predicted data sample 150.

[0143] refer to Fig. 22 , a block diagram of a particular illustrative implementation of a device is depicted and generally designated 2200. In various implementations, the device 2200 may have more Fig. 22 In an illustrative embodiment, the device 2200 may correspond to the transmitting device 102, the receiving device 104, or both. In an illustrative embodiment, the device 2200 may perform the reference Figure 1-Figure 21 Describes one or more operations.

[0144] In a particular embodiment, the device 2200 includes a processor 2206 (e.g., a CPU). The device 2200 may include one or more additional processors 2210 (e.g., one or more DSPs, one or more GPUs, or a combination thereof). The processor 2210 may include a speech and music coder-decoder (CODEC) 2208. The speech and music codec 2208 may include a speech coder ("vocoder") encoder 2236, a vocoder decoder 2238, or both. In a particular aspect, the vocoder encoder 2236 includes the FRAE 110, the neural network 214, and the residual determination unit 202. In a particular aspect, the vocoder decoder 2238 includes a decoder portion 117 and the neural network 114.

[0145] The device 2200 also includes a memory 2286 and a CODEC 2234. The memory 2286 may include instructions 2256 that may be executed by one or more additional processors 2210 (or processor 2206) to implement the functionality described with reference to the transmitting device 102, the receiving device 104, or both. The device 2200 may include a modem 2240 coupled to an antenna 2290 via a transceiver 2250.

[0146] Device 2200 may include a display 2228 coupled to a display controller 2226. Speaker 2296 and microphone 2294 may be coupled to CODEC 2234. CODEC 2234 may include a digital-to-analog converter (DAC) 2202 and an analog-to-digital converter (ADC) 2204. In particular embodiments, CODEC 2234 may receive an analog signal from microphone 2294, convert the analog signal into a digital signal using analog-to-digital converter 2204, and provide the digital signal to voice and music codec 2208 (e.g., as a Figure 4 The speech and music codec 2208 can process the digital signal. In a particular embodiment, the speech and music codec 2208 can process the digital signal (e.g., Figure 4 The renderer 478 output) is provided to the codec 2234. The CODEC 2234 can convert the digital signal into an analog signal using the digital-to-analog converter 2202, and can provide the analog signal to the speaker 2296.

[0147] In certain embodiments, the device 2200 may be included in a system-in-package or system-on-chip device 2222 corresponding to the transmitting device 102 or the receiving device 104. In certain embodiments, the memory 2286, the processor 2206, the processor 2210, the display controller 2226, the CODEC 2234, and the modem 2240 are included in the system-in-package or system-on-chip device 2222. In certain embodiments, the input device 2230 and the power supply 2244 are coupled to the system-in-package or system-on-chip device 2222. In addition, in certain embodiments, as shown in FIG. Fig. 22 2200, the display 2228, input device 2230, speaker 2296, microphone 2294, antenna 2290, and power supply 2244 are external to the system-in-package or system-on-chip device 2222. In a particular embodiment, each of the display 2228, input device 2230, speaker 2296, microphone 2294, antenna 2290, and power supply 2244 can be coupled to a component of the system-in-package or system-on-chip device 2222, such as an interface or controller. In some embodiments, the device 2200 includes additional memory that is external to the system-in-package or system-on-chip device 2222 and coupled to the system-in-package or system-on-chip device 2222 via an interface or controller.

[0148] Device 2200 may include a smart speaker (e.g., processor 2206 may execute instructions 2256 to run a voice-controlled digital assistant application), a speaker boom, a mobile communication device, a smart phone, a cellular phone, a laptop, a computer, a tablet computer, a personal digital assistant, a display device, a television, a game console, a music player, a radio, a digital video player, a DVD player, a tuner, a camera, a navigation device, a head-mounted device, an augmented reality head-mounted device, a mixed reality head-mounted device, a virtual reality head-mounted device, a vehicle, or any combination thereof.

[0149] In conjunction with the described embodiments, the apparatus includes a component for generating a first reconstructed data sample based on a first latent vector of a feedback loop autoencoder (FRAE). The first reconstructed data sample corresponds to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream. For example, the component for generating the first reconstructed data sample includes a decoder portion 115 of the FRAE 110, a decoder portion 117 of the receiving device 104, a subsystem 410, a processor 910, a processor 2206, a processor 2210, a speech and music codec 2208, a vocoder decoder 2238, one or more other circuits or components configured to generate the first reconstructed data sample, or any combination thereof.

[0150] The apparatus also includes a component for generating a second reconstructed data sample based on a second latent vector of the FRAE. The second reconstructed data sample corresponds to a reconstructed version of the second data sample in the time series of the data sample. For example, the component for generating the second reconstructed data sample includes the decoder portion 115 of the FRAE 110, the decoder portion 117 of the receiving device 104, the subsystem 410, the processor 910, the processor 2206, the processor 2210, the speech and music codec 2208, the vocoder decoder 2238, one or more other circuits or components configured to generate the second reconstructed data sample, or any combination thereof.

[0151] The apparatus also includes a component for providing the first reconstructed data sample and the second reconstructed data sample as input to the neural network. The neural network is configured to use machine learning predictive decoding to generate a network-predicted data sample. The network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of data samples, and the specific data sample is arranged between the first data sample and the second data sample. For example, the component for providing includes FRAE 110, a decoder portion 117 of the receiving device 104, a processor 910, a processor 2206, a processor 2210, a speech and music codec 2208, a vocoder decoder 2238, one or more other circuits or components configured to provide the reconstructed data sample as input to the neural network, or any combination thereof.

[0152] In some embodiments, the non-transitory computer-readable medium includes instructions that, when executed by one or more processors of the device, cause the one or more processors to generate a first reconstructed data sample (e.g., reconstructed data sample 126A) based on a first latent vector of the FRAE (e.g., latent vector 124A). The first reconstructed data sample corresponds to a reconstructed version of a first data sample (e.g., data sample 120A) in a time series of data samples (e.g., data sample 120) of a portion of a data stream (e.g., data stream 404). Execution of the instructions also causes the one or more processors to generate a second reconstructed data sample (e.g., reconstructed data sample 126C) based on a second latent vector of the FRAE (e.g., latent vector 124C). The second reconstructed data sample corresponds to a reconstructed version of a second data sample (e.g., data sample 120C) in the time series of data samples. Execution of the instructions also causes the one or more processors to provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network (e.g., neural network 114). The neural network is configured to generate a network-predicted data sample (e.g., network-predicted data sample 150) using machine learning predictive decoding. The data sample predicted by the network corresponds to a predicted version of a specific data sample (eg, data sample 120B) in the time series of data samples, and the specific data sample is disposed between the first data sample and the second data sample.

[0153] Certain aspects of the present disclosure are described below in sets of interrelated examples:

[0154] Example 1

[0155] A device comprises: a memory; and one or more processors, which are coupled to the memory and are operably configured to: generate a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream; generate a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of the data samples; and provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network, the neural network being configured to use machine learning predictive decoding to generate a network-predicted data sample, the network-predicted data sample corresponding to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample is located between the first data sample and the second data sample.

[0156] Example 2

[0157] A device according to Example 1, wherein the one or more processors are operably configured to: provide the network predicted data sample and the first reconstructed data sample as input to the neural network, the neural network is configured to use the machine learning predictive decoding to generate another network predicted data sample, the other network predicted data sample corresponds to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

[0158] Example 3

[0159] The apparatus of Example 2, wherein the one or more processors are operably configured to provide a position input to the neural network, wherein the position input indicates a relative position of the another particular data sample to the first data sample and the particular data sample.

[0160] Example 4

[0161] A device according to any one of Examples 1 to 3, wherein the one or more processors are operably configured to: initiate transmission of data representing the first data sample as part of a first packet to a receiving device, wherein zero bits of the first packet are dedicated to the particular data sample; and initiate transmission of data representing the second data sample as part of a second packet to the receiving device.

[0162] Example 5

[0163] A device according to any one of Examples 1 to 4, wherein the one or more processors are operably configured to: determine a residual vector associated with a data sample predicted by the network; quantize the residual vector using a codebook to generate a residual code; and initiate transmission of the residual code to a receiving device.

[0164] Example 6

[0165] The apparatus of example 5, wherein the residual vector is based on a comparison of the particular data sample and the data sample predicted by the network.

[0166] Example 7

[0167] An apparatus according to any of Examples 5 to 6, wherein a residual vector is determined and quantized in response to determining that a network condition fails to meet a threshold.

[0168] Example 8

[0169] A device according to any one of Examples 1 to 7, wherein the one or more processors are operably configured to: receive a first packet from a sending device, the first packet including data representing the first data sample; and receive a second packet from the sending device, the second packet including data representing the second data sample.

[0170] Example 9

[0171] According to the device of Example 8, the one or more processors are operably configured to: receive a residual code from the transmitting device; and modify the network predicted data samples based on the residual code.

[0172] Example 10

[0173] A method comprising: generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream; generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of the data samples; and providing the first reconstructed data sample and the second reconstructed data sample as input to a neural network, the neural network being configured to use machine learning predictive decoding to generate a network-predicted data sample, the network-predicted data sample corresponding to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample being located between the first data sample and the second data sample.

[0174] Example 11

[0175] The method according to Example 10 also includes: providing the network predicted data sample and the first reconstructed data sample as input to the neural network, the neural network being configured to use the machine learning predictive decoding to generate another network predicted data sample, the other network predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

[0176] Example 12

[0177] The method of Example 11 further includes providing a position input to the neural network, wherein the position input indicates a temporal position of the another specific data sample.

[0178] Example 13

[0179] The method according to any one of Examples 11 to 12 also includes providing an auxiliary vector to the neural network, wherein the auxiliary vector indicates a transition characteristic between a data sample predicted by the network and the first reconstructed data sample.

[0180] Example 14

[0181] The method according to any one of Examples 10 to 13 also includes: sending data representing the first data sample as part of a first packet to a receiving device, wherein the zero bit of the first packet is dedicated to the specific data sample; and sending data representing the second data sample as part of a second packet to the receiving device.

[0182] Example 15

[0183] The method according to any one of Examples 10 to 14 also includes: determining a residual vector associated with the data sample predicted by the network; quantizing the residual vector using a codebook to generate a residual code; and sending the residual code to a receiving device.

[0184] Example 16

[0185] The method of Example 15, wherein the residual vector is based on a comparison of the particular data sample with a data sample predicted by the network.

[0186] Example 17

[0187] A method according to any one of Examples 15 to 16, wherein the residual vector is determined and quantized in response to determining that a network condition does not meet a threshold.

[0188] Example 18

[0189] The method according to any one of Examples 10 to 17 also includes: receiving data representing the first data sample from a sending device; and receiving data representing the second data sample from the sending device.

[0190] Example 19

[0191] The method according to any one of Examples 10 to 18 also includes: receiving a residual code from a transmitting device; and modifying the network predicted data samples based on the residual code.

[0192] Example 20

[0193] A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: generate a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream; generate a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples; and provide the first reconstructed data sample and the second reconstructed data sample as input to a neural network configured to use machine learning predictive decoding to generate a network-predicted data sample, the network-predicted data sample corresponding to a predicted version of a specific data sample in the time series of data samples, and the specific data sample is located between the first data sample and the second data sample.

[0194] Example 21

[0195] According to the non-transitory computer-readable medium of Example 20, the instructions, when executed, also cause the one or more processors to: provide the network-predicted data sample and the first reconstructed data sample as input to the neural network, the neural network being configured to use the machine learning predictive decoding to generate another network-predicted data sample, the other network-predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

[0196] Example 22

[0197] The non-transitory computer-readable medium of Example 21, wherein the instructions, when executed, further cause the one or more processors to provide a position input to the neural network, wherein the position input indicates a relative position of the other particular data sample to the first data sample and the particular data sample.

[0198] Example 23

[0199] A non-transitory computer-readable medium according to any one of Examples 20 to 22, wherein the instructions, when executed, further cause the one or more processors to: generate a first latent vector based on the first data sample; and generate a second latent vector based on the second data sample.

[0200] Example 24

[0201] According to a non-transitory computer-readable medium described in any of Examples 20 to 23, wherein the instructions, when executed, also cause the one or more processors to: initiate transmission of data representing the first data sample as part of a first group to a receiving device, wherein zero bits of the first group are dedicated to the particular data sample; and initiate transmission of data representing the second data sample as part of a second group to the receiving device.

[0202] Example 25

[0203] A non-transitory computer-readable medium according to any one of Examples 20 to 24, wherein the instructions, when executed, further cause the one or more processors to: determine a residual vector associated with the network predicted data sample; quantize the residual vector using a codebook to generate a residual code; and initiate transmission of the residual code to a receiving device.

[0204] Example 26

[0205] The non-transitory computer readable medium of example 25, wherein the residual vector is based on a comparison of the particular data sample to a data sample predicted by the network.

[0206] Example 27

[0207] The non-transitory computer readable medium of any one of Examples 25 to 26, wherein the residual vector is determined and quantized in response to determining that a network condition does not satisfy a threshold.

[0208] Example 28

[0209] According to the non-transitory computer-readable medium described in any of Examples 20 to 27, wherein the instructions, when executed, also cause the one or more processors to: receive a first latent code from a sending device, the first latent code including data representing the first data sample; and receive a second latent code from the sending device, the second latent code including data representing the second data sample.

[0210] Example 29

[0211] The non-transitory computer-readable medium of any one of Examples 20 to 28, wherein the instructions, when executed, further cause the one or more processors to: receive a residual code from the transmitting device; and modify network predicted data samples based on the residual code.

[0212] Example 30

[0213] An apparatus comprising: a component for generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream; a component for generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of the data samples; and a component for providing the first reconstructed data sample and the second reconstructed data sample as input to a neural network, the neural network being configured to use machine learning predictive decoding to generate a network-predicted data sample, the network-predicted data sample corresponding to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample being located between the first data sample and the second data sample.

[0214] Example 31

[0215] The apparatus according to Example 30 further includes: a component for providing the network-predicted data sample and the first reconstructed data sample as input to the neural network, the neural network being configured to use the machine learning predictive decoding to generate another network-predicted data sample, the another network-predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the another specific data sample is located between the first data sample and the specific data sample.

[0216] Example 32

[0217] The apparatus according to Example 31 further includes a component for providing a position input to the neural network, wherein the position input indicates a relative position of the other specific data sample to the first data sample and the specific data sample.

[0218] Example 33

[0219] The apparatus according to any one of Examples 30 to 32 further includes: a component for generating a first latent vector based on the first data sample; and a component for generating a second latent vector based on the second data sample.

[0220] Example 34

[0221] The apparatus of any of Examples 30 to 33 further includes: a component for sending data representing a first data sample as part of a first packet to a receiving device, wherein zero bits of the first packet are dedicated to a specific data sample; and a component for sending data representing a second data sample as part of a second packet to the receiving device.

[0222] Example 35

[0223] The apparatus according to any one of Examples 30 to 34 further includes: a component for determining a residual vector associated with the data sample predicted by the network; a component for quantizing the residual vector using a codebook to generate a residual code; and a component for sending the residual code to a receiving device.

[0224] Example 36

[0225] The apparatus of Example 35, wherein the residual vector is based on a comparison of a particular data sample and a data sample predicted by the network.

[0226] Example 37

[0227] An apparatus according to any of Examples 35 to 36, wherein a residual vector is determined and quantized in response to determining that a network condition fails to meet a threshold.

[0228] Example 38

[0229] The apparatus of any one of Examples 30 to 37 further includes: a component for receiving a first latent code from a sending device, the first latent code including data representing the first data sample; and a component for receiving a second latent code from the sending device, the second latent code including data representing the second data sample.

[0230] Example 39

[0231] The apparatus according to any one of Examples 30 to 38 further includes: a component for receiving a residual code from a transmitting device; and a component for modifying a network predicted data sample based on the residual code.

[0232] Those skilled in the art will further appreciate that the various illustrative logic blocks, configurations, modules, circuits, and algorithm steps described in conjunction with the embodiments disclosed herein may be implemented as electronic hardware, computer software executed by a processor, or a combination of the two. Various illustrative components, blocks, configurations, modules, circuits, and steps have been generally described above with respect to their functionality. Whether such functionality is implemented as hardware or processor executable instructions depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art may implement the described functionality in different ways for each specific application, and such implementation decisions should not be interpreted as causing a departure from the scope of the present invention.

[0233] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be directly embodied in hardware, in a software module executed by a processor, or in a combination of the two. The software module may reside in a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a register, a hard disk, a removable disk, a compact disk read-only memory (CD-ROM), or any other form of non-transient storage medium known in the art. An exemplary storage medium is coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium. In an alternative, the storage medium may be integrated into the processor. The processor and the storage medium may reside in an application specific integrated circuit (ASIC). The ASIC may reside in a computing device or a user terminal. In an alternative, the processor and the storage medium may reside in a computing device or a user terminal as discrete components. The previous description of the disclosed embodiments is provided to enable a person skilled in the art to perform or use the disclosed embodiments. Various modifications to these embodiments will be apparent to those skilled in the art, and the principles defined herein may be applied to other embodiments without departing from the scope of the present disclosure. Therefore, the present invention is not intended to be limited to the embodiments shown herein, and should be consistent with the broadest possible scope consistent with the principles and novel features as defined by the appended claims.

Claims

1. A device comprising: Memory; as well as one or more processors coupled to the memory and operably configured to: generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in the time series of data samples of a portion of a data stream; generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples; as well as The first reconstructed data sample and the second reconstructed data sample are provided as input to a neural network, and the neural network is configured to use machine learning predictive decoding to generate a network-predicted data sample, wherein the network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample is located between the first data sample and the second data sample.

2. The device according to claim 1, wherein The one or more processors may be operably configured to: The network-predicted data sample and the first reconstructed data sample are provided as inputs to the neural network, and the neural network is configured to use the machine learning predictive decoding to generate another network-predicted data sample, the other network-predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

3. The device according to claim 2, wherein: The one or more processors may be operably configured to provide a temporal position input to the neural network, wherein the temporal position input indicates a temporal position of the another particular data sample relative to the first data sample and the particular data sample.

4. The device according to claim 1, wherein: The one or more processors may be operably configured to: generating a first packet based on the first reconstructed data samples; and A second packet is generated based on the second reconstructed data samples.

5. The apparatus of claim 1 , wherein the one or more processors are operably configured to: initiating transmission of data representing the first data sample to a receiving device as part of a first packet, wherein zero bits of the first packet are dedicated to the particular data sample; and Transmission of data representing the second data sample as part of a second packet to the receiving device is initiated.

6. The device according to claim 1, wherein: The one or more processors may be operably configured to: determining a residual vector associated with a data sample predicted by the network; quantizing the residual vector using a codebook to generate a residual code; as well as Initiate transmission of the residual code to a receiving device.

7. The device according to claim 6, wherein: The residual vector is based on a comparison of the particular data sample with a data sample predicted by the network.

8. The device according to claim 6, wherein: The residual vector is determined and quantized in response to determining that a network condition does not satisfy a threshold.

9. The device according to claim 1, wherein: The one or more processors may be operably configured to: receiving a first packet from a sending device, the first packet comprising data representing the first data sample; and A second packet is received from the sending device, the second packet including data representing the second data sample.

10. The apparatus of claim 9, wherein the one or more processors are operably configured to: receiving a residual code from the transmitting device; and The network predicted data samples are modified based on the residual code.

11. The device according to claim 1, wherein: The first data sample is represented by a first latent vector of a Feedback Recurrent Autoencoder (FRAE), and wherein the second data sample is represented by a second latent vector of the FRAE.

12. A method comprising: generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in the time series of data samples of a portion of a data stream; generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples; as well as The first reconstructed data sample and the second reconstructed data sample are provided as input to a neural network, and the neural network is configured to use machine learning predictive decoding to generate a network-predicted data sample, wherein the network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample is located between the first data sample and the second data sample.

13. The method according to claim 12, further comprising: The network-predicted data sample and the first reconstructed data sample are provided as inputs to the neural network, and the neural network is configured to use the machine learning predictive decoding to generate another network-predicted data sample, the other network-predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

14. The method of claim 13, further comprising providing a temporal position input to the neural network, wherein: The time position input indicates a time position of the another specific data sample relative to the first data sample and the specific data sample.

15. The method according to claim 12, further comprising: generating a first group based on the first reconstructed data samples; as well as A second packet is generated based on the second reconstructed data samples.

16. The method according to claim 12, further comprising: transmitting data representing the first data sample to a receiving device as part of a first packet, wherein zero bits of the first packet are dedicated to the particular data sample; as well as Data representing the second data sample is sent to the receiving device as part of a second packet.

17. The method according to claim 12, further comprising: determining a residual vector associated with a data sample predicted by the network; quantizing the residual vector using a codebook to generate a residual code; as well as The residual code is sent to a receiving device.

18. The method according to claim 17, wherein: The residual vector is based on a comparison of the particular data sample with a data sample predicted by the network.

19. The method according to claim 17, wherein: The residual vector is determined and quantized in response to determining that a network condition does not satisfy a threshold.

20. The method of claim 12, further comprising: receiving a first packet from a sending device, the first packet including data representing the first data sample; as well as A second packet is received from the sending device, the second packet including data representing the second data sample.

21. The method according to claim 20, further comprising: receiving a residual code from the transmitting device; as well as The network predicted data samples are modified based on the residual code.

22. A non-transitory computer readable medium comprising instructions that, when executed by one or more processors, cause the one or more processors to: generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in the time series of data samples of a portion of a data stream; generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in the time series of data samples; and The first reconstructed data sample and the second reconstructed data sample are provided as input to a neural network, and the neural network is configured to use machine learning predictive decoding to generate a network-predicted data sample, wherein the network-predicted data sample corresponds to a predicted version of a specific data sample in the time series of the data samples, and the specific data sample is located between the first data sample and the second data sample.

23. The non-transitory computer readable medium of claim 22, wherein the instructions, when executed, further cause the one or more processors to: The network-predicted data sample and the first reconstructed data sample are provided as inputs to the neural network, and the neural network is configured to use the machine learning predictive decoding to generate another network-predicted data sample, the other network-predicted data sample corresponding to a predicted version of another specific data sample in the time series of the data samples, and the other specific data sample is located between the first data sample and the specific data sample.

24. The non-transitory computer-readable medium of claim 23, wherein the instructions, when executed, further cause the one or more processors to provide a temporal position input to the neural network, wherein the temporal position input indicates a temporal position of the another particular data sample relative to the first data sample and the particular data sample.

25. The non-transitory computer readable medium of claim 22, wherein the instructions, when executed, further cause the one or more processors to: generating a first packet based on the first reconstructed data samples; and A second packet is generated based on the second reconstructed data samples.

26. The non-transitory computer readable medium of claim 22, wherein the instructions, when executed, further cause the one or more processors to: initiating transmission of data representing the first data sample to a receiving device as part of a first packet, wherein zero bits of the first packet are dedicated to the particular data sample; and Transmission of data representing the second data sample as part of a second packet to the receiving device is initiated.

27. The non-transitory computer readable medium of claim 22, wherein the instructions, when executed, further cause the one or more processors to: determining a residual vector associated with a data sample predicted by the network; quantizing the residual vector using a codebook to generate a residual code; and Initiate transmission of the residual code to a receiving device.

28. The non-transitory computer-readable medium of claim 27, wherein the residual vector is based on a comparison of the particular data sample to a data sample predicted by the network.

29. An apparatus comprising: means for generating a first reconstructed data sample corresponding to a reconstructed version of a first data sample in a time series of data samples of a portion of a data stream; means for generating a second reconstructed data sample corresponding to a reconstructed version of a second data sample in said time series of data samples; as well as A component for providing the first reconstructed data sample and the second reconstructed data sample as input to a neural network, the neural network being configured to use machine learning predictive coding to generate a network-predicted data sample, the network-predicted data sample corresponding to a predicted version of a particular data sample in the time series of data samples, and the particular data sample being located between the first data sample and the second data sample.

30. The apparatus of claim 29, further comprising: means for determining a residual vector associated with a data sample predicted by the network; A means for quantizing the residual vector using a codebook to generate a residual code; as well as A means for transmitting the residual code to a receiving device.

Images