Method for conditioning an intraframe codec on itself in an end-to-end learned codec

Description

1 / 72 METHOD FOR CONDITIONING AN INTRAFRAME CODEC INTO ITSELF INTO AN END-TO-END LEARNED CODEC TECHNOLOGICAL FIELD

[0001] An exemplary embodiment refers to video and image encoder-decoders (codecs) and, more particularly, to end-to-end learned encoder-decoders. BACKGROUND

[0002] Neural networks (NNs) are used in an increasing number of applications for a variety of devices, such as mobile phones. Neural networks are used for image and video analysis and processing, social media data analysis, device usage analysis, and other applications. Neural networks are computer graphs consisting of multiple layers of computation. Each layer may consist of one or more units that perform elementary computations. A unit is connected to one or more other units, and the connection is often associated with a weight used to scale a signal passing through the connection. The weight is a value that can be learned from training data. In addition, batch normalization layer parameters can be learned from training data.

[0003] Feed-forward neural networks do not have feedback loops, with each layer receiving input from the previous layer and providing its output to the next layer. Initial layers close to the input data semantically extract low-level features, such as edges and textures in images, while intermediate and final layers extract higher-level features. Furthermore, one or more layers may perform tasks such as classification, semantic segmentation, object detection, noise suppression, style transfer, superresolution, and / or similar tasks. Recurrent neural networks have a feedback loop, making the neural network stateful. Recurrent neural networks are capable of memorizing information or a state.

[0004] Neural networks are capable of learning properties of input data in a supervised or unsupervised manner. This capability is the result of a training algorithm or a meta-level neural network that provides a training signal. A training algorithm alters properties of a neural network. Petition 870250095061, dated 10 / 17 / 2025, page 131 / 543 2 / 72 so that an output is as close as possible to a desired output. For example, for a neural network that classifies objects in images, the neural network output can be used to derive a category or class index for the objects. Training minimizes or reduces output error, referred to as loss. Examples of losses include mean squared error and cross-entropy. Training can be an iterative process where, in each iteration, an algorithm modifies the neural network weights to gradually decrease the network's loss. The ultimate goal of training the neural network is to make the neural network learn the properties of the data distribution from a limited training dataset, or to “generalize” data that was not used to train the neural network. A validation dataset that was not used for training is often used to check the neural network's performance.

[0005] When checking the performance of a neural network, the training set error should decrease. Otherwise, the neural network is in the underfitting regime. If the neural network is working to generalize, the validation set error should decrease and should not be much larger than the training set error. Otherwise, the neural network is in the overfitting regime (that is, the neural network has memorized the properties of the training set and works well only on that set).

[0006] Neural network-based codecs can be used to compress and decompress data such as images in the case of image codecs. Neural network-based image codecs may comprise a component called an autoencoder, which comprises a neural encoder and a neural decoder. A neural encoder receives an image as input and produces a latent tensor that may require fewer bits than the input image. The latent tensor can be quantized and losslessly compressed to obtain a bitstream representing the encoded image. A neural decoder receives the bitstream and reconstructs the image that was fed into the neural encoder.

[0007] Neural encoders and decoders are trained to minimize a combination of bit rate and distortion. Distortion can be based on Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), or similar. These distortion metrics are correlated with the quality of human visual perception, so improving them improves the visual quality of a decoded image as perceived by humans.

[0008] A video codec includes a neural encoder that transforms a video into Petition 870250095061, dated 10 / 17 / 2025, page 132 / 543 3 / 72 input in a compressed representation suitable for storage / transmission and a neural decoder that decompresses the compressed video representation back into a viewable form. A neural encoder typically discards some information in the original video to represent the video in a more compact form (at a lower bitrate). A hybrid video codec can encode video information in two phases. First, pixel values ​​for a certain block are predicted. For example, pixel values ​​can be predicted by means of motion compensation (i.e., finding and indicating an area in one of the previously recorded video frames that closely corresponds to the block being encoded) or by spatial means (i.e., using the pixel values ​​around the block to be encoded in a specified manner).Secondly, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. For example, this can be done by transforming the difference into pixel values ​​using a specified transform (e.g., Discrete Cosine Transform), quantizing the coefficients, and entropy-encoding the quantized coefficients. By varying the fidelity of the quantization process, the neural encoder controls a balance between the precision of the pixel presentation (pixel quality) and the size of the resulting encoded video representation (file size or transmission bitrate).

[0009] Interprediction (also known as temporal prediction, motion compensation, or motion-compensated prediction) exploits temporal redundancy. Interprediction uses previously decoded images as the prediction sources. In contrast, intraprediction uses adjacent pixels within the same two images. Intraprediction can be performed in the spatial domain to predict sample values ​​or the transform domain to predict transform coefficients. Intraprediction is often exploited in intracoding, where no prediction is applied.

[0010] An output of the encoding procedure is a set of encoding parameters, such as motion vectors and quantized transform coefficients. Parameters can be entropy-encoded more efficiently if they are predicted first from spatially or temporally neighboring parameters.For example, a motion vector can be predicted from spatially adjacent motion vectors, and only the difference relative to the motion vector predictor can be encoded. Predictions of encoding parameters and intraprediction are possible. Petition 870250095061, dated 10 / 17 / 2025, page 133 / 543 4 / 72 collectively referred to as image prediction.

[0011] A decoder reconstructs a video input for the encoder by applying prediction methods similar to those used by the encoder. In doing so, the decoder forms a predicted representation of the pixel blocks using the motion or spatial information created by the encoder and stored in the compressed representation. The decoder performs prediction error decoding (the inverse operation of prediction error encoding that recovers the quantized prediction error signal in a spatial pixel domain). After applying prediction and prediction error decoding, the decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder and encoder may also apply additional filtering to improve the quality of the output video before passing it to display or storing it as a prediction reference for subsequent frames in the video sequence.

[0012] In typical video encoders / decoders, motion information is indicated with motion vectors associated with each motion-compensated image block. Each motion vector represents the displacement of the respective image block in the figure to be encoded or decoded and the prediction source block in one of the previously encoded or decoded figures. To represent motion vectors efficiently, they are typically differentially encoded with respect to specific block-predicted motion vectors. In typical video codecs, the predicted motion vectors are created in a predefined way. For example, they can be created by calculating the median of the encoded or decoded motion vectors of adjacent blocks.Another method of creating motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or colocalized blocks in temporal reference images and flag a chosen candidate as the motion vector predictor. Additionally, the reference index of a previously encoded or decoded image can be predicted. The reference index can be predicted from adjacent blocks and / or colocalized blocks in a temporal reference image. Furthermore, typical high-efficiency video codecs employ an additional motion information encoding / decoding mechanism. This mechanism is often called a merge mode, where all motion field information (including the motion vector and the image index) is combined. Petition 870250095061, dated 10 / 17 / 2025, page 134 / 543 5 / 72 corresponding reference for each available reference image list) are predicted and used without modification or correction. Similarly, motion field information prediction is performed using motion field information from adjacent blocks and / or colocalized blocks in temporal reference images. The motion field information used is selected from a list of motion field candidates with available motion field information from adjacent and / or colocalized blocks.

[0013] In typical video codecs, the prediction residual after motion compensation is first transformed with a transform kernel such as DCT and then encoded. This is done to reduce the correlation within the prediction residual and provide more efficient encoding. Typical video encoders use Lagrangian cost functions to find optimal encoding modes (e.g., the desired Macroblock mode and associated motion vectors). This is a cost function that uses a weighting factor λ to combine image distortion due to lossy encoding methods and the amount of information required to represent pixel values ​​in an image area.The cost can be represented by the equation C = D + λR, where C is the Lagrangian cost to be minimized, D is the image distortion (as Mean Squared Error) with the considered mode and motion vectors, and R is the number of bits needed to represent the data required to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors).

[0014] In some circumstances, supplementary enhancement information (SEI) messages may be used. In some circumstances, SEI NAL units including prefix SEI NAL units and suffix SEI NAL units are used. Prefix SEI NAL units are capable of initiating an image unit and suffix SEI NAL units are capable of terminating an image unit. An SEI unit contains one or more SEI messages, which are not required for decoding output images, but may assist in related processes such as image output timing, post-processing of decoded images, rendering, error detection, error hiding, and resource reservation.

[0015] Image and video codecs can use a set of filters to enhance the visual quality of the predicted visual content. These filters can be applied in a loop, Petition 870250095061, dated 10 / 17 / 2025, page 135 / 543 6 / 72 out of loop, or both. In the case of loop filters, a filter applied to a block in a currently encoded frame will affect the encoding of another block in the same frame and / or in another frame that was predicted from the currently encoded frame. A loop filter can affect the bitrate and / or visual quality. An enhanced block will cause a smaller residual (the difference between the original block and the predicted and filtered block), therefore requiring fewer bits to be encoded. An out-of-loop filter will be applied to a frame after it has been reconstructed. In this circumstance, the filtered visual content will not be a source for prediction and may only impact the visual quality of frames that are emitted by the decoder.

[0016] Embodiments in the present document disclose enhanced video and image codecs, improving rate-distortion performance of video and image compression using an end-to-end learned codec. Embodiments of the present disclosure provide lower bitrate and higher luma PSNR for a resulting reconstructed image than previous embodies. Embodiments of the present disclosure achieve better rate-distortion performance in a reconstructed image. BRIEF SUMMARY

[0017] A method, apparatus, and computer program product are revealed for an end-to-end learned intraframe codec. By using end-to-end learning with multiple codecs, bitrate is reduced and PSNR is improved.

[0018] In an exemplary embodiment, a method is provided comprising receiving, by a first codec, fundamental truth data. The method further comprises, based on the fundamental truth data, generating a first bitstream. The method further comprises, based on the first bitstream, generating initial reconstruction data, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data. The method further comprises emitting, by the first codec, the initial reconstruction data. The method further comprises determining residual fundamental truth data based at least on the initial reconstruction data and the fundamental truth data. The method further comprises determining auxiliary data based at least on the first bitstream or based at least on data derived from the first bitstream.The method further comprises receiving, via a second codec, the residual fundamental truth data and the auxiliary data. Petition 870250095061, dated 10 / 17 / 2025, page 136 / 543 The 7 / 72 method further comprises, based on the residual fundamental truth data and auxiliary data, generating a second bit stream.

[0019] The method of an exemplary embodiment further comprises, based on the second bitstream, generating residual reconstruction data, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data. The method further comprises outputting, via the second codec, the residual reconstruction data.

[0020] The method of an exemplary embodiment further comprises determining combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0021] The method of an exemplary embodiment further comprises determining a combined bit stream based on at least the first bit stream and the second bit stream.

[0022] In an exemplary embodiment, residual fundamental truth data comprise a difference between the fundamental truth data and the initial reconstruction data.

[0023] In an exemplary embodiment, the fundamental truth data comprise an image comprising brightness data and color data.

[0024] The method of an exemplary embodiment further comprises converting, using a first neural encoder, the fundamental truth data into a first latent tensor. The method further comprises generating, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element. The method further comprises, for a respective symbol or element of the at least one symbol or element of the first quantized latent tensor, determining, using a first probability model, an estimated first probability distribution of possible values.The method further comprises encoding, using a first entropy encoder, the respective symbol or element of at least one symbol or element of the first quantized latent tensor in the first bit stream based on at least the first estimated probability distribution of possible values.

[0025] The method of an exemplary modality additionally comprises Petition 870250095061, dated 10 / 17 / 2025, page 137 / 543 8 / 72 Decode, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor. The method further comprises generating, using a first dequantizer, a first resulting latent tensor. The method further comprises converting, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

[0026] The method of an exemplary embodiment further comprises converting, using a second neural encoder, the residual fundamental truth data into a second latent tensor. The method further comprises generating, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based at least on the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element. The method further comprises converting, using an auxiliary encoder, the auxiliary data into auxiliary features. The method further comprises inserting the auxiliary features into a second probability model.The method further comprises, for a given symbol or element of at least one symbol or element of the second quantized latent tensor, determining, using the second probability model, a second estimated probability distribution of possible values. The method further comprises encoding, using a second entropy encoder, the given symbol or element of at least one symbol or element of the second quantized latent tensor in the second bitstream based on at least the second estimated probability distribution of possible values.

[0027] The method of an exemplary embodiment further comprises decoding, using a second entropy decoder, the auxiliary encoder or another auxiliary encoder that is the same as the auxiliary encoder, and the second probability model or another probability model that is the same as the second probability model, the second bitstream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor. The method further comprises generating, using a second dequantizer, a second resulting latent tensor. The method further comprises converting, using a second neural decoder, the second resulting latent tensor into data. Petition 870250095061, dated 10 / 17 / 2025, page 138 / 543 9 / 72 of residual reconstruction.

[0028] In an exemplary embodiment, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises neural network components.

[0029] In an exemplary embodiment, the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss.

[0030] In an exemplary embodiment, the first codec is trained before the second codec, and the second codec is trained based at least on the first codec or based at least on data that is generated by the first codec.

[0031] In an exemplary embodiment, the first codec and the second codec are trained simultaneously.

[0032] In an exemplary embodiment, the first codec and the second codec are trained at alternating intervals.

[0033] In an exemplary embodiment, the auxiliary data comprise one or more of the initial reconstruction data, the first latent tensor and the first resulting latent tensor.

[0034] In an exemplary embodiment, the residual fundamental truth data are determined by the first codec.

[0035] In an exemplary embodiment, the combined reconstruction data is determined by the second codec.

[0036] The method of an exemplary embodiment further comprises determining additional residual fundamental truth data, based at least on the combined reconstruction data and the fundamental truth data. The method further comprises determining additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream. The method further comprises receiving, by a different codec, the additional residual fundamental truth data and the additional auxiliary data. The method further comprises, based at least on the additional residual fundamental truth data and the additional auxiliary data, generating an additional bitstream. The method further comprises, based on the additional bitstream, generating additional reconstruction data, wherein the reconstruction data Petition 870250095061, dated 10 / 17 / 2025, page 139 / 543 10 / 72 additional data comprises a reconstruction of the additional residual fundamental truth data, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. The method further comprises outputting, via a different codec, at least one of the additional reconstruction data and the composite reconstruction data.

[0037] In an exemplary embodiment, a method is provided comprising receiving a first bit stream. The method further comprises, based on the first bit stream, generating initial reconstruction data. The method further comprises determining auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream. The method further comprises outputting the initial reconstruction data. The method further comprises receiving a second bit stream. The method further comprises, based on the second bit stream and the auxiliary data, generating residual reconstruction data. The method further comprises outputting the residual reconstruction data.

[0038] The method of an exemplary embodiment further comprises determining combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0039] In an exemplary embodiment, the first bit stream and the second bit stream are received as parts of a combined bit stream.

[0040] The method of an exemplary embodiment further comprises decoding, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor. The method further comprises generating, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor. The method further comprises converting, using a first neural decoder, the first latent tensor into the initial reconstruction data.

[0041] The method of an exemplary embodiment further comprises decoding, using a second entropy decoder, an auxiliary encoder and a second probability model, the second bit stream into a second quantized latent tensor. The method further comprises generating, using a second dequantizer and based on the second quantized latent tensor, a second tensor Petition 870250095061, dated 10 / 17 / 2025, page 140 / 543 11 / 72 latent. The method further comprises converting, using a second neural decoder, the second latent tensor in the residual reconstruction data.

[0042] In an exemplary embodiment, at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises neural network components.

[0043] The method of an exemplary embodiment further comprises receiving an additional bit stream. The method further comprises, based on the additional bit stream, generating additional reconstruction data, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. The method further comprises emitting at least one of the additional reconstruction data and the composite reconstruction data.

[0044] In an exemplary embodiment, an apparatus is provided comprising at least one processor and at least one memory that includes computer program code configured to, with the at least one processor, cause the apparatus to receive, by a first codec, fundamental truth data. The at least one memory and the computer program code are further configured to, based on the fundamental truth data, generate a first bit stream. The at least one memory and the computer program code are further configured to, based on the first bit stream, generate initial reconstruction data, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data. The at least one memory and the computer program code are further configured to output, by the first codec, the initial reconstruction data.At least one memory and computer program code are additionally configured to determine residual fundamental truth data based at least on the initial reconstruction data and the fundamental truth data. At least one memory and computer program code are additionally configured to determine auxiliary data based at least on the first bitstream or based at least on data derived from the first bitstream. At least one memory and computer program code are additionally configured to receive, via a second codec, the residual fundamental truth data and the auxiliary data. At least one memory and computer program code are additionally configured to... Petition 870250095061, dated 10 / 17 / 2025, page 141 / 543 Based on the residual fundamental truth data and auxiliary data, generate a second bit stream.

[0045] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, make the apparatus, based on the second bit stream, generate residual reconstruction data, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data. At least one memory and computer program code are further configured to output, via the second codec, the residual reconstruction data.

[0046] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to determine combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0047] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the device to determine a combined bit stream based on at least the first bit stream and the second bit stream.

[0048] In an exemplary embodiment, residual fundamental truth data comprise a difference between fundamental truth data and initial reconstruction data.

[0049] In an exemplary embodiment, the fundamental truth data comprise an image comprising brightness data and color data.

[0050] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to convert, using a first neural encoder, the fundamental truth data into a first latent tensor. At least one memory and computer program code are further configured to generate, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based on at least the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element. At least one memory and computer program code are further configured to, for a respective symbol or element of the at least one symbol or element of the first quantized latent tensor, Petition 870250095061, dated 10 / 17 / 2025, page 142 / 543 13 / 72 determine, using a first probability model, a first estimated probability distribution of possible values. At least one memory and computer program code are further configured to encode, using a first entropy encoder, the respective symbol or element of at least one symbol or element of the first quantized latent tensor in the first bit stream based on at least the first estimated probability distribution of possible values.

[0051] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to decode, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor. At least one memory and computer program code are further configured to generate, using a first dequantizer, a first resulting latent tensor. At least one memory and computer program code are further configured to convert, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

[0052] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, make the apparatus convert, using a second neural encoder, the residual fundamental truth data into a second latent tensor. At least one memory and computer program code are further configured to generate, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based on at least the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element. At least one memory and computer program code are further configured to convert, using an auxiliary encoder, the auxiliary data into auxiliary features.At least one memory and computer program code are additionally configured to insert auxiliary resources into a second probability model. At least one memory and computer program code are additionally configured for, for a respective symbol or element of, at least one symbol or element of. Petition 870250095061, dated 10 / 17 / 2025, page 143 / 543 14 / 72 second quantized latent tensor, determine, using the second probability model, a second estimated probability distribution of possible values. At least one memory and computer program code are additionally configured to encode, using a second entropy encoder, the respective symbol or element of at least one symbol or element of the second quantized latent tensor in the second bitstream based on at least the second estimated probability distribution of possible values.

[0053] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to decode, using a second entropy decoder, the auxiliary encoder and the second probability model or a copy of the second probability model, the second bit stream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor. At least one memory and computer program code are further configured to generate, using a second dequantizer, a second resulting latent tensor. At least one memory and computer program code are further configured to convert, using a second neural decoder, the second resulting latent tensor into residual reconstruction data.

[0054] In an exemplary embodiment, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises neural network components.

[0055] In an exemplary embodiment, the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss.

[0056] In an exemplary embodiment, the first codec is trained before the second codec, and the second codec is trained based at least on the first codec or based at least on data that is generated by the first codec.

[0057] In an exemplary embodiment, the first codec and the second codec are trained simultaneously.

[0058] In an exemplary embodiment, the first codec and the second codec are trained at alternating intervals. Petition 870250095061, dated 10 / 17 / 2025, page 144 / 543 15 / 72

[0059] In an exemplary embodiment, the auxiliary data comprise one or more of the initial reconstruction data, the first latent tensor and the first resulting latent tensor.

[0060] In an exemplary embodiment, the residual fundamental truth data are determined by the first codec.

[0061] In an exemplary embodiment, the combined reconstruction data is determined by the second codec.

[0062] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to determine additional residual fundamental truth data, based at least on the combined reconstruction data and the fundamental truth data. At least one memory and computer program code are further configured to determine additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream. At least one memory and computer program code are further configured to receive, by a different codec, the additional residual fundamental truth data and the additional auxiliary data.At least one memory and computer program code are further configured to generate an additional bitstream based on at least the additional residual fundamental truth data and the additional auxiliary data. At least one memory and computer program code are further configured to generate additional reconstruction data based on the additional bitstream, wherein the additional reconstruction data comprises a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. At least one memory and computer program code are further configured to output the additional reconstruction data through a different codec.

[0063] In an exemplary embodiment, an apparatus is provided comprising at least one processor and at least one memory that includes computer program code configured to, with the at least one processor, cause the apparatus to receive a first bit stream. The at least one memory and the computer program code are further configured to, based on the first bit stream of Petition 870250095061, dated 10 / 17 / 2025, page 145 / 543 16 / 72 bits, generate initial reconstruction data. At least one memory and computer program code are additionally configured to determine auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream. At least one memory and computer program code are additionally configured to output the initial reconstruction data. At least one memory and computer program code are additionally configured to receive a second bit stream. At least one memory and computer program code are additionally configured to generate residual reconstruction data based on the second bit stream and auxiliary data. At least one memory and computer program code are additionally configured to output the residual reconstruction data.

[0064] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to determine combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0065] In an exemplary embodiment, the first bit stream and the second bit stream are received as parts of a combined bit stream.

[0066] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to decode, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor. At least one memory and computer program code are further configured to generate, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor. At least one memory and computer program code are further configured to convert, using a first neural decoder, the first latent tensor into the initial reconstruction data.

[0067] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to decode, using a second entropy decoder, an auxiliary encoder, and a second probability model, the second bit stream into a second quantized latent tensor. At least one memory and the code Petition 870250095061, dated 10 / 17 / 2025, page 146 / 543 17 / 72 of the computer program is further configured to generate, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor. At least one memory and computer program code is further configured to convert, using a second neural decoder, the second latent tensor into the residual reconstruction data.

[0068] In an exemplary embodiment, at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises neural network components.

[0069] In an exemplary embodiment, at least one memory and computer program code are further configured to, with at least one processor, cause the device to receive an additional bit stream. At least one memory and computer program code are further configured to, based on the additional bit stream, generate additional reconstruction data, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. At least one memory and computer program code are further configured to output the additional reconstruction data.

[0070] In an exemplary embodiment, a non-transient computer-readable storage medium is provided comprising computer instructions which, when executed by an apparatus, cause the apparatus to receive, by a first codec, fundamental truth data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate a first bit stream based on the fundamental truth data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate initial reconstruction data based on the first bit stream, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data.The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to output, via the first codec, the initial reconstruction data. The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to determine residual fundamental truth data based at least on the initial reconstruction data and the data from... Petition 870250095061, dated 10 / 17 / 2025, page 147 / 543 18 / 72 fundamental truth. The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to determine auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream. The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to receive, by a second codec, the residual fundamental truth data and the auxiliary data. The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to generate a second bit stream based on the residual fundamental truth data and the auxiliary data.

[0071] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to generate residual reconstruction data based on the second bitstream, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to output the residual reconstruction data via the second codec.

[0072] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to determine combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0073] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to determine a combined bitstream based on at least the first bitstream and the second bitstream.

[0074] In an exemplary embodiment, residual fundamental truth data comprise a difference between fundamental truth data and initial reconstruction data.

[0075] In an exemplary embodiment, the fundamental truth data comprise an image comprising brightness data and color data.

[0076] The non-transient, computer-readable storage medium of a Petition 870250095061, dated 10 / 17 / 2025, page 148 / 543 19 / 72 exemplary mode further includes computer instructions configured, upon execution, to convert, using a first neural encoder, the fundamental truth data into a first latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element.The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to determine, for a given symbol or element of at least one symbol or element of the first quantized latent tensor, using a first probability model, a first estimated probability distribution of possible values. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to encode, using a first entropy encoder, the given symbol or element of at least one symbol or element of the first quantized latent tensor in the first bitstream based on at least the first estimated probability distribution of possible values.

[0077] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to decode, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a first dequantizer, a first resulting latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to convert, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

[0078] The non-transient computer-readable storage medium of an exemplary embodiment additionally includes computer instructions. Petition 870250095061, dated 10 / 17 / 2025, page 149 / 543 20 / 72 configured, upon execution, to convert, using a second neural encoder, the residual fundamental truth data into a second latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based at least on the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to convert, using an auxiliary encoder, the auxiliary data into auxiliary features.The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to insert auxiliary features into a second probability model. The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to determine, for a respective symbol or element of at least one symbol or element of the second quantized latent tensor, using the second probability model, a second estimated probability distribution of possible values.The non-transient computer-readable storage medium additionally includes computer instructions configured, upon execution, to encode, using a second entropy encoder, the respective symbol or element of at least one symbol or element of the second quantized latent tensor in the second bit stream based on at least the second estimated probability distribution of possible values.

[0079] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to decode, using a second entropy decoder, the auxiliary encoder and the second probability model or a copy of the second probability model, the second bit stream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a second dequantizer, a second resulting latent tensor. The non-transient computer-readable storage medium further includes Petition 870250095061, dated 10 / 17 / 2025, page 150 / 543 21 / 72 computer instructions configured, upon execution, to convert, using a second neural decoder, the second latent tensor resulting in the residual reconstruction data.

[0080] In an exemplary embodiment, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises neural network components.

[0081] In an exemplary embodiment, the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss.

[0082] In an exemplary embodiment, the first codec is trained before the second codec, and the second codec is trained based at least on the first codec or based at least on data that is generated by the first codec.

[0083] In an exemplary embodiment, the first codec and the second codec are trained simultaneously.

[0084] In an exemplary embodiment, the first codec and the second codec are trained at alternating intervals.

[0085] In an exemplary embodiment, the auxiliary data comprise one or more of the initial reconstruction data, the first latent tensor and the first resulting latent tensor.

[0086] In an exemplary embodiment, the residual fundamental truth data are determined by the first codec.

[0087] In an exemplary embodiment, the combined reconstruction data is determined by the second codec.

[0088] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to determine additional residual fundamental truth data based at least on the combined reconstruction data and the fundamental truth data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to determine additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream. The non-transient computer-readable storage medium further includes Petition 870250095061, dated 10 / 17 / 2025, page 151 / 543 22 / 72 computer instructions configured, upon execution, to receive, by a different codec, the additional residual fundamental truth data and the additional auxiliary data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate an additional bitstream based on at least the additional residual fundamental truth data and the additional auxiliary data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate additional reconstruction data based on the additional bitstream, wherein the additional reconstruction data comprises a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data.The non-transient, computer-readable storage medium additionally includes configured computer instructions, upon execution, to output, via a different codec, the additional reconstruction data.

[0089] In an exemplary embodiment, a non-transient computer-readable storage medium is provided comprising computer instructions which, when executed by an apparatus, cause the apparatus to receive a first bit stream. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate initial reconstruction data based on the first bit stream. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to determine auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to output the initial reconstruction data.Non-transient computer-readable storage media additionally includes computer instructions configured, upon execution, to receive a second bit stream. Non-transient computer-readable storage media additionally includes computer instructions configured, upon execution, to generate residual reconstruction data based on the second bit stream and auxiliary data. Non-transient computer-readable storage media additionally includes computer instructions configured, upon execution... Petition 870250095061, dated 10 / 17 / 2025, page 152 / 543 23 / 72 execution, to issue the residual reconstruction data.

[0090] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to determine combined reconstruction data based on at least the initial reconstruction data and the residual reconstruction data.

[0091] In an exemplary embodiment, the first bit stream and the second bit stream are received as parts of a combined bit stream.

[0092] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to decode, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to convert, using a first neural decoder, the first latent tensor into the initial reconstruction data.

[0093] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to decode, using a second entropy decoder, an auxiliary encoder, and a second probability model, the second bit stream into a second quantized latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to convert, using a second neural decoder, the second latent tensor into residual reconstruction data.

[0094] In an exemplary embodiment, at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises neural network components. Petition 870250095061, dated 10 / 17 / 2025, page 153 / 543 24 / 72

[0095] The non-transient computer-readable storage medium of an exemplary embodiment further includes computer instructions configured, upon execution, to receive an additional bit stream. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to generate additional reconstruction data based on the additional bit stream, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. The non-transient computer-readable storage medium further includes computer instructions configured, upon execution, to output the additional reconstruction data.

[0096] In an exemplary embodiment, an apparatus is provided comprising means for receiving, by a first codec, fundamental truth data. The apparatus further comprises means for generating a first bit stream based on the fundamental truth data. The apparatus further comprises means for generating initial reconstruction data based on the first bit stream, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data. The apparatus further comprises means for outputting the initial reconstruction data by the first codec. The apparatus further comprises means for determining residual fundamental truth data based at least on the initial reconstruction data and the fundamental truth data. The apparatus further comprises means for determining auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream.The apparatus further comprises means for receiving, by a second codec, the residual fundamental truth data and the auxiliary data. The apparatus further comprises means for generating a second bitstream based on the residual fundamental truth data and the auxiliary data.

[0097] An exemplary embodiment of the apparatus further comprises means for generating residual reconstruction data based on the second bitstream, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data. The apparatus further comprises means for outputting the residual reconstruction data via the second codec.

[0098] The apparatus of an exemplary embodiment further comprises means for determining combined reconstruction data based on at least the Petition 870250095061, dated 10 / 17 / 2025, page 154 / 543 25 / 72 initial reconstruction data and residual reconstruction data.

[0099] The apparatus of an exemplary embodiment further comprises means for determining a combined bit stream based on at least the first bit stream and the second bit stream.

[0100] In an exemplary embodiment, residual fundamental truth data comprise a difference between fundamental truth data and initial reconstruction data.

[0101] In an exemplary embodiment, the fundamental truth data comprise an image comprising brightness data and color data.

[0102] The apparatus of an exemplary embodiment further comprises means for converting, using a first neural encoder, the fundamental truth data into a first latent tensor. The apparatus further comprises means for generating, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element. The apparatus further comprises means for, for a respective symbol or element of the at least one symbol or element of the first quantized latent tensor, determining, using a first probability model, a first estimated probability distribution of possible values.The apparatus further comprises means for encoding, using a first entropy encoder, the respective symbol or element of at least one symbol or element of the first quantized latent tensor in the first bit stream based on at least the first estimated probability distribution of possible values.

[0103] The apparatus of an exemplary embodiment further comprises means for decoding, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor. The apparatus further comprises means for generating, using a first dequantizer, a first resulting latent tensor. The apparatus further comprises means for converting, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

[0104] The apparatus of an exemplary embodiment additionally comprises Petition 870250095061, dated 10 / 17 / 2025, page 155 / 543 26 / 72 means to convert, using a second neural encoder, the residual fundamental truth data into a second latent tensor. The apparatus further comprises means to generate, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based at least on the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element. The apparatus further comprises means to convert, using an auxiliary encoder, the auxiliary data into auxiliary features. The apparatus further comprises means to insert the auxiliary features into a second probability model. The apparatus further comprises means to, for a respective symbol or element of at least one symbol or element of the second quantized latent tensor, determine, using the second probability model, a second estimated probability distribution of possible values.The apparatus further comprises means for encoding, using a second entropy encoder, the respective symbol or element of at least one symbol or element of the second quantized latent tensor in the second bit stream based on at least the second estimated probability distribution of possible values.

[0105] An exemplary embodiment of the apparatus further comprises means for decoding, using a second entropy decoder, the auxiliary encoder or another auxiliary encoder that is the same as the auxiliary encoder, and the second probability model or another probability model that is the same as the second probability model, the second bit stream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor. The apparatus further comprises means for generating, using a second dequantizer, a second resulting latent tensor. The apparatus further comprises means for converting, using a second neural decoder, the second resulting latent tensor into residual reconstruction data.

[0106] In an exemplary embodiment, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises neural network components.

[0107] In an exemplary embodiment, the first codec and the second codec are trained in an end-to-end manner, minimizing at least one loss of Petition 870250095061, dated 10 / 17 / 2025, page 156 / 543 27 / 72 distortion and rate loss.

[0108] In an exemplary embodiment, the first codec is trained before the second codec, and the second codec is trained based at least on the first codec or based at least on data that is generated by the first codec.

[0109] In an exemplary embodiment, the first codec and the second codec are trained simultaneously.

[0110] In an exemplary embodiment, the first codec and the second codec are trained at alternating intervals.

[0111] In an exemplary embodiment, the auxiliary data comprise one or more of the initial reconstruction data, the first latent tensor and the first resulting latent tensor.

[0112] In an exemplary embodiment, the residual fundamental truth data are determined by the first codec.

[0113] In an exemplary embodiment, the combined reconstruction data is determined by the second codec.

[0114] An exemplary embodiment of the apparatus further comprises means for determining additional residual fundamental truth data, based at least on the combined reconstruction data and the fundamental truth data. The apparatus further comprises means for determining additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream. The apparatus further comprises means for receiving, by a different codec, the additional residual fundamental truth data and the additional auxiliary data. The apparatus further comprises means for generating an additional bitstream based at least on the additional residual fundamental truth data and the additional auxiliary data.The apparatus further comprises means for generating additional reconstruction data based on the additional bitstream, wherein the additional reconstruction data comprises a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. The apparatus further comprises means for outputting, via the different codec, at least one of the additional reconstruction data and the composite reconstruction data. Petition 870250095061, dated 10 / 17 / 2025, page 157 / 543 28 / 72

[0115] In an exemplary embodiment, an apparatus is provided comprising means for receiving a first bit stream. The apparatus further comprises means for generating initial reconstruction data based on the first bit stream. The apparatus further comprises means for determining auxiliary data based at least on the first bit stream or at least on data derived from the first bit stream. The apparatus further comprises means for outputting the initial reconstruction data. The apparatus further comprises means for receiving a second bit stream. The apparatus further comprises means for generating residual reconstruction data based on the second bit stream and the auxiliary data. The apparatus further comprises means for outputting the residual reconstruction data.

[0116] The apparatus of an exemplary embodiment further comprises means for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0117] In an exemplary embodiment, the first bit stream and the second bit stream are received as parts of a combined bit stream.

[0118] The apparatus of an exemplary embodiment further comprises means for decoding, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor. The apparatus further comprises means for generating, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor. The apparatus further comprises means for converting, using a first neural decoder, the first latent tensor into the initial reconstruction data.

[0119] The apparatus of an exemplary embodiment further comprises means for decoding, using a second entropy decoder, an auxiliary encoder and a second probability model, the second bit stream into a second quantized latent tensor. The apparatus further comprises means for generating, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor. The apparatus further comprises means for converting, using a second neural decoder, the second latent tensor into residual reconstruction data.

[0120] In an exemplary embodiment, at least one of the first decoder Petition 870250095061, dated 10 / 17 / 2025, page 158 / 543 29 / 72 neural, from the first probability model, from the second neural decoder, and from the second probability model comprises neural network components.

[0121] An exemplary embodiment of the apparatus further comprises means for receiving an additional bit stream. The apparatus further comprises means for, based on the additional bit stream, generating additional reconstruction data, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data. The apparatus further comprises means for emitting at least one of the additional reconstruction data and the composite reconstruction data. BRIEF DESCRIPTION OF THE DRAWINGS

[0122] Having described, then, certain exemplary embodiments of the present disclosure in general terms, reference will henceforth be made to the accompanying drawings, which are not necessarily drawn to scale and in which:

[0123] Figure 1 is a block diagram of an exemplary communication system in which exemplary embodiments of the present disclosure can be implemented;

[0124] Figure 2 is a block diagram of a system that includes both a service device and a user device designed to perform functions according to exemplary modalities in this document;

[0125] Figure 3 is an illustration of a codec that uses neural networks as pipeline components in accordance with previous modalities;

[0126] Figure 4A is an illustration of a video encoding pipeline that uses neural networks in accordance with previous modalities;

[0127] Figure 4B is an illustration of a video encoding pipeline that uses neural networks on the encoding and decoding sides in accordance with previous modalities;

[0128] Figure 5 is an illustration of an end-to-end learned video coding system based on a neural network according to previous modalities;

[0129] Figure 6A is an illustration of a VCM pipeline according to previous embodiments;

[0130] Figure 6B is an illustration of a VCM pipeline that uses an end-to-end learned approach based on previous modalities;

[0131] Figure 7 is an illustration of training a state-of-the-art learned system. Petition 870250095061, dated 10 / 17 / 2025, page 159 / 543 30 / 72 at the tip according to previous modalities;

[0132] Figure 8 is an illustration of a dense, exemplary divided attention block according to exemplary modalities of the present disclosure;

[0133] Figure 9 is an illustration of an end-to-end learned intraframe codec in accordance with exemplary modalities of the present disclosure;

[0134] Figure 10 is a block diagram of a system with more than two codecs according to exemplary embodiments of the present disclosure;

[0135] Figure 11 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to generate bit streams based on fundamental truth data;

[0136] Figure 12 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to encode a first bit stream;

[0137] Figure 13 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to generate initial reconstruction data;

[0138] Figure 14 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to encode a second bit stream;

[0139] Figure 15 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to generate residual reconstruction data;

[0140] Figure 16 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to output additional reconstruction data;

[0141] Figure 17 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to output residual reconstruction data;

[0142] Figure 18 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to generate initial reconstruction data;

[0143] Figure 19 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to generate residual reconstruction data; and

[0144] Figure 20 is a flowchart that demonstrates operations performed, such as by the device in Figure 2, to output additional reconstruction data. DETAILED DESCRIPTION

[0145] Some embodiments of the present invention will now be described more fully hereafter in this document with reference to the accompanying drawings, in which some, but not all, embodiments of the disclosure are shown. In fact, various embodiments of the invention can be incorporated in many forms. Petition 870250095061, dated 10 / 17 / 2025, p. 160 / 543 31 / 72 different embodiments and should not be interpreted as limited to the embodiments set forth in this document; rather, these embodiments are provided so that this disclosure satisfies applicable legal requirements. Similar reference numerals refer to similar elements throughout the document. As used in this document, the terms “data,” “content,” “information,” and similar terms may be used interchangeably to refer to data capable of being transmitted, received, and / or stored in accordance with embodiments of the present invention. Thus, the use of any such terms should not be understood to limit the spirit and scope of embodiments of the present disclosure.

[0146] Although the terms first, second, etc. may be used in this document to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first element may be called a second element, and similarly, a second element may be called a first element, without departing from the scope of this disclosure. As used in this document, the term “and / or” includes any and all combinations of one or more of the associated listed items.

[0147] When an element is referred to as being “connected” or “coupled” to another element, it may be directly connected or coupled to the other element, or intervening elements may be present. Conversely, when an element is referred to as being “directly connected” or “directly coupled” to another element, there are no intervening elements present. Other words used to describe the relationship between elements should be interpreted similarly (e.g., “between” versus “directly between”, “adjacent” versus “directly adjacent”, etc.).

[0148] The terminology used in this document is intended to describe particular modalities only and is not intended to be limiting. As used in this document, the singular forms a, an, and the are intended to include plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises”, “that comprises”, “includes” and / or “that includes”, when used in this document, specify the presence of features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, Petition 870250095061, dated 10 / 17 / 2025, p. 161 / 543 32 / 72 operations, elements, components and / or groups thereof.

[0149] It should also be noted that, in some alternative implementations, the observed functions / acts may occur out of the order shown in the figures. For example, two figures shown in succession may, in fact, be executed concurrently or may sometimes be executed in reverse order, depending on the functionality / acts involved.

[0150] Specific details are provided in the following description to give a complete understanding of exemplary modalities. However, it will be understood by one with ordinary skill in the technique that exemplary modalities can be practiced without such specific details. For example, systems may be shown in block diagrams so as not to obscure exemplary modalities with unnecessary details. In other cases, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring exemplary modalities.

[0151] Additionally, as used in this document, the term “circuit assembly” may refer to one or more or all of the following: (a) hardware-only circuit implementations (such as analog circuit assembly and / or digital circuit assembly implementations); (b) combinations of circuits and software, such as (where applicable): (i) a combination of analog and / or digital hardware circuit(s) with software / firmware and (ii) any portions of hardware processor(s) with software (including digital signal processor(s), software and memory(ies) that work together to enable an appliance, such as a server or mobile phone, to perform various functions) and (c) hardware circuit(s) and / or processor(s), such as microprocessor(s) or a portion of microprocessor(s), that require software (e.g., firmware) to operate, but the software may not be present when required for operation.This definition of “circuit assembly” applies to all uses of that term in this application, including in any claims. As a further example, as used in this application, the term circuit assembly also covers an implementation of merely a hardware circuit or processor (or multiple processors) or portions of a hardware circuit or processor and its (or their) attached software and / or firmware. The term circuit assembly also covers, for example, and if applicable to the particular claim element, a baseband integrated circuit or processor integrated circuit for a mobile phone or a... Petition 870250095061, dated 10 / 17 / 2025, page 162 / 543 33 / 72 similar integrated circuit in a server, a cellular network device, or other computing or networking device.

[0152] Additionally, as used in this document, the terms “model”, “neural network”, “neural mesh”, and “network” are used interchangeably. Furthermore, neural network weights may be referred to as learnable parameters or parameters.

[0153] Additionally, as used in this document, the terms “machine” and “task neural network” are used interchangeably, meaning any process or algorithm (learned from data or not) that analyzes or processes data for a certain task.

[0154] Additionally, as used in this document, the terms “receiver side” and “decoder side” refer to the physical or abstract entity or device that may contain one or more machines, and may run that one or more machines on some encoded and possibly decoded video representation that is encoded by another physical or abstract entity or device, the “encoder side device”.

[0155] Additionally, as used in this document, the terms “intraframe”, “frame”, and “image” may be used interchangeably. These terms may refer to at least part of the input data and at least part of the output data of an end-to-end learned intraframe codec (e2e). In one or more embodiments, these terms refer to the image as the data type. However, the proposed embodiments may be extended to other data types such as video, audio, etc.

[0156] As defined in this document, a “computer-readable storage medium”, which refers to a physical storage medium (e.g., volatile or non-volatile memory device), can be distinguished from a “computer-readable transmission medium”, which refers to an electromagnetic signal.

[0157] In the following description, illustrative embodiments will be described in reference to acts and symbolic representations of operations (for example, in the form of flowcharts, flow diagrams, data flow diagrams, structural diagrams, block diagrams, etc.) that can be implemented as program modules or functional processes that include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement abstract data types. Petition 870250095061, dated 10 / 17 / 2025, page 163 / 543 34 / 72 specific and can be implemented using existing hardware in existing network elements. Such existing hardware may include one or more Central Processing Units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays (FPGAs), computers, or similar.

[0158] Although a flowchart may describe operations as a sequential process, many of the operations may be performed in parallel, concurrently, or simultaneously. Furthermore, the order of operations may be rearranged. A process may terminate when its operations are completed, but it may also have additional steps not included in the figure. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to a return from the function to the calling function or to the main function.

[0159] As disclosed in this document, the term “storage medium” or “computer-readable storage medium” may represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage media, optical storage media, flash memory devices, and / or other tangible machine-readable media for storing information. The term “computer-readable medium” may include, but is not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing, or carrying instruction(s) and / or data.

[0160] In addition, exemplary embodiments may be implemented by hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, the program code or code segments to perform the necessary tasks may be stored on a computer-readable medium or machine as a computer-readable storage medium. When implemented in software, a processor or processors will perform the necessary tasks.

[0161] A code segment can represent a procedure, function, subprogram, program, routine, subroutine, module, software package, class or Petition 870250095061, dated 10 / 17 / 2025, page 164 / 543 35 / 72 Any combination of instructions, data structures, or program instructions. A code segment can be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or transmitted via any suitable means, including memory sharing, message transport, token transport, network transmission, etc.

[0162] Exemplary modalities that can be used in conjunction with RANs include: Universal Mobile Telecommunications System (UMTS); Global System for Mobile Communications (GSM); Advanced Telephony Service System (AMPS); Narrowband AMPS System (NAMPS); Total Access Communications System (TACS); Personal Cellular Digital System (PDC); United States Digital Cellular System (USDC); Code Division Multiple Access (CDMA) system described in EIA / TIA IS-95; High Rate Packet Data (HRPD) system; Worldwide Interoperability for Microwave Access (WiMAX); Mobile Ultra Wideband (UMB); and 3GPP LTE Partnership Project (3GPP LTE).

[0163] As described in this document, a method, apparatus and computer program product are provided for video and image compression and reconstruction using end-to-end learned encoders-decoders.

[0164] To perform end-to-end learned compression and reconstruction, an apparatus 10 is provided as shown, for example, in FIG. 1. The apparatus may be incorporated by or be in communication with any of a variety of different types of computing devices including, for example, a video processing system, an image processing system or any other system configured to decompress images captured by an instant compression capture system. As shown in FIG. 1, the apparatus of an exemplary embodiment includes, is associated with, or is otherwise in communication with a processor 12, an associated memory 14 and a communication interface 16.

[0165] Processor 12 (and / or coprocessors or any other set of circuits to assist or otherwise associated with the processor) may communicate with memory device 14 via a bus to pass information between components of the device 10. The memory device may be Petition 870250095061, dated 10 / 17 / 2025, p. 165 / 543 36 / 72 non-transient and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer-readable storage medium) comprising ports configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device such as the processor). The memory device may be configured to store information, data, content, applications, instructions, or the like to enable the device to perform various functions according to an exemplary embodiment of the present disclosure. For example, the memory device could be configured to buffer input data for processing by the processor. Additionally or alternatively, the memory device could be configured to store instructions for execution by the processor.

[0166] The device 10 can, in some embodiments, be incorporated into various computing devices as described above. However, in some embodiments, the device can be incorporated as a chip or set of chips. In other words, the device can comprise one or more physical packages (e.g., chips) including materials, components, and / or wires in a structural assembly (e.g., a baseboard). The structural assembly can provide physical strength, size conservation, and / or electrical interaction limitation for the component circuitry assembly included therein. The device can, therefore, in some cases, be configured to implement an embodiment of the present invention on a single chip or as a single “system on a chip”. Thus, in some cases, a chip or set of chips can constitute means for performing one or more operations to provide the functionalities described in this document.

[0167] The processor 12 can be incorporated in several different ways. For example, the processor can be incorporated as one or more of several hardware processing means, such as a processor, coprocessor, microprocessor, controller, digital signal processor (DSP), processing element with or without an associated DSP, or various other circuit assemblies, including integrated circuits such as, for example, an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), a microcontroller unit (MCU), a hardware accelerator, a computer chip. Petition 870250095061, dated 10 / 17 / 2025, page 166 / 543 37 / 72 specific purpose or similar. Thus, in some embodiments, the processor may include one or more processing cores configured to operate independently. A multi-core processor may enable multiprocessing in a single physical package. Additionally or alternatively, the processor may include one or more processors configured in tandem via the bus to enable independent instruction execution, pipelined execution, and / or multithreading.

[0168] In an exemplary embodiment, the processor 12 may be configured to execute instructions stored in the memory device 14 or otherwise accessible to the processor. Alternatively or additionally, the processor may be configured to execute embedded functionality in code. Thus, whether configured by hardware or software methods, or by a combination thereof, the processor may represent an entity (e.g., physically incorporated in the circuit assembly) capable of performing operations in accordance with an embodiment of the present disclosure while configured accordingly. Thus, for example, when the processor is incorporated as an ASIC, FPGA, or similar, the processor may be hardware specifically configured to conduct the operations described herein.Alternatively, as another example, when the processor is incorporated as an instruction executor, the instructions may specifically configure the processor to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, the processor may be a processor of a specific device (e.g., an image processing system) configured to employ an embodiment of the present invention by further configuring the processor through instructions to perform the algorithms and / or operations described herein. The processor may include, among other things, a clock, an arithmetic logic unit (ALU), and logic gates configured to support the processor's operation.

[0169] The communication interface 16 can be any means such as a device or set of circuits embedded in hardware or a combination of hardware and software that is configured to receive and / or transmit data, such as receiving frames from an instant compression capture system or an external memory device and / or providing the reconstructed signal to an imaging system or other type of display for presentation or to an external memory device. Petition 870250095061, dated 10 / 17 / 2025, page 167 / 543 38 / 72 for storage. In this sense, the communication interface may include, for example, an antenna (or multiple antennas) and supporting hardware and / or software to enable communication with a wireless communication network. Additionally or alternatively, the communication interface may include the circuitry to interact with the antenna(s) to transmit signals via the antenna(s) or to manage the reception of signals received via the antenna(s). In some environments, the communication interface may alternatively or also support wired communication. Thus, for example, the communication interface may include a communication modem and / or other hardware / software to support communication via cable, digital subscriber line (DSL), Universal Serial Bus (USB), or other mechanisms.

[0170] Processor 12 can be configured to execute instructions from a computer program by performing arithmetic, logical, and input / output operations of the system. Instructions can be supplied to processor 12 by memory 14.

[0171] Various interfaces of device 10 may include components that interface with processor 12 with an antenna or other input / output components. As will be understood, the interfaces and programs stored in memory 14 to present the special-purpose functionalities of device 10 will vary depending on the implementation of device 10.

[0172] In an exemplary embodiment, device 10 may be any known or to be developed device including, but not limited to, a mobile phone, a laptop, a tablet, a personal computer, a portable media device such as a television set, a multifunction camera, a drone, an electric vehicle, etc.

[0173] Video and image compression codecs are devices or computer programs that encode and / or decode, for example, a stream of digital data, a bitstream, a sequence of images, a signal, or similar, related to video and / or images. Still image codecs may be compatible with standards such as JPEG, GIF, PNG, or similar. Video codecs may be compatible with standards such as Cinepak, MPEG, MPEG-2, H.264, VP8, H.265, or similar.

[0174] Considering now Figure 2, an exemplary system for implementing the end-to-end codec is provided. In this exemplary embodiment, service device 22 can perform encoding (where encoding can comprise performing Petition 870250095061, dated 10 / 17 / 2025, page 168 / 543 39 / 72 also decoding) of an input image or video. In one or more embodiments, service device 22 can transmit encoded bitstream data resulting from the input image to consumer device 24. In one exemplary embodiment, the bitstream data can be transmitted wirelessly over a network. In one exemplary embodiment, the bitstream data can be transmitted via a wired connection between service device 22 and consumer device 24. In one or more embodiments, consumer device 24 can decode the bitstream data received from service device 22 into one or more reconstructed images or videos. In one or more embodiments, service device 22 can save bitstream data in a memory 14. Memory 14 can be internal or external to service device 22.In one or more embodiments, service device 22 is capable of retrieving bitstream data from memory 14 and reconstructing the input image. In one or more embodiments, service device 22 and / or consumer device 24 may be incorporated by the device 10.

[0175] Considering now Figure 3, an example of a previous embodiment of a codec 300 using neural networks as part of a pipeline is provided. In one or more embodiments, the loop filter 310 may comprise a neural network. In one example, the loop filter 310 comprises one or more neural network-based loop filters and one or more non-neural network-based loop filters. In another example, the loop filter 310 comprises only one or more neural network-based loop filters. In one or more embodiments, intra-frame prediction 320 may comprise one or more neural networks. In one or more embodiments, inter-frame prediction 330 may comprise one or more neural networks. In one or more embodiments, transforms and / or inverse transforms 340 may comprise one or more neural networks. In one or more modalities, lossless coding (e.g., entropy coding) 350 may comprise or use a neural network-based probability model.Other neural networks can be used throughout the pipeline in various modes.

[0176] Considering now Figure 4A, an illustration is provided for an earlier embodiment of a 400 video encoding pipeline with the main components replaced by neural networks. The 400 video encoding pipeline uses an end-to-end learned compression approach with neural networks. Figure Petition 870250095061, dated 10 / 17 / 2025, page 169 / 543 40 / 72 Figure 4B illustrates an exemplary 410 pipeline that uses neural networks on the encoder and decoder sides. In one or more embodiments, the 410 pipeline comprises an analysis network 420, quantization means and an arithmetic encoder in block 430, an arithmetic decoder 440, and a synthesis network 450. In one or more embodiments, the analysis network 420 comprises an encoding neural network, and the synthesis network 450 comprises a decoding neural network. In one or more embodiments, the analysis network 420 and the synthesis network 450 are part of a self-encoding neural architecture. In one or more embodiments, the analysis network 420 is configured to perform a nonlinear transform, and the synthesis network 450 is configured to perform a nonlinear inverse transform.

[0177] In one or more modes, the input data is analyzed by the 420 parsing network, which outputs a new representation of that input data. The new representation may be more compact. This new representation may then be quantized to a discrete number of values ​​in the 430 block. The quantized data is then losslessly encoded, for example, by an arithmetic encoder in the 430 block, thus obtaining a bitstream. On the decoding side, the bitstream is first losslessly decoded, for example, using a 440 arithmetic decoder. The losslessly decoded data is dequantized and then fed into the 450 synthesis network. The output is the reconstructed or decoded data.

[0178] In one or more modes in which lossy compression is performed, the lossy steps may comprise 420 parsing network and 430 block quantization.

[0179] In an exemplary embodiment, to train pipeline 410, a training objective function (also called “training loss”) is typically used, which may comprise one or more terms, or loss terms, or simply losses. In an exemplary embodiment, the training loss comprises a reconstruction loss term and a rate loss term. In an exemplary embodiment, the reconstruction loss prompts the system to decode data that is similar to the input data, according to some similarity metric. Examples of reconstruction losses include mean squared error (MSE), multiscale structural similarity (MS-SSIM), losses derived from the use of a pre-trained network, losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec, or similar. An example of Petition 870250095061, dated 10 / 17 / 2025, page 170 / 543 41 / 72 losses derived from the use of a pre-trained network are error(f1, f2), where f1 and f2 are the features extracted by a pre-trained neural network for the input data and the decoded data, respectively, and error() is a distance or error function, such as L1 norm or L2 norm. An example of losses derived from the use of a neural network that is trained simultaneously with the end-to-end learned codec is adversarial loss, which is the loss provided by a discriminating neural network that is trained adversarially with respect to the codec, following the configurations proposed in the context of Generative Adversarial Networks (GANs) and their variants.

[0180] Rate loss can encourage the system to compact (i.e., reduce the number of bits) the output of the encoding stage, such as the output of the arithmetic encoder. In one or more embodiments, when a lossless entropy-based encoder is used (e.g., an arithmetic encoder), rate loss encourages the output of the 420 parsing network to have low entropy. Examples of rate losses include a differentiable estimate of entropy, a sparsification loss (i.e., a loss that encourages the output of the 420 parsing network or the output of the quantization to have many zeros such as L0 norm, L1 norm, L1 norm divided by L2 norm), a cross-entropy loss applied to the output of a probability model, where the probability model may be a neural network used to estimate the probability of the next symbol being encoded by an arithmetic encoder or similar.

[0181] In one or more modalities, one or more reconstruction losses can be used, and one or more rate losses can be used, as a weighted sum. In one or more modalities, the different loss terms are weighted using different weights, and these weights determine how the final system performs in terms of rate-loss distortion. For example, if more weight is given to reconstruction losses relative to rate losses, the system can learn to compact less but reconstruct more accurately (as measured by a metric that correlates with reconstruction losses). In one or more modalities, these weights are hyperparameters of the training session and can be set manually by the person designing the training session, or automatically, for example, grid search or using additional neural networks.

[0182] In one or more modalities, non-neural networks can be used in end-to-end learned approaches, such as arithmetic codecs.

[0183] Considering now Figure 5, a video encoding system is provided. Petition 870250095061, dated 10 / 17 / 2025, page 171 / 543 42 / 72 learned end-to-end based on neural network 500 according to previous modalities. In one or more modalities, system 500 contains encoder 510, quantizer 520, probability model 530, an entropy codec (arithmetic encoder 540 and arithmetic decoder 550), dequantizer 560, and decoder 570. In one or more modalities, encoder 510 and decoder 570 are two neural networks. In one or more modalities, encoder 510 and decoder 570 primarily comprise neural network components. In one or more modalities, probability model 530 primarily comprises neural network components. In one or more modalities, quantizer 520, dequantizer 560, and the entropy codec are not based on neural network components. In one or more alternative modalities, these components comprise neural network components.

[0184] In one or more embodiments, the 510 encoder receives a video as input and converts the video from its original signal space into a latent representation (also referred to as a latent tensor) that may comprise a more compactable representation of the input. In one or more embodiments, in the case of an input image, the latent representation may be a three-dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represents the “channels” containing information at that specific location.In one or more embodiments where the input image is a 128x128x3 RGB image (with a horizontal size of 128 pixels, a vertical size of 128 pixels, and 3 channels for the Red, Green, and Blue color components), and the 510 encoder subsamples the input tensor by 2 and expands the channel dimension to 32 channels, the latent representation comprises a tensor with dimensions (or “format”) of 64x64x32 (that is, with a horizontal size of 64 elements, a vertical size of 64 elements, and 32 channels). In various embodiments, the order of the different dimensions may differ depending on the convention used. In an exemplary embodiment, for the input image, the channel dimension may be the first dimension. In this example, the format above the input tensor may be represented as 3x128x128, instead of 128x128x3. In an example using an input video, another dimension in the input tensor can be used to represent temporal information.In an exemplary embodiment, the 520 quantizer quantizes the latent representation into discrete values ​​given a predefined set of quantization levels. The output of... Petition 870250095061, dated 10 / 17 / 2025, page 172 / 543 The 520 quantizer can be referred to as a quantized latent tensor. In one or more embodiments, the 530 probability model and the arithmetic codec component work together to perform lossless compression for the quantized latent representation and generate bitstreams to be sent to the decoder side. Given a symbol to be encoded into the bitstream, the 530 probability model estimates the probability distribution of all possible values ​​for that symbol based on a context that is constructed from information available in the current encoding / decoding state, such as data that has already been encoded / decoded. In one or more embodiments, the 540 arithmetic encoder encodes the input symbols into bitstreams using the estimated probability distributions.

[0185] In one or more modes, on the decoder side, opposite operations are performed. In one or more modes, the 550 arithmetic decoder and the 530 probability model first decode symbols from the bitstream to recover the quantized latent representation. In one or more modes, the 560 dequantizer reconstructs the latent representation into continuous values ​​and passes them to the 570 decoder to recover the input video / image. In one or more modes, the 530 probability model is shared between the encoding and decoding systems. In exemplary modes, one copy of the probability model is used on the encoder side, and another exact copy is used on the decoder side.

[0186] In an exemplary embodiment of system 500, the encoder 510, probability model 530, and decoder 570 are based on deep neural networks. In an exemplary embodiment, the system can be trained in an end-to-end manner by minimizing the following rate-distortion loss function: L = D + λ R. In an exemplary embodiment, D is the distortion loss term, R is the rate loss term, and λ is the weight that controls the balance between the two losses. In an exemplary embodiment, the distortion loss term can be the mean squared error (MSE), structure similarity (SSIM), or other metrics that assess the quality of the reconstructed video. In one or more embodiments, multiple distortion losses can be used and integrated into D, as a weighted sum of MSE and SSIM.In one or more embodiments, the rate loss term is the estimated entropy of the quantized latent representation, which indicates the number of bits required to represent the encoded symbols, for example, bits per pixel (bpp). Petition 870250095061, dated 10 / 17 / 2025, page 173 / 543 44 / 72

[0187] In an exemplary system for lossless video / image compression, the system may contain only a 530 probability model, a 540 arithmetic encoder, and a 550 arithmetic decoder. In one or more modes, the system loss function contains only rate loss, since distortion loss is always zero (i.e., no information loss).

[0188] In one or more modalities, decoded data can be analyzed by machines. Examples of such analysis are object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. Illustrative use cases and applications include autonomous cars, video security cameras and public safety, smart sensor networks, smart TV and smart advertising, person re-identification, intelligent traffic monitoring, drones and / or similar. In one or more modalities, different quality metrics and dedicated algorithms can be used to compress and decompress data for machine consumption that differ from those for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for machine consumption is referred to here as Machine Video Coding.

[0189] In one or more modalities, a receiver-side device has multiple “machines” or neural networks (NNs). In one or more modalities, these multiple machines can be used in a certain combination determined by an orchestrating subsystem. In one or more modalities, the multiple machines can be used in succession, based on the output of the previously used machine, and / or in parallel. For example, a video that has been compressed and then decompressed can be analyzed by one machine (NN) to detect pedestrians, by another machine (another NN) to detect cars, and by yet another machine (another NN) to estimate the depth of all pixels in the frames.

[0190] In one or more embodiments, the encoded video data can be stored on a memory device, for example, as a file. In one or more embodiments, the stored file can subsequently be provided to another device. In one or more alternative embodiments, the encoded video data can be streamed continuously from one device to another.

[0191] Considering now Figure 6A, an illustration of the Video Encoding pipeline for 600 Machines is provided according to previous embodiments. In Petition 870250095061, dated 10 / 17 / 2025, page 174 / 543 In one or more modes, a VCM 610 encoder encodes the video input into a bitstream. In one or more modes, a bitrate can be computed from the bitstream to evaluate the bitstream size. In one or more modes, a VCM 620 decoder decodes the bitstream output from the VCM 610 encoder. In one or more modes, the output of the VCM 620 decoder comprises machine-decoded data. In one or more modes, this data can be decoded or reconstructed video. However, in one or more modes, this data may not have the same or similar characteristics to the original video that was output to the VCM 610 encoder. For example, this data may not be easily understandable by a human simply by rendering the data on a screen. In one or more modes, the output of the VCM 620 decoder is then fed into one or more 631-63N task neural networks.In one or more modes, there can be any number of task neural networks. In one or more modes, VCM 600 has the purpose of achieving a low data rate while ensuring that the 631-63N task neural networks perform well in terms of the evaluation metric associated with each task.

[0192] In one or more embodiments, an end-to-end learned approach to perform video coding for machines can be used. In one or more embodiments, the VCM 610 encoder and VCM 620 decoder consist primarily of neural networks. Figure 6B illustrates an example of a pipeline for the end-to-end learned approach according to previous embodiments. In one or more embodiments, the video is fed into a neural network encoder 650. In one or more embodiments, the output of the neural network encoder is fed into a lossless encoder 670, such as an arithmetic encoder, which outputs a bitstream. In one or more embodiments, the lossless codec may comprise a probability model 640, both in the lossless encoder 670 and in the lossless decoder 680, which predict the probability of the next symbol being encoded and decoded.In one or more modes, the 640 probability model can be learned; for example, it can be a neural network. In one or more modes, on the decoder side, the bitstream is fed into a lossless 680 decoder, such as an arithmetic decoder, whose output is fed into a neural network 660 decoder. In one or more modes, the output of the neural network 660 decoder is the machine-decoded data, which can be fed into one or more task NNs. Petition 870250095061, dated 10 / 17 / 2025, page 175 / 543 46 / 72 631-63N.

[0193] Now consider Figure 7, an example of how an end-to-end learned system 700 can be trained according to previous modes. For simplicity, only one task NN 631 is illustrated, but any number of task NNs can be used. In one or more modes, a rate loss can be computed from the output of the likelihood model 640. In one or more modes, rate loss provides an approximation of the bit rate required to encode the input video data. In one or more modes, a task loss can be computed from the output of the task NN 631.

[0194] In one or more modes, rate loss and task loss can then be used to train the neural networks used in the system, such as the 650 neural network encoder, the 640 likelihood model, the 660 neural network decoder. In one or more modes, training can be performed by first computing gradients of each loss with respect to the neural networks that are contributing to or affecting the computation of the loss. In one or more modes, the gradients are then used by an optimization method, such as Adam, to update the trainable parameters of the neural networks.

[0195] In one or more modalities, machine learning tasks may be performed on the decoder side instead of the encoder side. For example, this may be done because the encoder-side device does not have the capabilities (computational, power, memory) to run the neural networks that perform these tasks. As another example, this may be done because some aspects or the performance of the task neural networks have changed or improved by the time the decoder-side device needs the task results (e.g., different or additional semantic classes, better neural network architecture). In one or more modalities, there could be a need for customization, where different clients would run different neural networks to perform these machine learning tasks.

[0196] Considering now Figure 8, a Dense Divided Attention (DSA) 800 block is provided according to an exemplary modality, in which an NN network type is a Resblock 810 comprising NN layers. In one or more modalities, DSA 800 is an attention block that estimates one or more attention maps, and applies the one or more attention maps to one or more data tensors. In Petition 870250095061, dated 10 / 17 / 2025, page 176 / 543 47 / 72 In one or more modalities, an attention map can be a vector, a matrix, or a tensor. In one example, an attention map can have values ​​in the range [0, 1]. In one or more modalities, one or more data tensors can comprise one or more input tensors for the attention block 800, one or more feature maps that are extracted within the attention block 800, and / or one or more feature maps that are extracted outside the attention block 800. In one or more modalities, applying one or more attention maps to one or more data tensors can comprise multiplying one or more attention map values ​​by one or more data tensors, for example, using element-by-element multiplication. In one or more modalities, other operations can also be considered.

[0197] In one or more embodiments, the DSA 800 block comprises extracting features from its input based on at least one or more initial NN layers. In one or more embodiments, the DSA 800 block further comprises splitting the extracted features along the channel geometric axis to obtain two split features. In one or more embodiments, the DSA 800 block further comprises summing the split features. In one or more embodiments, the DSA 800 block further comprises performing a global averaging operation on the summed features. In one or more embodiments, the DSA 800 block further comprises processing the output of the global averaging operation based on at least one or more NN layers. In one or more embodiments, the DSA 800 block further comprises inserting the result of this processing into a Softmax operation.In one or more modalities, the DSA 800 block further comprises dividing the result of the Softmax operation on the geometric channel axis to obtain two attention tensors. In one or more modalities, the DSA 800 block further comprises multiplying the two attention tensors by the two previously determined split features to obtain two attended split features. In one or more modalities, the DSA 800 block further comprises summing the two attended split features. In one or more modalities, the DSA 800 block further comprises concatenating the summed attended split features with features determined based on at least one or more initial NN layers. In one or more modalities, the DSA 800 block further comprises processing the result of the concatenation through at least one or more NN layers. Petition 870250095061, dated 10 / 17 / 2025, page 177 / 543 48 / 72 In one or more modes, the DSA 800 block additionally comprises summing the output of this processing with the input of the DSA 800 block to obtain the output of the DSA 800 block. In one or more modes, the global averaging operation may be a global pooling operation (an average pooling) that calculates the average value for fragments of a resource map. In one or more modes, the global averaging operation may aggregate spatial information from a resource map to a single channel, to help explore the interchannel relationship of resources.

[0198] In one or more embodiments, the DSA 800 block can be used as one of the blocks for the picture encoder and picture decoder in an end-to-end learned codec. In the picture encoder and picture decoder, the DSA blocks can follow one or more convolutional layers or one or more transposition convolutional layers.

[0199] Considering now Figure 9, an end-to-end learned intraframe (e2e) codec 900 is provided to optimize rate-distortion performance. Exemplary embodiments in this document refer to an end-to-end learned intraframe codec. However, at least some embodiments described in this document may also be applied to an end-to-end learned interframe codec used for video compression, or to an end-to-end learned video codec that compresses both intra- and interframes. The intraframe codec 900 comprises a first-stage codec 905 and a second-stage codec 910, where an output of the first-stage codec is an input to the second-stage codec. In one or more embodiments, the e2e intraframe codec may process the input frame independently, without using any information from other frames.In one or more exemplary embodiments, YUV is the format for data input in first-stage codec 905. In embodiments where YUV format is used, in YUV format, “Y” represents brightness, or “luma” value; and “UV” represents color, or “chroma” values. In one example, the input image might be a 4:4:4 YUV color format image, represented as a three-dimensional array (or tensor) with size 256x256x3, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for Y, U, V components, respectively. In another example, the input image might be a 4:2:0 YUV color format image, represented by the combination of a 256x256 matrix for the luma component and a two-dimensional array (or tensor) of size 128x128x2 for the chroma component. Petition 870250095061, dated 10 / 17 / 2025, page 178 / 543 49 / 72 chroma component. However, the proposed modalities can also be extended to other formats such as RGB.

[0200] In one or more embodiments, end-to-end learned intraframe codec 900 comprises components including picture encoders 916 and 934, quantizers, probability models 924 and 942, entropy encoders 920 and 936, entropy decoders 922 and 940, dequantizers, and picture decoders 928 and 938. Entropy encoders 920 and 936 may comprise lossless encoders, for example, arithmetic encoders. In one or more embodiments, entropy decoders 922 and 940 may comprise lossless decoders, for example, arithmetic decoders. Picture encoders 916 and 934, probability models 924 and 942, and picture decoders 928 and 938 may comprise neural network components. In one or more modalities, quantizers, entropy codecs, and dequantizers can comprise neural network components.

[0201] In one or more embodiments, a codec comprises an encoder and a decoder. For example, first-stage codec 905 comprises an encoder 912 and a decoder 914. In one or more embodiments, the encoder 912 may comprise an image encoder 916, the quantizer, the probability model 924, and the entropy encoder 920. In one or more embodiments, the decoder may comprise the entropy decoder 922, the probability model 924, the dequantizer, and the image decoder 928. In the embodiment illustrated, the probability model 924 in the encoder and the probability model 924 in the decoder are the same probability model. In this context, “same” may refer to two instances of the probability model, one probability model being a copy of two other identical probability models, or both being incorporated into the same instance of the probability model.Similarly, this use of “same” can be used to refer to auxiliary encoders and quantized latent tensors that are identical instances, copies, a modality. For simplicity, identical instances of probability models and auxiliary encoders are labeled with the same numbers in Figure 9.

[0202] In one or more modes, the intraframe codec includes two steps to encode the input data: a first-step codec 905 and a second-step codec 910. In one or more modes, the first-step codec comprises a Petition 870250095061, dated 10 / 17 / 2025, p. 179 / 543 A 50 / 72 encoder 912 receives fundamental truth data as input and outputs a first bitstream. Fundamental truth data may be the data to be compressed. However, a codec, such as a first-stage codec, may receive other inputs besides fundamental truth data, such as an indication of the desired quality of the reconstructed data, or an indication of the desired data rate of the encoded data, or an indication of a characteristic of the fundamental truth data, such as its resolution (in the case of image or video data). In one or more embodiments, the first-stage codec comprises a decoder 914 that receives the first bitstream as input and outputs an initial reconstruction of the input data. In one or more embodiments, the second-stage codec 910 encoder 930 receives a residue, where the residue is computed based on the fundamental truth and the first-stage codec reconstruction.In one or more embodiments, the 942 second-stage codec 910 probability model receives the 905 first-stage codec reconstruction as an auxiliary input. In one or more embodiments, the 930 second-stage codec 910 encoder emits a second bit stream. In one or more embodiments, the 932 second-stage codec 910 decoder receives the second bit stream as input and reconstructs the residue. In one or more embodiments, the reconstructed residue of the 910 second-stage codec is added to the 910 first-stage codec reconstruction to obtain the final reconstruction of the fundamental truth. In one or more embodiments, the sum of the bit stream emitted by the 912 first-stage codec 905 encoder (i.e., the first bit stream) and the bit stream emitted by the 930 second-stage codec 910 encoder (i.e., the second bit stream) represents the encoded fundamental truth.

[0203] This mode provides technical advantages over previous encoder-decoder modes. In one or more modes, with auxiliary input to the second-stage codec probability model 910, the second-stage codec is conditioned on the initial reconstruction of the input data, which improves the rate-distortion performance of the second-stage codec 910. Therefore, the rate-distortion performance of the complete codec 900 (the combination of the first-stage codec 905 and the second-stage codec 910) is improved.

[0204] In one or more embodiments, the learned e2e 900 intraframe codec can be used as part of a video codec, wherein the intraframe codec can encode one or more first frames (e.g., intraframes) of a video. Petition 870250095061, dated 10 / 17 / 2025, page 180 / 543 51 / 72 independently of any other frame, and wherein another codec comprising an interframe codec may encode one or more second frames of a video based on at least one or more third frames, wherein the one or more third frames may comprise zero or more of the one or more first frames and / or zero or more of the one or more second frames. However, in one or more modes, the intraframe codec learned from e2e 900 may be used to encode a subset of all frames that are encoded independently of any other frame (i.e., a subset of all intraframes in a video sequence).In one example, the learned e2e 900 intraframe codec is used to encode a first subset of all intraframes in a video, a conventional intraframe codec is used to encode a second subset of all intraframes in the video, and a conventional interframe codec is used to encode the interframes of the video.

[0205] In one or more modes, in encoder 912, an image encoder 916 receives an image as input and converts the image into a latent tensor. In one or more modes, given a predefined set of quantization levels, the quantizer quantizes the latent tensor into discrete values ​​to obtain a quantized latent tensor. In one or more modes, for each symbol or element in the quantized latent tensor, the probability model 924 estimates the probability distribution of all possible values ​​based on a context that is constructed from information available in the current encoding / decoding state. In one or more modes, the arithmetic encoder 920 encodes (in a lossless manner) the input symbols or elements into a bitstream based at least on the estimated probability distributions.In one or more modalities, this series of steps comprises image compression / encoding, and the resulting bitstream represents the compressed or encoded image.

[0206] In one or more embodiments, in decoder 914, the arithmetic decoder 922 and the probability model 924 first decode the bitstream to recover the quantized latent tensor. In one or more embodiments, it may be required that the probability model 924 used in decoder 914 be the same probability model 924 used on the encoder side. In one or more embodiments, the dequantizer then reconstructs the latent tensor into continuous values ​​and passes it to image decoder 928 to obtain a reconstruction of the input image. In Petition 870250095061, dated 10 / 17 / 2025, page 181 / 543 52 / 72 in one or more modes, the process in decoder 914 describes image decompression / decoding and the resulting image represents the reconstructed, or decoded, or decompressed image (these terms may be used synonymously in at least some of the present modes).

[0207] In one or more modes, learned intraframe codec 900 can be trained in an end-to-end manner minimizing D + λR, where D is the distortion loss term, R is the rate loss term, and λ is the weight that controls the balance between the two losses. In one or more modes, an applied optimization process results in a rate-distortion tradeoff, where a balance is found between distortion D and rate loss R. In one or more modes, rate loss R can indicate a bit rate of the encoded image, and distortion can indicate pixel fidelity distortion such as mean squared error (MSE), multiscale structural similarity (MS-SSIM), multiple distortion losses as a weighted sum of MSE and MS-SSIM, or other metrics that assess the quality of the reconstructed image.

[0208] In one or more modalities, the distortion may relate to the performance of one or more machine analysis tasks or to the estimated performance of one or more machine analysis tasks. In one or more modalities, the one or more machine analysis tasks may comprise object detection, image segmentation, instance segmentation, and / or similar tasks. In one or more modalities, the estimated performance of one or more machine analysis tasks may comprise a distortion computed based on at least a first set of features extracted from the output of a 914 or 932 decoder and a second set of features extracted from respective ground truth data, wherein the first set of features and the second set of features are output by one or more layers of a pretrained feature extraction neural network.

[0209] In one or more modalities, optimization or training can be performed jointly with respect to distortion loss D and rate loss R. In one or more modalities, optimization or training can be performed in two alternating phases, wherein, in a first phase of the two alternating phases, only distortion loss D can be used and, in a second phase of the two alternating phases, only rate loss R can be used.

[0210] As illustrated in Figure 9, in one or more modes, codec of Petition 870250095061, dated 10 / 17 / 2025, p. 182 / 543 53 / 72 intraframe 900 includes a first codec 905 that is used to initially encode the input image and can be referred to as the first-stage codec, and a second codec 910 that is used to encode the residue of the fundamental truth (e.g., the difference between the decoder output in the first-stage codec and the respective fundamental truth data) and can be referred to as the second-stage codec. In one or more embodiments, the input to the first-stage codec 905 may comprise a fundamental truth (e.g., a block or complete image) or part of the fundamental truth (e.g., masked fundamental truth). An example of input data in first-stage codec 905 is a fundamental truth.For example, the fundamental truth could be a YUV 4:4:4 color format image, represented as a 256x256x3 multidimensional array or tensor, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for the Y, U, and V components, respectively. In another example, the fundamental truth could be a YUV 4:2:0 color format image, represented by the combination of a 256x256 matrix for the luma component and a two-dimensional array (or tensor) of 128x128x2 size for the chroma component. In another example, the fundamental truth could include only the luma component, represented as a one-dimensional array (or tensor) of 256x256x1 size. In yet another example, the fundamental truth could include only the chroma component, represented as a two-dimensional array (or tensor) of 128x128x2 size. Another example of input data in first-stage codec 905 is partial fundamental truth.For example, the input data might be the result of fundamental truth multiplied with a mask or a Gaussian filter output. In one or more exemplary embodiments, moreover, the input to the first-stage codec 905 might comprise one or more extra data, such as block / image resolution.

[0211] In the exemplary embodiment illustrated in Figure 9, input data x is a fundamental truth including both luma and chroma components, with hxwx 3 describing the size of x with height h, width w and number of channels 3. In one or more embodiments, the first-stage codec 905 is used to initially encode the input data with encoder 916. In one or more embodiments, the decoder output 914 represents the initial reconstruction of the input data.

[0212] In one or more embodiments, the first-stage codec encoder 912 may comprise a neural encoder 916, a quantizer, a model of Petition 870250095061, dated 10 / 17 / 2025, p. 183 / 543 54 / 72 probability 924 and an entropy encoder 920. The quantizer is not illustrated in Figure 9, but is present in one or more modalities. In one or more modalities, the neural encoder 916 may comprise a first convolutional layer (“Conv5x5, 48, 1”, where conv means convolution, 5x5 is the kernel size, 48 is the number of output channels, and 1 is the step value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. In one or more modalities, the neural encoder 916 emits a latent tensor. In one or more modes, the latent tensor is converted by a quantizer into a quantized latent tensor.In one or more modes, the latent tensor or a quantized latent tensor can be inserted into the 924 probability model, with the dimensions of the latent tensor describing h / / 16 x w / / 16 x 128, where h / / 16 indicates the height, w / / 16 indicates the width, and 128 indicates the number of channels. In one or more modes, the 924 probability model outputs an estimate of the probability of each element of the (quantized) latent tensor. In one or more modes, the 924 probability model can be learned from data using machine learning techniques – for example, the 924 probability model can be a neural network and can be trained jointly with the other neural networks in the codec. In one or more embodiments, on the encoder side 912, the output of the probability model 924 is used as one of the inputs to an entropy encoder 920. In one or more embodiments, the entropy encoder may be an arithmetic encoder.In one or more embodiments, the 920 entropy encoder receives at least the latent tensor (quantized) and the 924 probability model output and emits a bit stream. In one or more embodiments, the latent tensor that is emitted to the entropy encoder may first be quantized.

[0213] In one or more embodiments, the first-stage codec decoder 914 may comprise an entropy decoder 922, a probability model 924, a dequantizer, and a neural decoder 928. In Figure 9, the dequantizer is not illustrated, but it is present in one or more embodiments. In one or more embodiments, the entropy decoder 922 may be an arithmetic decoder. In one or more embodiments, the entropy decoder 922 receives at least the stream Petition 870250095061, dated 10 / 17 / 2025, page 184 / 543 55 / 72 bits and the 924 probability model output and emits a decoded (quantized) latent tensor. In one or more modes, the 924 probability model may need to be the same probability model that is available on the 912 encoder side. In one or more modes, the decoded latent tensor may undergo dequantization. In one or more modes, the decoded latent tensor or the dequantized decoded latent tensor is then output to the 928 neural decoder.In one or more embodiments, the 928 neural decoder may comprise a first convolutional transposition layer (“UpConv5x5, 384, 2”, where UpConv refers to convolutional transposition, 5x5 is the kernel size, 384 is the number of output channels, and 2 is the step value), a first DSA block, a second convolutional transposition layer, a second DSA block, a third convolutional transposition layer, a third DSA block, a fourth convolutional transposition layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer. In one or more embodiments, the output x of the 928 neural decoder is an initial reconstruction of the input data x, where the size may be hxwx 3.

[0214] In one or more embodiments, the input to the second-stage codec 910 may comprise at least the residue between the fundamental truth (e.g., a block or complete image) and the initial reconstruction x of the first-stage codec. For example, the input to second-stage codec 910 may comprise the residue between the fundamental truth and the initial reconstruction x of first-stage codec 905. In one or more embodiments, the fundamental truth may be an image in YUV 4:4:4 color format, represented as a 256x256x3 multidimensional array or tensor, wherein the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for Y, U, V components, respectively. In one or more embodiments, the output x of first-stage codec 905 represents an initial reconstruction of the fundamental truth, wherein the size may be 256x256x3.In one or more modes, the residue may be the difference between the fundamental truth and reconstruction X, whose size may also be 256x256x3. In one or more modes, the input to the second-stage codec may also comprise one or more extra data, such as block / image resolution.

[0215] In the exemplary second-stage codec 910 shown in Figure 9, the input data is the residue between the fundamental truth and the Petition 870250095061, dated 10 / 17 / 2025, p. 185 / 543 56 / 72 initial reconstruction x of the first-stage codec 905. In one or more modes, hxwx 3 describes the size of r, with height h, width we, and number of channels 3. In one or more modes, the second-stage codec 910 is used to encode the residual information of the fundamental truth. In one or more modes, the bitstream emitted by the encoder 930 of the second-stage codec represents the encoded residual information, and the output of the decoder 932 represents the reconstructed residue. In one or more modes, the bitstream of the first-stage codec 905 and the bitstream of the second-stage codec 910 represent the encoded fundamental truth, and the combination (e.g., a sum) of the initial reconstruction of the first-stage codec 905 and the reconstructed residue of the second-stage codec 910 represents the final reconstructed fundamental truth.

[0216] In one or more embodiments, the second-stage codec 910 encoder 930 may comprise a neural encoder 934, an auxiliary encoder 944, a quantizer (not shown in Figure 9), a probability model 942, and an entropy encoder 936. In one or more embodiments, similar to the first-stage codec 905 neural encoder 916, the second-stage codec 910 neural encoder 934 may also comprise a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. In one or more embodiments, the dimension of the latent tensor output by the 934 neural encoder is described by h / / 16 x w / / 16 x 128, where the height is h / / 16, the width is w / / 16, and the number of channels is 128.

[0217] In one or more embodiments, the second-stage codec 910 also includes an auxiliary encoder 944 that generates auxiliary input for the probability model 942. In one or more embodiments, the auxiliary encoder 944 is used on the encoder side 930 and on the decoder side 932. In one or more embodiments, two copies of the same auxiliary encoder 944 can be created, where a first copy is used on the encoder side 930 and a second copy is used on the decoder side 932. In one or more embodiments, different auxiliary encoders are used on the encoder side 930 and on the decoder side 932. In one or more embodiments, the input x to the auxiliary encoder 944 is the first-stage codec output 905, which is the initial reconstruction of fundamental truth data. In one or Petition 870250095061, dated 10 / 17 / 2025, page 186 / 543 In 57 / 72 more modes, the input x for the auxiliary encoder 944 is a masked version of the initial reconstruction of fundamental truth data, which can be obtained via a masking operation performed on the initial reconstructed fundamental truth data. In one or more modes, the masking operation can mask (e.g., set to zero or another predetermined value) some of the elements of the initial reconstructed fundamental truth, such as elements that do not exist in the residual latent tensor being encoded or decoded by the entropy encoder 936 or entropy decoder 940. In one or more modes, the input x can first be fed into a prediction neural network to predict the residual information, and the predicted residual information can be used as an input for the auxiliary encoder 944.In one or more embodiments, the input x to the auxiliary encoder 944 comprises the latent tensor emitted by the entropy decoder 922 of the first-stage codec 905, or the dequantized latent tensor emitted by the dequantizer of the first-stage codec 905. In one or more embodiments, the auxiliary encoder 944 may have the same architecture as the neural encoder 934 of the second-stage codec 910. However, any architecture suitable for extracting features from an image may be suitable. In one or more embodiments, the output of auxiliary encoder 944 may be fed into the probability model 942 to function as extra context information. In one or more embodiments, the extra context information fed to the probability model 942 may comprise the latent tensor of the first-stage codec 905 or the dequantized latent tensor of the first-stage codec 905.In one or more embodiments, in a circumstance where the intraframe codec includes separate luma and chroma codecs, in addition to the possible auxiliary input described above, the x-input for the auxiliary encoder 944 may also comprise the reconstructed luma component, the reconstructed chroma component, or the corresponding latent tensors. In one or more embodiments where the initial reconstruction consists of sufficient information from the input data, the auxiliary input for the probability model 944 may enhance the accuracy of the probability density function estimated for the elements in the quantized latent tensor that must be encoded by the entropy encoder 936 or that must be decoded by the entropy decoder 940. This more accurate estimate by the probability model may bring significant performance gains to the encoding of residual information. Petition 870250095061, dated 10 / 17 / 2025, page 187 / 543 58 / 72

[0218] In one or more modes, the input size for auxiliary encoder 944 may be larger than the input and output size of second-stage codec 910. For example, when an input image is processed in blocks, the input for auxiliary encoder 944 may be larger than the size of a block, obtaining data from a larger area of ​​the image reconstructed by first-stage codec 905.

[0219] In one or more embodiments, the input x can be used for a hyperprior network. In one or more embodiments, the hyperprior network is an extra-compaction network, which provides side information μ and σ as prior information about the entropy parameters. In one or more embodiments, the side information μ and σ comprise hyperpriors. In one or more embodiments in which the element distribution in the residual information can be similar to the element distribution in the initial fundamental truth reconstruction, the input x related to the initial fundamental truth reconstruction can be used alone or in conjunction with the latent residual information tensor as the input to the hyperprior network.

[0220] In one or more embodiments, the latent tensor that is output by encoder 930 of second-stage codec 910 (e.g., the residual latent tensor) can be fed into probability model 942. In one or more embodiments, probability model 942 can be a neural network. In one or more embodiments, with auxiliary input as extra context information, probability model 942 outputs an estimate of the probability of each element of the residual latent tensor. In one or more embodiments, on the encoder 930 side, the output of probability model 942 is used as one of the inputs to an entropy encoder 936. In one or more embodiments, entropy encoder 936 can be an arithmetic encoder. In one or more modes, the entropy encoder 936 receives at least the latent (quantized) tensor and the output of the probability model 942 and outputs a bit stream representing the encoded residual information.In one or more modes, the latent tensor that is fed into the entropy encoder 936 is first quantized.

[0221] In one or more embodiments, the second-stage codec decoder 910 may comprise an entropy decoder 940, a probability model 942, a dequantizer (not shown in the figure), an auxiliary encoder 944, and a neural decoder 938. In one or more embodiments, the entropy decoder 940 may be an arithmetic decoder. In one or more embodiments, the decoder of Petition 870250095061, dated 10 / 17 / 2025, page 188 / 543 59 / 72 entropy 940 receives at least the bit stream emitted by encoder 930 and the output of probability model 942, and outputs a decoded (quantized) latent tensor. In one or more modes, probability model 942 may need to be the same probability model that is available on the encoder side 930. In one or more modes, auxiliary encoder 944 may also need to be the same auxiliary encoder 944 that is available on the encoder side 930. In one or more modes, the decoded (quantized) latent tensor may undergo dequantization. In one or more modes, after dequantization, the dequantized decoded latent tensor is fed into neural decoder 932. In one or more modes, the dequantized decoded latent tensor may be concatenated with the auxiliary input across the channel dimension.In one or more modes, after concatenation, the newly decoded latent tensor is fed into the 938 neural decoder. In one or more modes, the 938 neural decoder of second-stage codec 910 may comprise a first convolutional transposition layer, a first DSA block, a second convolutional transposition layer, a second DSA block, a third convolutional transposition layer, a third DSA block, a fourth convolutional transposition layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer. In one or more modes, the output of the 938 neural decoder is a reconstruction of the residual component, and the size may be hxwx 3. In one or more modes, the reconstructed residue is added to the initial reconstruction x, to obtain the final reconstruction.

[0222] In one or more modes, in a first training phase, the first-stage codec 905 can be trained and, in a second training phase, the second-stage codec 910 can be trained considering the initial reconstruction of the first-stage codec 905 as an input, wherein, in the first training phase, the second-stage codec 910 is not trained and, in the second training phase, the first-stage codec 905 is not trained. In one or more modes, the first-stage codec 905 and the second-stage codec 910 can be trained together (i.e., the second-stage codec 910 receives as one of its inputs an output from the decoder 914 in the first-stage codec 905 and both are trained at the same time). In one or more modes, the first-stage codec 905 and the second-stage codec 910 can be trained alternately (by Petition 870250095061, dated 10 / 17 / 2025, page 189 / 543 60 / 72 example, the first-stage codec 905 is trained for a first number of iterations, then the second-stage codec 910 is trained for a second number of iterations, then the first-stage codec 905 is trained for a third number of iterations, then the second-stage codec 910 is trained for a fourth number of iterations, and so on). A combination of these modes may also be possible, where, for example, in a first stage, the first-stage codec 905 and the second-stage codec 910 are trained sequentially (e.g., first the first-stage codec 905 is trained, then the second-stage codec 910 is trained) and, in a second stage, the first-stage codec 905 and the second-stage codec 910 are trained or tuned together.

[0223] Considering now Figure 10, an illustration of an exemplary system 1000 is provided in which more than two codecs are used. In one or more modes, the learned intraframe e2e codec may comprise n codecs, where n >= 2. In one or more modes illustrated in Figure 10, block 1010 represents a first-stage codec, block 1020 represents a second-stage codec, and the process performed by block 1020 is repeated one or more times after block 1020 culminates in block 10N representing an n-stage codec. In one or more modes, in each codec, the auxiliary data comprises the most recently reconstructed fundamental truth data. In one or more modes, the most recently reconstructed fundamental truth data is derived based on an output from a decoder of the previous codec. In one or more modes, the most recent fundamental truth data is a direct output from a decoder of the previous codec.In one or more embodiments, the most recently reconstructed fundamental truth data is determined based on output data obtained from the decoders of one or more previous codecs. In one or more embodiments, the most recently reconstructed fundamental truth data is the sum of the most recently reconstructed fundamental truth data from one or more previous codecs and the most recently reconstructed residual data.

[0224] In one or more embodiments, where the learned e2e codec comprises more than two encoding steps, for each of the one or more encoding steps subsequent to the first-step encoding, a residue is computed based on the fundamental truth data and a reconstruction that was determined based on a codec decoder output from the previous step, and Petition 870250095061, dated 10 / 17 / 2025, p. 190 / 543 61 / 72 This residue can be inserted into the encoder of the current encoding stage. In one or more embodiments, the probability model of each codec or the auxiliary encoder of each codec can receive as input one or more of the reconstructions that were determined based on an output from the codec decoder of the previous stage, the output from the codec decoder of the previous stage, and the latent tensor determined by the codec of the previous stage. In one or more embodiments, the codec can comprise a first-stage codec, a second-stage codec, and a third-stage codec.

[0225] In one or more modes, the input to the first-stage codec is an image representing the fundamental truth data. In one or more modes, the first-stage codec decoder outputs a first latent tensor. In one or more modes, the first-stage codec decoder outputs a first reconstructed image. In one or more modes, a first residue is computed as the difference between the first reconstructed image and the fundamental truth data.

[0226] In one or more modes, the first residue is fed into the second-stage codec. In one or more modes, the second-stage codec auxiliary encoder may receive as input the first latent tensor, the first reconstructed image, features extracted from the first latent tensor, features extracted from the first reconstructed image, or similar. In one or more modes, the second-stage codec decoder outputs a second latent tensor. In one or more modes, the second-stage codec decoder outputs a first reconstructed residue. In one or more modes, the first reconstructed residue is added to the first reconstructed image to obtain a second reconstructed image. In one or more modes, a second residue is computed as the difference between the second reconstructed image and the ground truth.

[0227] In one or more modes, the second residue is inserted into the third-stage codec. In one or more modes, the auxiliary encoder of the third-stage codec may receive as input the second latent tensor that is determined by the second-stage codec, the second reconstructed image, the first reconstructed residue, features extracted from the second latent tensor, features extracted from the second reconstructed image, features extracted from the first reconstructed residue, or similar. In one or more modes, the decoder of the third-stage codec outputs a second reconstructed residue. In one or more modes, the second Petition 870250095061, dated 10 / 17 / 2025, page 191 / 543 The 62 / 72 reconstructed residue is added to the second reconstructed image to obtain a third reconstructed image. In one or more modes, the third reconstructed image represents the final reconstructed image of the codec.

[0228] Considering now Figure 11, an illustrative flowchart is illustrated for a process 1100 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to receive residual fundamental truth data and ancillary data.

[0229] As shown in block 1110 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to receive, by a first codec, fundamental truth data. In one or more embodiments, the fundamental truth data comprises an image comprising brightness data and color data.

[0230] As shown in block 1120 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to, based on the fundamental truth data, generate a first bit stream.

[0231] As shown in block 1130 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to, based on the first bit stream, generate initial reconstruction data, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data.

[0232] As shown in block 1140 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to transmit, via the first codec, the initial reconstruction data.

[0233] As shown in block 1150 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine residual fundamental truth data, based at least on the initial reconstruction data and the fundamental truth data. In one or more embodiments, the residual fundamental truth data comprises a difference between the fundamental truth data and the initial reconstruction data. In one or more embodiments, fundamental truth data Petition 870250095061, dated 10 / 17 / 2025, p. 192 / 543 63 / 72 residuals are determined by the first codec.

[0234] As shown in block 1160 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream. In one or more embodiments, the auxiliary data comprises one or more of the initial reconstruction data, the first latent tensor and the first resulting latent tensor.

[0235] As shown in block 1170 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to receive, by a second codec, the residual fundamental truth data and the auxiliary data. In one or more embodiments, the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss. In one or more embodiments, the first codec is trained before the second codec, and the second codec is trained based at least on the first codec or based at least on data that is generated by the first codec. In one or more embodiments, the first codec and the second codec are trained simultaneously. In one or more embodiments, the first codec and the second codec are trained at alternating intervals.

[0236] As shown in block 1180 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to generate a second bit stream based on the residual fundamental truth data and auxiliary data. In one or more embodiments, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine a combined bit stream based at least on the first bit stream and the second bit stream.

[0237] As shown in optional block 1190 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to generate residual reconstruction data based on the second bit stream, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data. In one or more embodiments, the apparatus incorporated by the service device 22 includes means, such as the Petition 870250095061, dated 10 / 17 / 2025, page 193 / 543 64 / 72 processor 12, communication interface 16 or similar to determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. In one or more embodiments, the combined reconstruction data is determined by the second codec.

[0238] As shown in optional block 1195 of Figure 11, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to transmit, via the second codec, the residual reconstruction data.

[0239] Considering now Figure 12, an illustrative flowchart is illustrated for a process 1200 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to encode fundamental truth data into a first bit stream.

[0240] As shown in block 1210 of Figure 12, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a first neural encoder, the fundamental truth data into a first latent tensor.

[0241] As shown in block 1220 of Figure 12, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or the like, to generate, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element.

[0242] As shown in block 1230 of Figure 12, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine, for a respective symbol or element of at least one symbol or element of the first quantized latent tensor, using a first probability model, a first estimated probability distribution of possible values.

[0243] As shown in block 1240 of Figure 12, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, for encoding, using a first entropy encoder, the respective symbol or element of at least one symbol or element of the first Petition 870250095061, dated 10 / 17 / 2025, page 194 / 543 65 / 72 quantized latent tensor in the first bit stream based at least on the first estimated probability distribution of possible values.

[0244] Considering now Figure 13, an illustrative flowchart is illustrated for a process 1300 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to decode a first bit stream into initial reconstruction data.

[0245] As shown in block 1310 of Figure 13, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to decode, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor.

[0246] As shown in block 1320 of Figure 13, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to generate, using a first dequantizer, a first resulting latent tensor.

[0247] As shown in block 1330 of Figure 13, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a first neural decoder, the resulting first latent tensor into the initial reconstruction data.

[0248] Considering now Figure 14, an illustrative flowchart is illustrated for a process 1400 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to encode residual fundamental truth data into a second bit stream.

[0249] As shown in block 1410 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a second neural encoder, the residual fundamental truth data into a second latent tensor.

[0250] As shown in block 1420 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to generate, using a second quantizer and a second Petition 870250095061, dated 10 / 17 / 2025, page 195 / 543 66 / 72 predefined set of quantization levels, a second quantized latent tensor based at least on the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element.

[0251] As shown in block 1430 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using an auxiliary encoder, the auxiliary data into auxiliary resources.

[0252] As shown in block 1440 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to insert auxiliary resources into a second probability model.

[0253] As shown in block 1450 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine, using the second probability model, a second estimated probability distribution of possible values ​​for a respective symbol or element of the second quantized latent tensor.

[0254] As shown in block 1460 of Figure 14, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to encode, using a second entropy encoder, the respective symbol or element of at least one symbol or element of the second quantized latent tensor in the second bit stream based on at least the second estimated probability distribution of possible values.

[0255] Considering now Figure 15, an illustrative flowchart is illustrated for a process 1500 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to decode a second bit stream into residual reconstruction data.

[0256] As shown in block 1510 of Figure 15, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, for decoding, using a second entropy decoder, the auxiliary encoder or another auxiliary encoder that is the same as the auxiliary encoder, and the second probability model or another model of Petition 870250095061, dated 10 / 17 / 2025, page 196 / 543 67 / 72 probability which is the same as the second probability model, the second bit stream in the second quantized latent tensor or in a quantized latent tensor which is the same as the second quantized latent tensor.

[0257] As shown in block 1520 of Figure 15, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to generate, using a second dequantizer, a second resulting latent tensor.

[0258] As shown in block 1530 of Figure 15, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a second neural decoder, the resulting second latent tensor into the residual reconstruction data. In one or more embodiments, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model and the auxiliary encoder comprise neural network components.

[0259] Considering now Figure 16, an illustrative flowchart is illustrated for a process 1600 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) service device 22 to emit additional reconstruction data.

[0260] As shown in block 1610 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine additional residual fundamental truth data, based at least on the combined reconstruction data and the fundamental truth data.

[0261] As shown in block 1620 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to determine additional auxiliary data based at least on the second bit stream or based at least on data derived from the second bit stream.

[0262] As shown in block 1630 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to receive, by a different codec, the actual data. Petition 870250095061, dated 10 / 17 / 2025, page 197 / 543 68 / 72 fundamental additional residuals and additional auxiliary data.

[0263] As shown in block 1640 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to, based at least on the additional residual fundamental truth data and additional auxiliary data, generate an additional bit stream.

[0264] As shown in block 1650 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or the like, to, based at least on the additional bit stream, generate additional reconstruction data, wherein the additional reconstruction data comprise a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data are operable to combine with the combined reconstruction data to form composite reconstruction data.

[0265] As shown in block 1660 of Figure 16, the apparatus incorporated by the service device 22 includes means, such as the processor 12, the communication interface 16 or similar, to transmit, by the different codec, at least one of the additional reconstruction data and the composite reconstruction data.

[0266] Considering now Figure 17, an illustrative flowchart is illustrated for a process 1700 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) consumer device 24 to decode bit streams into reconstruction data.

[0267] As shown in block 1710 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to receive a first bit stream.

[0268] As shown in block 1720 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to, based on the first bit stream, generate initial reconstruction data.

[0269] As shown in block 1730 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to determine auxiliary data based at least on Petition 870250095061, dated 10 / 17 / 2025, page 198 / 543 69 / 72 first bit stream or based at least on data derived from the first bit stream.

[0270] As shown in block 1740 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to output the initial reconstruction data.

[0271] As shown in block 1750 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to receive a second bit stream. In one or more embodiments, the first bit stream and the second bit stream are received as parts of a combined bit stream.

[0272] As shown in block 1760 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to, based on the second bit stream and auxiliary data, generate residual reconstruction data.

[0273] As shown in block 1770 of Figure 17, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to output the residual reconstruction data. In one or more embodiments, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to determine combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. In one or more embodiments, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to

[0274] Considering now Figure 18, an illustrative flowchart is illustrated for an 1800 process performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) consumer device 24 to decode a first bit stream into initial reconstruction data.

[0275] As shown in block 1810 of Figure 18, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to decode, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor. Petition 870250095061, dated 10 / 17 / 2025, page 199 / 543 70 / 72

[0276] As shown in block 1820 of Figure 18, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to generate, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor.

[0277] As shown in block 1830 of Figure 18, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a first neural decoder, the first latent tensor in the initial reconstruction data.

[0278] Considering now Figure 19, an illustrative flowchart is illustrated for a process 1900 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) consumer device 24 to decode a second bit stream into residual reconstruction data.

[0279] As shown in block 1910 of Figure 19, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to decode, using a second entropy decoder, an auxiliary encoder, and a second probability model, the second bit stream into a second quantized latent tensor.

[0280] As shown in block 1920 of Figure 19, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to generate, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor.

[0281] As shown in block 1930 of Figure 19, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to convert, using a second neural decoder, the second latent tensor in the residual reconstruction data. In one or more embodiments, at least one of the first neural decoder, the first probability model, the second neural decoder and the second probability model comprises neural network components.

[0282] Considering now Figure 20, an illustrative flowchart is shown for a process 2000 performed by an apparatus incorporated by, associated with, or otherwise communicating with (hereinafter generally referred to in this document as being incorporated by) a consumer device 24 to decode a flow Petition 870250095061, dated 10 / 17 / 2025, page 200 / 543 71 / 72 additional bits in additional reconstruction data.

[0283] As shown in block 2010 of Figure 20, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to receive an additional bit stream.

[0284] As shown in block 2020 of Figure 20, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to, based on the additional bit stream, generate additional reconstruction data, wherein the additional reconstruction data are operable to combine with the combined reconstruction data to form composite reconstruction data.

[0285] As shown in block 2030 of Figure 20, the apparatus incorporated by the consumer device 24 includes means, such as the processor 12, the communication interface 16 or similar, to output at least one of the additional reconstruction data and the composite reconstruction data.

[0286] Figures 11-20 illustrate flowcharts showing methods according to an exemplary embodiment of the present disclosure. It will be understood that each block of the flowcharts, and combination of blocks in the flowcharts, can be implemented by various means, such as hardware, firmware, processor, circuit assembly and / or other communication devices associated with software execution including one or more computer program instructions. For example, one or more of the procedures described above can be incorporated by computer program instructions. In this sense, the computer program instructions incorporating the procedures described above can be stored by a memory device 14 of an apparatus employing an embodiment and executed by a processor 12.As will be understood, any of these computer program instructions can be loaded into a computer or other programmable device (e.g., hardware) to produce a machine, so that the resulting computer or other programmable device implements the functions specified in the flowchart blocks. These computer program instructions can also be stored in computer-readable memory that can direct a computer or other programmable device to operate in a particular way, so that the instructions stored in the computer-readable memory produce a manufactured article whose execution implements the function specified in the blocks. Petition 870250095061, dated 10 / 17 / 2025, page 201 / 543 72 / 72 flowchart. Computer program instructions can also be loaded into a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide operations to implement the functions specified in the flowchart blocks.

[0287] Consequently, flowchart blocks support combinations of means to perform the specified functions and combinations of operations to perform the specified functions. It will also be understood that one or more flowchart blocks and combinations of blocks in flowcharts may be implemented by computer systems based on special-purpose hardware that perform the specified functions or combinations of special-purpose hardware and computer instructions.

[0288] Many modifications and other embodiments set forth herein will occur to a person skilled in the art to which this disclosure pertains, having the benefit of the teachings set forth in the preceding descriptions and associated drawings. Therefore, it will be understood that the disclosure is not to be limited to the specific embodiments disclosed and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0289] Furthermore, although the preceding descriptions and associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this sense, for example, combinations of elements and / or functions other than those explicitly described above are also contemplated, as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used only in a generic and descriptive sense and not for purposes of limitation. Petition 870250095061, dated 10 / 17 / 2025, page 202 / 543

Claims

1 / 14 CLAIMS 1. A computer-implemented method characterized by comprising: receiving, by a first codec, fundamental truth data; based on the fundamental truth data, generating a first bit stream; based on the first bit stream, generating initial reconstruction data, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data; outputting, by the first codec, the initial reconstruction data; determining residual fundamental truth data based at least on the initial reconstruction data and the fundamental truth data; determining auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream; receiving, by a second codec, the residual fundamental truth data and the auxiliary data; and based on the residual fundamental truth data and the auxiliary data, generating a second bit stream.

2. A computer-implemented method according to claim 1, characterized by further comprising: based on the second bitstream, generating residual reconstruction data, wherein the residual reconstruction data comprises a reconstruction of the residual fundamental truth data; and outputting, via the second codec, the residual reconstruction data.

3. A computer-implemented method, according to any one of claims 1-2, characterized by further comprising: determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

4. A computer-implemented method, according to any one of claims 1-3, characterized by further comprising: determining a combined bitstream based on at least the first bitstream and the second bitstream.

5. A computer-implemented method, according to any one of claims 1-4, characterized in that the residual fundamental truth data comprises a difference between the fundamental truth data and the initial reconstruction data.

6. A computer-implemented method, according to any one of claims 1-5, characterized in that the fundamental truth data comprises an image comprising brightness data and color data.

7. A computer-implemented method according to any one of claims 1-6, characterized in that generating the first bitstream comprises: converting, using a first neural encoder, the fundamental truth data into a first latent tensor; generating, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element; for a respective symbol or element of the first quantized latent tensor, determining, using a first probability model, an estimated first probability distribution of possible values;and encode, using a first entropy encoder, the respective symbol or element of at least one symbol or element of the first quantized latent tensor in the first bit stream based on at least the first estimated probability distribution of possible values.

8. A computer-implemented method according to any one of claims 1-7, characterized in that generating the initial reconstruction data comprises: decoding, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream Petition 870250095061, 10 / 17 / 2025, p. 204 / 543 3 / 14 into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor; generating, using a first dequantizer, a first resulting latent tensor; and converting, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

9. A computer-implemented method according to any one of claims 1-8, characterized in that generating the second bitstream comprises: converting, using a second neural encoder, the residual fundamental truth data into a second latent tensor; generating, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based at least on the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element; converting, using an auxiliary encoder, the auxiliary data into auxiliary features; inserting the auxiliary features into a second probability model; for a respective symbol or element of at least one symbol or element of the second quantized latent tensor, determining, using the second probability model, a second estimated probability distribution of possible values;and encode, using a second entropy encoder, the respective symbol or element of at least one symbol or element of the second quantized latent tensor in the second bitstream based on at least the second estimated probability distribution of possible values.

10. A computer-implemented method according to any one of claims 1-9, characterized in that generating the residual reconstruction data comprises: decoding, using a second entropy decoder, the auxiliary encoder Petition 870250095061, 10 / 17 / 2025, p. 205 / 543 4 / 14 or another auxiliary encoder that is the same as the auxiliary encoder, and the second probability model or another probability model that is the same as the second probability model, the second bit stream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor; generating, using a second dequantizer, a second resulting latent tensor; and converting, using a second neural decoder, the second resulting latent tensor into the residual reconstruction data.

11. A computer-implemented method, according to any one of claims 7-10, characterized in that at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprise neural network components.

12. A computer-implemented method, according to any one of claims 1-11, characterized in that the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss.

13. A computer-implemented method according to claim 12, characterized in that the first codec is trained before the second codec, and in that the second codec is trained based at least on the first codec or based at least on data generated by the first codec.

14. A computer-implemented method according to claim 12, characterized in that the first codec and the second codec are trained simultaneously.

15. A computer-implemented method according to claim 12, characterized in that the first codec and the second codec are trained at alternating intervals.

16. A computer-implemented method, in accordance with any of claims 1-15, as described in Petition 870250095061, dated 10 / 17 / 2025, page 206 / 543 5 / 14, characterized in that the auxiliary data comprises one or more of the initial reconstruction data, the first latent tensor, and the first resulting latent tensor.

17. A computer-implemented method, according to any one of claims 1-16, characterized in that the residual fundamental truth data is determined by the first codec.

18. A computer-implemented method, according to any one of claims 3-17, characterized in that the combined reconstruction data is determined by the second codec.

19. A computer-implemented method according to any one of claims 3-18, characterized by further comprising: determining additional residual fundamental truth data based at least on the combined reconstruction data and the fundamental truth data; determining additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream; receiving, by a different codec, the additional residual fundamental truth data and the additional auxiliary data; based at least on the additional residual fundamental truth data and the additional auxiliary data, generating an additional bitstream;Based on the additional bitstream, generate additional reconstruction data, wherein the additional reconstruction data comprises a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data; and output, by the different codec, at least one of the additional reconstruction data and the composite reconstruction data.

20. A computer-implemented method characterized by comprising: receiving a first bit stream; based on the first bit stream, generating initial reconstruction data; determining auxiliary data based at least on the first bit stream or Petition 870250095061, dated 10 / 17 / 2025, page 207 / 543 6 / 14 based at least on data derived from the first bit stream; emitting the initial reconstruction data; receiving a second bit stream; based on the second bit stream and the auxiliary data, generating residual reconstruction data; and emitting the residual reconstruction data.

21. A computer-implemented method according to claim 20, characterized by further comprising: determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

22. A computer-implemented method, according to any one of claims 20-21, characterized in that the first bit stream and the second bit stream are received as parts of a combined bit stream.

23. A computer-implemented method, according to any one of claims 20-22, characterized in that generating initial reconstruction data comprises: decoding, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor; generating, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor; and converting, using a first neural decoder, the first latent tensor into the initial reconstruction data.

24. A computer-implemented method according to any one of claims 20-23, characterized in that generating residual reconstruction data comprises: decoding, using a second entropy decoder, an auxiliary encoder and a second probability model, the second bitstream into a second quantized latent tensor; generating, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor; and converting, using a second neural decoder, the second latent tensor into residual reconstruction data.

25. A computer-implemented method, according to any one of claims 20-24, characterized in that at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises neural network components.

26. A computer-implemented method according to any one of claims 21-25, characterized by further comprising: receiving an additional bit stream; based on the additional bit stream, generating additional reconstruction data, wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data; and outputting at least one of the additional reconstruction data and the composite reconstruction data.

27. An apparatus characterized by comprising: at least one processor; and at least one memory that stores instructions which, when executed by at least one processor, cause the apparatus to at least perform: receiving, by a first codec, fundamental truth data; based on the fundamental truth data, generating a first bit stream; based on the first bit stream, generating initial reconstruction data, wherein the initial reconstruction data comprises a reconstruction of the fundamental truth data; outputting, by the first codec, the initial reconstruction data; determining residual fundamental truth data based at least on the initial reconstruction data and the fundamental truth data; determining auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream; receiving, by a second codec, the residual fundamental truth data and the auxiliary data;and Petition 870250095061, dated 10 / 17 / 2025, page 209 / 543 8 / 14 based on the residual fundamental truth data and auxiliary data, generate a second bit stream.; 28. Apparatus, according to claim 27, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: based on the second bitstream, generate residual reconstruction data, wherein the residual reconstruction data comprise a reconstruction of the residual fundamental truth data; and output, by the second codec, the residual reconstruction data.

29. Apparatus, according to any one of claims 27-28, characterized in that at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to: determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

30. Apparatus, according to any one of claims 27-29, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: determine a combined bit stream based on at least the first bit stream and the second bit stream.

31. Apparatus, according to any one of claims 27-30, characterized in that the residual fundamental truth data comprise a difference between the fundamental truth data and the initial reconstruction data.

32. Apparatus, according to any one of claims 27-31, characterized in that the fundamental truth data comprises an image comprising brightness data and color data.

33. Apparatus, according to claims 27-32, characterized in that at least one memory and computer program code are additionally configured to, with at least one processor, make the apparatus: Petition 870250095061, dated 10 / 17 / 2025, p.210 / 543 9 / 14 convert, using a first neural encoder, the fundamental truth data into a first latent tensor; generate, using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor based at least on the first latent tensor, wherein the first quantized latent tensor comprises at least one symbol or element; for a respective symbol or element of the at least one symbol or element of the first quantized latent tensor, determine, using a first probability model, a first estimated probability distribution of possible values; and encode, using a first entropy encoder, the respective symbol or element of the at least one symbol or element of the first quantized latent tensor into the first bit stream based at least on the first estimated probability distribution of possible values.

34. Apparatus, according to any one of claims 27-33, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: decode, using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream into the first quantized latent tensor or into a quantized latent tensor that is the same as the first quantized latent tensor; generate, using a first dequantizer, a first resulting latent tensor; and convert, using a first neural decoder, the first resulting latent tensor into the initial reconstruction data.

35. Apparatus, according to any one of claims 27-34, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: Petition 870250095061, 10 / 17 / 2025, p. 211 / 543 10 / 14 convert, using a second neural encoder, the residual fundamental truth data into a second latent tensor; generate, using a second quantizer and a second predefined set of quantization levels, a second quantized latent tensor based on at least the second latent tensor, wherein the second quantized latent tensor comprises at least one symbol or element; convert, using an auxiliary encoder, the auxiliary data into auxiliary features; insert the auxiliary features into a second probability model;For a given symbol or element of at least one symbol or element of the second quantized latent tensor, determine, using the second probability model, a second estimated probability distribution of possible values; and encode, using a second entropy encoder, the given symbol or element of at least one symbol or element of the second quantized latent tensor in the second bitstream based on at least the second estimated probability distribution of possible values.

36. Apparatus, according to any one of claims 27-35, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: decode, using a second entropy decoder, the auxiliary encoder and the second probability model or a copy of the second probability model, the second bit stream into the second quantized latent tensor or into a quantized latent tensor that is the same as the second quantized latent tensor; generate, using a second dequantizer, a second resulting latent tensor; and convert, using a second neural decoder, the second resulting latent tensor into residual reconstruction data.

37. Apparatus, according to any one of claims 33-36, characterized Petition 870250095061, of 10 / 17 / 2025, page 212 / 543 11 / 14 by the fact that at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model of the auxiliary encoder comprises neural network components.

38. Device, according to any one of claims 27-37, characterized in that the first codec and the second codec are trained in an end-to-end manner, decreasing at least one distortion loss and rate loss.

39. Device according to claim 38, characterized in that the first codec is trained before the second codec, and in that the second codec is trained based at least on the first codec or based at least on data generated by the first codec.

40. Device according to claim 38, characterized in that the first codec and the second codec are trained simultaneously.

41. Device according to claim 38, characterized in that the first codec and the second codec are trained at alternating intervals.

42. Apparatus, according to any one of claims 27-41, characterized in that the auxiliary data comprise one or more of the initial reconstruction data, the first latent tensor, the first resulting latent tensor.

43. Apparatus, according to any one of claims 27-42, characterized in that the residual fundamental truth data is determined by the first codec.

44. Device, according to any one of claims 29-43, characterized in that the combined reconstruction data is determined by the second codec.

45. Apparatus, according to any one of claims 29-44, characterized in that at least one memory and computer program code are additionally configured to, with at least one processor, enable the apparatus to: determine additional residual fundamental truth data, based at least on the combined reconstruction data and the fundamental truth data; Petition 870250095061, dated 10 / 17 / 2025, p.213 / 543 12 / 14 determine additional auxiliary data based at least on the second bitstream or based at least on data derived from the second bitstream; receive, by a different codec, the additional residual fundamental truth data and the additional auxiliary data; based at least on the additional residual fundamental truth data and the additional auxiliary data, generate an additional bitstream; based on the additional bitstream, generate additional reconstruction data, wherein the additional reconstruction data comprises a reconstruction of the additional residual fundamental truth data, and wherein the additional reconstruction data is operable to combine with the combined reconstruction data to form composite reconstruction data; and output, by the different codec, the additional reconstruction data.

46. ​​An apparatus characterized by comprising: at least one processor; and at least one memory that stores instructions which, when executed by at least one processor, cause the apparatus to at least: receive a first bit stream; based on the first bit stream, generate initial reconstruction data; determine auxiliary data based at least on the first bit stream or based at least on data derived from the first bit stream; output the initial reconstruction data; receive a second bit stream; based on the second bit stream and the auxiliary data, generate residual reconstruction data; and output the residual reconstruction data.

47. Apparatus, according to claim 46, characterized in that at least one memory and computer program code are further configured to, with at least one processor, enable the apparatus to: determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. Petition 870250095061, dated 10 / 17 / 2025, pp. 214 / 543 13 / 14 48. Apparatus, according to any one of claims 46-47, characterized in that the first bit stream and the second bit stream are received as parts of a combined bit stream.

49. Apparatus, according to any one of claims 46-48, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: decode, using a first entropy decoder and a first probability model, the first bit stream into a first quantized latent tensor; generate, using a first dequantizer and based on the first quantized latent tensor, a first latent tensor; and convert, using a first neural decoder, the first latent tensor into the initial reconstruction data.

50. Apparatus, according to any one of claims 46-49, characterized in that at least one memory and computer program code are further configured to, with at least one processor, make the apparatus: decode, using a second entropy decoder, an auxiliary encoder, and a second probability model, the second bit stream into a second quantized latent tensor; generate, using a second dequantizer and based on the second quantized latent tensor, a second latent tensor; and convert, using a second neural decoder, the second latent tensor into residual reconstruction data.

51. Apparatus, according to any one of claims 46-50, characterized in that at least one of the first neural decoder, the first probability model, the second neural decoder and the second probability model comprise neural network components.

52. Apparatus, according to any one of claims 46-51, characterized in that at least one memory and computer program code (Petition 870250095061, 10 / 17 / 2025, p. 215 / 543 14 / 14) are further configured to, with at least one processor, make the apparatus: receive an additional bit stream; based on the additional bit stream, generate additional reconstruction data, wherein the additional reconstruction data are operable to combine with the combined reconstruction data to form composite reconstruction data; and output the additional reconstruction data.

53. Non-transient computer-readable storage medium characterized by comprising computer instructions which, when executed by an apparatus, cause the apparatus to perform the method as defined in any one of claims 1-19.

54. Non-transient computer-readable storage medium characterized by comprising computer instructions which, when executed by an apparatus, cause the apparatus to perform the method as defined in any one of claims 20-26. Petition 870250095061, dated 10 / 17 / 2025, pp. 216 / 543

Images