A method for tuning an intraframe codec in an end-to-end trained codec.

Abstract

Methods, apparatus, and computer program products are provided. In the context of a method, the method receives ground truth data by a first codec. The method generates a first bitstream based on the ground truth data. The method generates initial reconstruction data based on the first bitstream, the initial reconstruction data includes reconstruction of the ground truth data. The method outputs the initial reconstruction data by a first codec. The method determines residual ground truth data based at least on the initial reconstruction data and the ground truth data. The method determines auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The method receives residual ground truth data and auxiliary data by a second codec. The method generates a second bitstream based on the residual ground truth data and auxiliary data.

Description

Technical Field

[0001] The exemplary embodiments relate to video and image coders - decoders (codecs), and more particularly, to end - to - end learned coders - decoders.

Background Art

[0002] Neural networks (NN) are being used in ever - increasing applications for various devices such as mobile phones. Neural networks are used for image and video analysis and processing, social media data analysis, device usage analysis, and other applications. A neural network is a computational graph consisting of several layers of computations. Each layer can consist of one or more units that perform elementary computations. The units are connected to one or more other units, and the connections are often associated with weights that are used to scale the signals passing through the connections. The weights are values that can be learned from training data. Further, batch normalization layer parameters can be learned from training data.

[0003] A feed - forward neural network has no feedback loop, and each layer receives an input from the previous layer and provides its output to the next layer. The initial layers, closer to the input data, extract low - level features such as edges and textures of an image, while the intermediate and final layers extract higher - level features. Further, one or more layers can perform tasks such as classification, semantic segmentation, object detection, noise removal, style transfer, super - resolution, and / or the like. A recurrent neural network has a feedback loop, making the neural network stateful. A recurrent neural network can store information or state.

[0004] Neural networks can learn properties from input data in a supervised or unsupervised manner. This ability arises as a result of a training algorithm or metalevel neural network providing training signals. The training algorithm modifies the properties of the neural network so that its output approaches the desired output as closely as possible. For example, if a neural network classifies objects in an image, the neural network output may be used to derive the object's class or category index. Training minimizes or reduces the error of the output, called loss. Examples of loss include mean squared error and cross-entropy. Training can be an iterative process in which the algorithm modifies the weights of the neural network so that the network's loss gradually decreases with each iteration. The ultimate goal of training a neural network is to have it learn the properties of the data distribution from a limited training dataset, or to have the neural network "generalize" data that was not used to train it. Validation datasets that were not used for training are often used to check the performance of the neural network.

[0005] When checking the performance of a neural network, the training set error should decrease. Otherwise, the neural network is underfitted. If the neural network is working to generalize, the validation set error should decrease and not be significantly higher than the training set error. Otherwise, the neural network is overfitted (i.e., the neural network has memorized the properties of the training set and performs well only on that set).

[0006] In the case of image codecs, neural network-based codecs can be used to compress and decompress data such as images. A neural network-based image codec may include a component called an autoencoder, which includes a neural encoder and a neural decoder. The neural encoder takes an image as input and generates a latent tensor that may require fewer bits than the input image. The latent tensor can be quantized and losslessly compressed to obtain a bitstream representing the encoded image. The neural decoder uses the bitstream to reconstruct the image input to the neural encoder.

[0007] Neural encoders and decoders are trained to minimize a combination of bitrate and distortion. Distortion can be based on mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index measure (SSIM), or similar metrics. These distortion metrics correlate with human visual quality, and therefore improving them improves the visual quality of the decoded image as perceived by humans.

[0008] A video codec includes a neural encoder that converts the input video into a compressed representation suitable for storage / transmission, and a neural decoder that decompresses the compressed video representation back into a viewable format. The neural encoder generally discards some information from the original video in order to represent the video in a more compact format (at a lower bitrate). A hybrid video codec can encode video information in two phases. First, the pixel values ​​of a particular picture block are predicted. For example, the pixel values ​​may be predicted by motion compensation means (i.e., finding and indicating one area of ​​a previously coded video frame that closely corresponds to the block being coded) or by spatial means (i.e., using pixel values ​​around the block that is to be coded in a specified manner). Second, the prediction error (i.e., the difference between the predicted block of pixels and the original block of pixels) is coded. For example, this may be done by transforming the difference in pixel values ​​using a specified transformation (e.g., a discrete cosine transform), quantizing the coefficients, and entropicoding the quantized coefficients. By varying the fidelity of the quantization process, neural encoders control the balance between the accuracy of the pixel presentation (pixel quality) and the size of the resulting coded video representation (file size or transmission bitrate).

[0009] Interpretation (also known as time prediction, motion compensation, or motion-compensated prediction) leverages time redundancy. Interpretation uses previously decoded pictures as the source of predictions. In contrast, intrapretation uses adjacent pixels within the same two pictures. Intrapretation can be performed in the spatial domain to predict sample values ​​or in the transformation domain to predict transformation coefficients. Intrapretation is often utilized in intracoding where no prediction is applied.

[0010] The output of the coding procedure is a set of coding parameters, such as motion vectors and quantized transformation coefficients. Parameters can be entropi-coded more efficiently if they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors may be predicted from spatially adjacent motion vectors, and only the difference to the motion vector predictor may be coded. Prediction and intra-prediction of coding parameters are collectively called in-picture predictions.

[0011] The decoder reconstructs the video input to the encoder by applying a similar prediction mechanism to the encoder. By doing so, the decoder uses the motion or spatial information produced by the encoder and stored in a compressed representation to form a predicted representation of the pixel blocks. The decoder performs prediction error decoding (the reverse operation of prediction error coding, which reconstructs the quantized prediction error signal in the spatial pixel domain). After applying prediction and prediction error decoding, the decoder sums the prediction and prediction error signals (pixel values) to form an output video frame. The decoder and encoder may also apply additional filtering to improve the quality of the output video before passing the output video for display or storing the output video as a prediction reference for the next frame in the video sequence.

[0012] In a typical video coder-decoder, motion information is represented by motion vectors associated with each motion-compensated image block. Each motion vector represents the displacement between each image block in the picture to be coded or decoded and the predicted source block in one of the previously coded or decoded pictures. To efficiently represent motion vectors, they are generally coded differently from block-specific predicted motion vectors. In common video codecs, predicted motion vectors are generated in a predefined manner. For example, a predicted motion vector may be generated by calculating the median of the encoded or decoded motion vectors of adjacent blocks. Another way to generate motion vector predictions is by generating a list of candidate predictions from adjacent blocks and / or blocks located in the same place in the time-referenced picture, and signaling the selected candidates as motion vector predictors. Furthermore, the reference index of the previously coded or decoded picture may be predicted. The reference index may be predicted from adjacent blocks and / or blocks located in the same place in the time-referenced picture. In addition, common high-efficiency video codecs employ additional motion information coding / decoding mechanisms. This mechanism, often referred to as merge mode, predicts and uses all motion field information (including motion vectors and corresponding reference picture indices for each available reference picture list) without modification or correction. Similarly, predicting motion field information is performed by using motion field information from adjacent blocks and / or blocks located in the same location in the time reference picture. The motion field information used is signaled from a list of motion field candidates, which includes motion field information from available adjacent and / or same-location blocks.

[0013] In typical video codecs, the predicted residuals after motion compensation are first transformed by a transformation kernel such as DCT and then coded. This is done to reduce the correlation between predicted residuals and provide more efficient coding. Typical video encoders use a Lagrangian cost function to find the optimal coding mode (e.g., the desired macroblock mode and associated motion vector). This is a costly function that uses a weighting factor λ to tie together the image distortion caused by the irreversible coding method with the amount of information required to represent the pixel values ​​in the image area. The cost can be expressed by the equation C = D + λR, where C is the Lagrangian cost to be minimized, D is the image distortion (mean squared error, etc.) considering the mode and motion vector, and R is the number of bits required to represent the data required to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vector).

[0014] In some situations, supplemental enhancement information (SEI) messages may be used. In some situations, SEI NAL units containing prefix SEI NAL units and suffix SEI NAL units are used. A prefix SEI NAL unit can start a picture unit, and a suffix SEI NAL unit can end a picture unit. An SEI unit contains one or more SEI messages, one or more of which are not required for decoding the output picture but can assist in related processes such as picture output timing, post-processing of the decoded picture, rendering, error detection, error hiding, and resource allocation.

[0015] Image and video codecs can use a set of filters to enhance the visual quality of predicted visual content. These filters can be applied in-loop, out-loop, or both. In the case of in-loop filters, a filter applied to one block in the frame currently being encoded will affect the encoding of another block in the same frame and / or in another frame predicted from the currently encoded frame. In-loop filters can affect bitrate and / or visual quality. Enhanced blocks will result in smaller residuals (the difference between the original block and the predicted and filtered block), and therefore require fewer bits to be encoded. Out-of-loop filters are applied to frames after the frame has been reconstructed. In this situation, the filtered visual content is not the source for prediction and can only affect the visual quality of the frame output by the decoder. [Overview of the project] [Problems that the invention aims to solve]

[0016] Embodiments of this specification disclose improved video and image codecs that improve rate-distortion performance of video and image compression using end-to-end trained codecs. Embodiments of this disclosure provide lower bitrates and higher PSNR in the resulting reconstructed image luma than prior embodiments. Embodiments of this disclosure achieve better rate-distortion performance in the reconstructed image. [Means for solving the problem]

[0017] Methods, apparatus, and computer program products for end-to-end trained intraframe codecs are disclosed. By using end-to-end training with multiple codecs, the bitrate is reduced and the PSNR is improved.

[0018] One exemplary form provides a method comprising receiving ground truth data by a first codec. The method further comprises generating a first bitstream based on the ground truth data. The method further comprises generating initial reconstructed data based on the first bitstream, wherein the initial reconstructed data comprises a reconstruction of the ground truth data. The method further comprises outputting the initial reconstructed data by the first codec. The method further comprises determining residual ground truth data based at least on the initial reconstructed data and the ground truth data. The method further comprises determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The method further comprises receiving residual ground truth data and auxiliary data by a second codec. The method further comprises generating a second bitstream based on the residual ground truth data and auxiliary data.

[0019] The exemplary method comprises generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of residual ground truth data. The method further comprises outputting the residual reconstruction data by a second codec.

[0020] The exemplary method further includes determining the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0021] The exemplary method further includes determining a combined bitstream based at least on a first bitstream and a second bitstream.

[0022] In one exemplary form, residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

[0023] In one exemplary form, the ground truth data includes an image containing brightness data and color data.

[0024] The exemplary method further includes converting ground truth data to a first latent tensor using a first neural encoder. The method further includes generating a first quantized latent tensor based on the first latent tensor using a first quantizer and a first predefined set of quantization levels, wherein the first quantized latent tensor contains at least one symbol or element. The method further includes determining a first estimated probability distribution of possible values ​​using a first probabilistic model for each of the at least one symbol or element of the first quantized latent tensor. The method further includes encoding each of the at least one symbol or element of the first quantized latent tensor into a first bitstream based on at least the first estimated probability distribution of possible values ​​using a first entropy encoder.

[0025] The exemplary method further includes decoding a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model. The method further includes generating a first resulting latent tensor using a first dequantizer. The method further includes converting the first resulting latent tensor into initial reconstructed data using a first neural decoder.

[0026] The method of the exemplary embodiment further includes using a second neural encoder to convert residual ground truth data into a second latent tensor. The method includes generating a second quantized latent tensor based at least on the second latent tensor using a second quantizer and a second pre-defined set of quantization levels, wherein the second quantized latent tensor includes at least one symbol or element. The method further includes using an auxiliary encoder to convert auxiliary data into auxiliary features. The method further includes inputting the auxiliary features into a second probability model. The method further includes using the second probability model to determine a second estimated probability distribution of possible values for each symbol or element of at least one symbol or element of the second quantized latent tensor. The method further includes using a second entropy encoder to encode each symbol or element of at least one symbol or element of the second quantized latent tensor into a second bit stream based at least on the second estimated probability distribution of possible values.

[0027] The method of the exemplary embodiment further includes decoding the second bit stream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probability model or another probability model identical to the second probability model. The method further includes using a second dequantizer to generate a second resulting latent tensor. The method further includes using a second neural decoder to convert the second resulting latent tensor into residual reconstruction data.

[0028] In one exemplary embodiment, at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder includes a neural network component.

[0029] In one exemplary embodiment, the first codec and the second codec are trained in an end-to-end fashion by reducing at least one of the distortion loss and the rate loss.

[0030] In one exemplary embodiment, the first codec is trained before the second codec, and the second codec is trained based at least in part on the first codec or on data generated by the first codec.

[0031] In one exemplary embodiment, the first codec and the second codec are trained simultaneously.

[0032] In one exemplary embodiment, the first codec and the second codec are trained at alternating intervals.

[0033] In one exemplary embodiment, the auxiliary data includes one or more of the initial reconstruction data, the first latent tensor, and the resulting first latent tensor.

[0034] In one exemplary embodiment, the residual ground truth data is determined by the first codec.

[0035] In one exemplary embodiment, the combined reconstruction data is determined by the second codec.

[0036] The exemplary method further includes determining additional residual ground truth data based on at least combined reconstructed data and ground truth data. The method further includes determining additional auxiliary data based at least on a second bitstream or at least on data derived from the second bitstream. The method further includes receiving the additional residual ground truth data and additional auxiliary data by a different codec. The method further includes generating an additional bitstream based at least on the additional residual ground truth data and additional auxiliary data. The method further includes generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. The method further includes outputting at least one of the additional reconstructed data and composite reconstructed data by a different codec.

[0037] One exemplary form provides a method that includes receiving a first bitstream. The method further includes generating initial reconstruction data based on the first bitstream. The method further includes determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The method further includes outputting the initial reconstruction data. The method further includes receiving a second bitstream. The method further includes generating residual reconstruction data based on the second bitstream and the auxiliary data. The method further includes outputting the residual reconstruction data.

[0038] The exemplary method further includes determining the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0039] In one exemplary form, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0040] The exemplary method further includes decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. The method further includes generating a first latent tensor using a first dequantizer and based on the first quantized latent tensor. The method further includes converting the first latent tensor into initial reconstructed data using a first neural decoder.

[0041] The exemplary method further includes decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. The method further includes generating a second latent tensor using a second dequantizer and based on the second quantized latent tensor. The method further includes converting the second latent tensor into residual reconstruction data using a second neural decoder.

[0042] In one exemplary form, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0043] The exemplary method further includes receiving an additional bitstream. The method further includes generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. The method further includes outputting at least one of the additional reconstructed data and the composite reconstructed data.

[0044] In one exemplary embodiment, a device is provided comprising at least one processor and at least one memory containing computer program code configured on the at least one processor to cause the device to receive ground truth data by a first codec. The at least one memory and computer program code are further configured to generate a first bitstream based on the ground truth data. The at least one memory and computer program code are further configured to generate initial reconstruction data based on the first bitstream, wherein the initial reconstruction data includes a reconstruction of the ground truth data. The at least one memory and computer program code are further configured to output the initial reconstruction data by a first codec. The at least one memory and computer program code are further configured to determine residual ground truth data based at least on the initial reconstruction data and the ground truth data. The at least one memory and computer program code are further configured to determine auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. At least one memory and computer program code are further configured to receive residual ground truth data and auxiliary data by a second codec. At least one memory and computer program code are further configured to generate a second bitstream based on the residual ground truth data and auxiliary data.

[0045] In one exemplary embodiment, at least one memory and computer program code are further configured to cause the device to generate residual reconstruction data based on a second bitstream, the residual reconstruction data including the reconstruction of residual ground truth data, using at least one processor. The at least one memory and computer program code are further configured to output the residual reconstruction data by a second codec.

[0046] In one exemplary embodiment, at least one memory and computer program code are further configured to cause the device to determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data, in at least one processor.

[0047] In one exemplary embodiment, at least one memory and computer program code are further configured to cause at least one processor to cause the device to determine a combined bitstream based at least on a first bitstream and a second bitstream.

[0048] In one exemplary form, residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

[0049] In one exemplary form, the ground truth data includes an image containing brightness data and color data.

[0050] In one exemplary embodiment, at least one memory and computer program code is further configured to cause the device to convert ground truth data into a first latent tensor using a first neural encoder, in at least one processor. The at least one memory and computer program code is further configured to generate a first quantized latent tensor based on at least a first latent tensor, using a first quantizer and a first predefined set of quantization levels, such that the first quantized latent tensor contains at least one symbol or element. The at least one memory and computer program code is further configured to determine a first estimated probability distribution of possible values ​​using a first probabilistic model for each of the at least one symbol or element of the first quantized latent tensor. At least one memory and computer program code are further configured to encode each symbol or element of at least one symbol or element of a first quantized latent tensor into a first bitstream based at least on a first estimated probability distribution of possible values, using a first entropy encoder.

[0051] In one exemplary embodiment, at least one memory and computer program code is further configured to cause the device to decode a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor, using at least one processor, a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model. At least one memory and computer program code is further configured to generate a first resulting latent tensor using a first dequantizer. At least one memory and computer program code is further configured to convert the first resulting latent tensor into initial reconstruction data using a first neural decoder.

[0052] In one exemplary form, at least one memory and computer program code is further configured to cause the device to convert residual ground truth data into a second latent tensor using a second neural encoder in at least one processor. At least one memory and computer program code is further configured to generate a second quantized latent tensor based on at least the second latent tensor using a second quantizer and a second predefined set of quantization levels, such that the second quantized latent tensor contains at least one symbol or element. At least one memory and computer program code is further configured to convert auxiliary data into auxiliary features using an auxiliary encoder. At least one memory and computer program code is further configured to input the auxiliary features into a second probabilistic model. At least one memory and computer program code is further configured to determine a second estimated probability distribution of possible values ​​using the second probabilistic model for each symbol or element of at least one symbol or element of the second quantized latent tensor. At least one memory and computer program code are further configured to encode each symbol or element of at least one symbol or element of a second quantized latent tensor into a second bitstream based at least on a second estimated probability distribution of possible values, using a second entropy encoder.

[0053] In one exemplary embodiment, at least one memory and computer program code is further configured to cause the device to decode a second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor, using at least one processor, a second entropy decoder, an auxiliary encoder, and a second probabilistic model or a copy of the second probabilistic model. At least one memory and computer program code is further configured to generate a second resulting latent tensor using a second dequantizer. At least one memory and computer program code is further configured to convert the second resulting latent tensor into residual reconstruction data using a second neural decoder.

[0054] In one exemplary embodiment, at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

[0055] In one exemplary form, the first and second codecs are trained in an end-to-end manner by reducing at least one of the distortion loss and rate loss.

[0056] In one exemplary form, the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

[0057] In one exemplary configuration, the first and second codecs are trained simultaneously.

[0058] In one exemplary configuration, the first and second codecs are trained at alternating intervals.

[0059] In one exemplary form, the auxiliary data includes one or more of the initial reconstruction data, the first latent tensor, and the first resulting latent tensor.

[0060] In one exemplary configuration, residual ground truth data is determined by a first codec.

[0061] In one example, the combined reconstructed data is determined by a second codec.

[0062] In one exemplary embodiment, at least one memory and computer program code are further configured to cause the device to determine additional residual ground truth data based at least on combined reconstructed data and ground truth data in at least one processor. At least one memory and computer program code are further configured to determine additional auxiliary data based at least on a second bitstream or at least on data derived from the second bitstream. At least one memory and computer program code are further configured to receive additional residual ground truth data and additional auxiliary data by different codecs. At least one memory and computer program code are further configured to generate an additional bitstream based at least on the additional residual ground truth data and additional auxiliary data. At least one memory and computer program code are further configured to generate additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. At least one memory and computer program code are further configured to output additional reconstructed data using different codecs.

[0063] In one exemplary embodiment, a device is provided comprising at least one processor and at least one memory containing computer program code configured on at least one processor to cause the device to receive a first bitstream. The at least one memory and computer program code are further configured to generate initial reconstruction data based on the first bitstream. The at least one memory and computer program code are further configured to determine auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The at least one memory and computer program code are further configured to output the initial reconstruction data. The at least one memory and computer program code are further configured to receive a second bitstream. The at least one memory and computer program code are further configured to generate residual reconstruction data based on the second bitstream and the auxiliary data. The at least one memory and computer program code are further configured to output the residual reconstruction data.

[0064] In one exemplary embodiment, at least one memory and computer program code are further configured to cause the device to determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data, in at least one processor.

[0065] In one exemplary form, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0066] In one exemplary embodiment, at least one memory and computer program code is further configured to cause the device to decode a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model in at least one processor. At least one memory and computer program code is further configured to generate a first latent tensor using a first dequantizer and based on the first quantized latent tensor. At least one memory and computer program code is further configured to convert the first latent tensor into initial reconstructed data using a first neural decoder.

[0067] In one exemplary embodiment, at least one memory and computer program code is further configured to cause the device to decode a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model, on at least one processor. At least one memory and computer program code is further configured to generate a second latent tensor using a second dequantizer and based on the second quantized latent tensor. At least one memory and computer program code is further configured to convert the second latent tensor into residual reconstruction data using a second neural decoder.

[0068] In one exemplary form, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0069] In one exemplary embodiment, at least one memory and computer program code are further configured to cause the device to receive an additional bitstream in at least one processor. The at least one memory and computer program code are further configured to generate additional reconstructed data based on the additional bitstream, such that the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. The at least one memory and computer program code are further configured to output the additional reconstructed data.

[0070] In one exemplary embodiment, a non-temporary computer-readable storage medium is provided, which includes a computer instruction that, when executed by the device, causes the device to receive ground truth data via a first codec. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to generate a first bitstream based on the ground truth data. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to generate initial reconstruction data based on the first bitstream, wherein the initial reconstruction data includes a reconstruction of the ground truth data. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to output the initial reconstruction data via a first codec. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to determine residual ground truth data based at least on the initial reconstruction data and the ground truth data. A non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine auxiliary data based at least on a first bitstream or at least on data derived from the first bitstream. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to receive residual ground truth data and auxiliary data by a second codec. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to generate a second bitstream based on the residual ground truth data and auxiliary data.

[0071] An exemplary non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to generate residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of residual ground truth data. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to output the residual reconstruction data by a second codec.

[0072] An exemplary non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0073] An exemplary non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine a combined bitstream based on at least a first bitstream and a second bitstream.

[0074] In one exemplary form, residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

[0075] In one exemplary form, the ground truth data includes an image containing brightness data and color data.

[0076] An exemplary non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to convert ground truth data into a first latent tensor using a first neural encoder. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to generate a first quantized latent tensor based on a first latent tensor using a first quantizer and a first predefined set of quantization levels, such that the first quantized latent tensor contains at least one symbol or element. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to determine a first estimated probability distribution of possible values ​​using a first probabilistic model for each of the at least one symbol or element of the first quantized latent tensor. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to encode each symbol or element of at least one symbol or element of a first quantized latent tensor into a first bitstream based at least on a first estimated probability distribution of possible values, using a first entropy encoder.

[0077] An exemplary non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to decode a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor, using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to generate a first resulting latent tensor using a first dequantizer. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to convert the first resulting latent tensor into initial reconstructed data using a first neural decoder.

[0078] An exemplary non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to convert residual ground truth data into a second latent tensor using a second neural encoder. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to generate a second quantized latent tensor based on a second latent tensor using a second quantizer and a second predefined set of quantization levels, such that the second quantized latent tensor contains at least one symbol or element. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to convert auxiliary data into auxiliary features using an auxiliary encoder. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to input the auxiliary features into a second probabilistic model. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine a second estimated probability distribution of possible values ​​for each of the at least one symbols or elements of a second quantized latent tensor, using a second probabilistic model. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to encode each of the at least one symbols or elements of the second quantized latent tensor into a second bitstream based at least on the second estimated probability distribution of possible values, using a second entropy encoder.

[0079] An exemplary non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to decode a second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model or a copy of the second probabilistic model. The non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to generate a second resulting latent tensor using a second dequantizer. The non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to convert the second resulting latent tensor into residual reconstruction data using a second neural decoder.

[0080] In one exemplary embodiment, at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

[0081] In one exemplary form, the first and second codecs are trained in an end-to-end manner by reducing at least one of the distortion loss and rate loss.

[0082] In one exemplary form, the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

[0083] In one exemplary configuration, the first and second codecs are trained simultaneously.

[0084] In one exemplary configuration, the first and second codecs are trained at alternating intervals.

[0085] In one exemplary form, the auxiliary data includes one or more of the initial reconstruction data, the first latent tensor, and the first resulting latent tensor.

[0086] In one exemplary configuration, residual ground truth data is determined by a first codec.

[0087] In one example, the combined reconstructed data is determined by a second codec.

[0088] An exemplary non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine additional residual ground truth data based at least on combined reconstructed data and ground truth data. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine additional auxiliary data based at least on a second bitstream or at least on data derived from the second bitstream. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to receive additional residual ground truth data and additional auxiliary data by different codecs. The non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to generate additional bitstreams based at least on the additional residual ground truth data and additional auxiliary data. The non-temporary computer-readable storage medium further includes computer instructions configured to, when executed, generate additional reconstructed data based on additional bitstreams, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with combined reconstructed data to form composite reconstructed data. The non-temporary computer-readable storage medium further includes computer instructions configured to, when executed, output the additional reconstructed data by different codecs.

[0089] In one exemplary embodiment, a non-temporary computer-readable storage medium is provided, which includes a computer instruction that, when executed by the device, causes the device to receive a first bitstream. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to generate initial reconstruction data based on the first bitstream. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to determine auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to output the initial reconstruction data. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to receive a second bitstream. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to generate residual reconstruction data based on the second bitstream and the auxiliary data. The non-temporary computer-readable storage medium further includes a computer instruction configured, when executed, to output the residual reconstruction data.

[0090] An exemplary non-temporary computer-readable storage medium further includes computer instructions configured, when executed, to determine combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0091] In one exemplary form, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0092] An exemplary non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to decode a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to generate a first latent tensor using a first dequantizer and based on the first quantized latent tensor. The non-temporal computer-readable storage medium further includes a computer instruction configured, when executed, to convert the first latent tensor into initial reconstructed data using a first neural decoder.

[0093] An exemplary non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to decode a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. The non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to generate a second latent tensor using a second dequantizer and based on the second quantized latent tensor. The non-temporal computer-readable storage medium further includes computer instructions configured, when executed, to convert the second latent tensor into residual reconstruction data using a second neural decoder.

[0094] In one exemplary form, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0095] An exemplary non-temporary computer-readable storage medium further includes a computer instruction configured to receive an additional bitstream when executed. The non-temporary computer-readable storage medium further includes a computer instruction configured to generate additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. The non-temporary computer-readable storage medium further includes a computer instruction configured to output additional reconstructed data when executed.

[0096] In one exemplary embodiment, an apparatus is provided comprising means for receiving ground truth data by a first codec. The apparatus further includes means for generating a first bitstream based on the ground truth data. The apparatus further includes means for generating initial reconstructed data based on the first bitstream, wherein the initial reconstructed data includes a reconstruction of the ground truth data. The apparatus further includes means for outputting the initial reconstructed data by the first codec. The apparatus further includes means for determining residual ground truth data based at least on the initial reconstructed data and the ground truth data. The apparatus further includes means for determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The apparatus further includes means for receiving residual ground truth data and auxiliary data by a second codec. The apparatus further includes means for generating a second bitstream based on the residual ground truth data and auxiliary data.

[0097] The exemplary apparatus includes means for generating residual reconstruction data based on a second bitstream, further including means for generating residual reconstruction data which includes reconstruction of residual ground truth data. The apparatus further includes means for outputting the residual reconstruction data by a second codec.

[0098] The exemplary apparatus further includes means for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0099] The exemplary apparatus further includes means for determining a combined bitstream based at least on a first bitstream and a second bitstream.

[0100] In one exemplary form, residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

[0101] In one exemplary form, the ground truth data includes an image containing brightness data and color data.

[0102] The exemplary apparatus further includes means for converting ground truth data into a first latent tensor using a first neural encoder. The apparatus further includes means for generating a first quantized latent tensor based on a first latent tensor using a first quantizer and a first predefined set of quantization levels, wherein the first quantized latent tensor includes at least one symbol or element. The apparatus further includes means for determining a first estimated probability distribution of possible values ​​using a first probabilistic model for each of the at least one symbol or element of the first quantized latent tensor. The apparatus further includes means for encoding each of the at least one symbol or element of the first quantized latent tensor into a first bitstream based at least on the first estimated probability distribution of possible values ​​using a first entropy encoder.

[0103] The exemplary apparatus further includes means for decoding a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor, using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model. The apparatus further includes means for generating a first resulting latent tensor using a first dequantizer. The apparatus further includes means for converting the first resulting latent tensor into initial reconstructed data using a first neural decoder.

[0104] The exemplary apparatus further includes means for converting residual ground truth data into a second latent tensor using a second neural encoder. The apparatus further includes means for generating a second quantized latent tensor based on a second latent tensor using a second quantizer and a second predefined set of quantization levels, wherein the second quantized latent tensor includes at least one symbol or element. The apparatus further includes means for converting auxiliary data into auxiliary features using an auxiliary encoder. The apparatus further includes means for inputting the auxiliary features into a second probabilistic model. The apparatus further includes means for determining a second estimated probability distribution of possible values ​​using the second probabilistic model for each symbol or element of the at least one symbol or element of the second quantized latent tensor. The apparatus further includes means for encoding each symbol or element of at least one symbol or element of a second quantized latent tensor into a second bitstream based at least on a second estimated probability distribution of possible values, using a second entropy encoder.

[0105] The exemplary apparatus further includes means for decoding a second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probabilistic model or another probabilistic model identical to the second probabilistic model. The apparatus further includes means for generating a second resulting latent tensor using a second dequantizer. The apparatus further includes means for converting the second resulting latent tensor into residual reconstruction data using a second neural decoder.

[0106] In one exemplary embodiment, at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

[0107] In one exemplary form, the first and second codecs are trained in an end-to-end manner by reducing at least one of the distortion loss and rate loss.

[0108] In one exemplary form, the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

[0109] In one exemplary configuration, the first and second codecs are trained simultaneously.

[0110] In one exemplary configuration, the first and second codecs are trained at alternating intervals.

[0111] In one exemplary form, the auxiliary data includes one or more of the initial reconstruction data, the first latent tensor, and the first resulting latent tensor.

[0112] In one exemplary configuration, residual ground truth data is determined by a first codec.

[0113] In one example, the combined reconstructed data is determined by a second codec.

[0114] The exemplary apparatus further includes means for determining additional residual ground truth data based at least on combined reconstructed data and ground truth data. The apparatus further includes means for determining additional auxiliary data based at least on a second bitstream or at least on data derived from the second bitstream. The apparatus further includes means for receiving the additional residual ground truth data and additional auxiliary data by different codecs. The apparatus further includes means for generating an additional bitstream based at least on the additional residual ground truth data and additional auxiliary data. The apparatus further includes means for generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to form composite reconstructed data in combination with the combined reconstructed data. The apparatus further includes means for outputting at least one of the additional reconstructed data and composite reconstructed data by different codecs.

[0115] In one exemplary embodiment, an apparatus is provided comprising means for receiving a first bitstream. The apparatus further includes means for generating initial reconstruction data based on the first bitstream. The apparatus further includes means for determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream. The apparatus further includes means for outputting the initial reconstruction data. The apparatus further includes means for receiving a second bitstream. The apparatus further includes means for generating residual reconstruction data based on the second bitstream and the auxiliary data. The apparatus further includes means for outputting the residual reconstruction data.

[0116] The exemplary apparatus further includes means for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0117] In one exemplary form, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0118] The exemplary apparatus further includes means for decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. The apparatus further includes means for generating a first latent tensor using a first dequantizer and based on the first quantized latent tensor. The apparatus further includes means for converting the first latent tensor into initial reconstructed data using a first neural decoder.

[0119] The exemplary apparatus further includes means for decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. The apparatus further includes means for generating a second latent tensor using a second dequantizer and based on the second quantized latent tensor. The apparatus further includes means for converting the second latent tensor into residual reconstruction data using a second neural decoder.

[0120] In one exemplary form, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0121] The exemplary apparatus further includes means for receiving an additional bitstream. The apparatus further includes means for generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. The apparatus further includes means for outputting at least one of the additional reconstructed data and the composite reconstructed data.

[0122] As described above, specific exemplary forms of this disclosure have been explained using general terminology, but references to the attached drawings will be made below, and the attached drawings are not necessarily drawn to scale. [Brief explanation of the drawing]

[0123] [Figure 1] This is a block diagram of an example communication system in which the exemplary forms of this disclosure may be implemented. [Figure 2] This is a block diagram of a system, as illustrated in this specification, which includes both service devices and user devices designed to perform functions. [Figure 3] This is a diagram of a codec that uses a neural network as a component of the pipeline according to a prior embodiment. [Figure 4A] This is a diagram of a video coding pipeline utilizing a neural network according to a prior embodiment. [Figure 4B] This is a diagram of a video coding pipeline that utilizes a neural network on the encoding and decoding sides, according to a prior embodiment. [Figure 5] This is a diagram of a neural network-based end-to-end trained video coding system according to a prior embodiment. [Figure 6A] This is a diagram of a VCM pipeline according to a prior embodiment. [Figure 6B] This is a diagram of a VCM pipeline using an end-to-end pre-trained approach according to a prior embodiment. [Figure 7] This is a diagram illustrating the training of an end-to-end trained system according to a prior embodiment. [Figure 8] This is a diagram of an example of a high-density split attention block according to the exemplary embodiments of this disclosure. [Figure 9] This is a diagram of an end-to-end trained intraframe codec according to the exemplary embodiment of this disclosure. [Figure 10]This is a block diagram of a system with three or more codecs, as illustrated in the present disclosure. [Figure 11] This flowchart shows the operations performed by the device shown in Figure 2 to generate a bitstream based on ground truth data. [Figure 12] This flowchart shows the operations performed by the device shown in Figure 2 to encode the first bitstream. [Figure 13] This flowchart shows the operations performed by the device shown in Figure 2 to generate the initial reconstruction data. [Figure 14] This flowchart shows the operations performed by the device shown in Figure 2 to encode the second bitstream. [Figure 15] This flowchart shows the operations performed by the apparatus shown in Figure 2 to generate residual reconstruction data. [Figure 16] This flowchart shows the operations performed by the device shown in Figure 2 to output additional reconstruction data. [Figure 17] This flowchart shows the operations performed by the device shown in Figure 2 to output residual reconstruction data. [Figure 18] This flowchart shows the operations performed by the device shown in Figure 2 to generate the initial reconstruction data. [Figure 19] This flowchart shows the operations performed by the apparatus shown in Figure 2 to generate residual reconstruction data. [Figure 20] This flowchart shows the operations performed by the device shown in Figure 2 to output additional reconstruction data. [Modes for carrying out the invention]

[0124] While some, but not all, embodiments of the present invention will be described more fully below with reference to the accompanying drawings, some embodiments of the present invention will be described more fully hereby. In practice, various embodiments of the present invention may be embodied in many different forms and should not be construed as being limited to the embodiments described herein, but rather these embodiments are provided so that this disclosure satisfies applicable legal requirements. Similar reference numbers refer to similar elements throughout. As used herein, the terms “data,” “content,” “information,” and similar terms may be used without distinction to refer to data that has the ability to be transmitted, received, and / or stored by embodiments of the present invention. Accordingly, the use of any such terms should not be taken as limiting the spirit and scope of embodiments of the present invention.

[0125] The terms "first," "second," etc., may be used herein to describe various elements, but these elements should not be limited by these terms. These terms are used solely to distinguish one element from another. For example, without departing from the scope of this disclosure, the first element may be referred to as the second element, and similarly, the second element may be referred to as the first element. As used herein, the terms "and / or" include any and all combinations of one or more of the associated listed items.

[0126] When an element is referred to as being "connected" or "linked" to another element, it can be directly connected or linked to the other element, or an intermediary element may exist. In contrast, when an element is referred to as being "directly connected" or "directly linked" to another element, no intermediary element exists. Other words used to describe the relationship between elements should be interpreted similarly (e.g., "between" vs. "directly between," "adjacent" vs. "directly adjacent," etc.).

[0127] The technical terms used herein are intended to describe only specific embodiments and are not intended to limit them. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural unless the context otherwise explicitly indicates. It will be further understood that the terms “equip,” “equip,” “include,” and / or “include,” as used herein, specify the presence of the described feature, integer, step, action, element, and / or component, but do not exclude the presence or addition of one or more other features, integers, steps, actions, elements, components, and / or groups thereof.

[0128] It should also be noted that in some alternative implementations, the described functions / actions may occur in a different order than that shown in the diagrams. For example, two diagrams shown consecutively may actually be executed simultaneously, or sometimes in reverse order depending on the functions / actions they contain.

[0129] Certain details are provided in the following description to provide a complete understanding of the exemplary forms. Nevertheless, it will be understood by those skilled in the art that the exemplary forms can be practiced without these specific details. For example, a system may be shown in a block diagram so as not to obscure the exemplary form with unnecessary details. In other examples, well-known processes, structures, and techniques may be shown without unnecessary details to avoid obscuring the exemplary forms.

[0130] Furthermore, as used herein, the term “circuit” may refer to one or more or all of the following: (a) a hardware-only circuit implementation (such as an implementation in an analog circuit and / or digital circuit); (b) a combination of circuit and software, etc. (where applicable): (i) a combination of analog and / or digital hardware circuitry and software / firmware; and (ii) any part of a hardware processor with software (including digital signal processors, software, and memory that work together to perform various functions in a device such as a mobile phone or server); and (c) a hardware circuit and / or processor such as a microprocessor or a part of a microprocessor that requires software (e.g., firmware) for operation, but may not be present when not required for operation. This definition of “circuit” applies to all use of the term in this application, including in any claim. As a further example, as used in this application, the term “circuit” also includes a mere hardware circuit or processor (or a number of processors), or a part of a hardware circuit or processor and any software and / or firmware attached thereto. The term “circuit” includes, for example, and where applicable to the specific claim, baseband integrated circuits, or processor integrated circuits for mobile phones, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0131] Furthermore, as used herein, the terms “model,” “neural network,” “neural net,” and “network” are used interchangeably. Additionally, the weights of a neural network may be referred to as learnable parameters or parameters.

[0132] Furthermore, as used herein, the terms “machine” and “task neural network” are used interchangeably and mean any process or algorithm (whether learned from the data or not) that analyzes or processes data for a particular task.

[0133] Furthermore, as used herein, the terms “receiver side” and “decoder side” refer to a physical or abstract entity or device which may include one or more machines that may perform some encoding and ultimately decoding on a video representation, which is encoded by another physical or abstract entity or device, the “encoder side device.”

[0134] Furthermore, as used herein, the terms “intraframe,” “frame,” and “image” may be used interchangeably. These terms may refer to at least a portion of the input data and at least a portion of the output data of an end-to-end trained intraframe codec. In one or more embodiments, these terms refer to an image as a data type. Nevertheless, the proposed embodiments can be extended to other types of data, such as video and audio.

[0135] As defined herein, “computer-readable storage medium” can be distinguished from “computer-readable transmission medium” which refers to a physical storage medium (e.g., a volatile or non-volatile memory device) as defined herein.

[0136] The following description will illustrate exemplary embodiments with reference to the actions and symbolic representations of behavior (in the form of flowcharts, flow diagrams, data flow diagrams, structural diagrams, block diagrams, etc.) that may be executed as program modules or functional processes, including routines, programs, objects, components, data structures, etc., that may be executed using existing hardware in existing network elements to perform specific tasks or implement specific abstract data types. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays (FPGAs), computers, or similar.

[0137] While flowcharts can describe operations as sequential processes, many operations may be performed in parallel, simultaneously, or all at once. Furthermore, the order of operations may be rearranged. A process may terminate when its operations are complete, but it may also include additional steps not shown in the diagram. A process can correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, its termination may correspond to the function returning to the calling function or the main function.

[0138] As disclosed herein, the terms “storage medium” or “computer-readable storage medium” can represent one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, magnetic disk storage medium, optical storage medium, flash memory devices, and / or other tangible machine-readable media for storing information. The terms “computer-readable medium” may include, but are not limited to, portable or fixed storage devices, optical storage devices, and various other media capable of storing, containing, or transporting instructions and / or data.

[0139] Furthermore, the exemplary forms can be implemented by hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof. When implemented by software, firmware, middleware, or microcode, the program code or code segments for performing the required tasks may be stored on a machine or computer-readable medium, such as a computer-readable storage medium. When executed by software, one or more processors will perform the required tasks.

[0140] A code segment can represent any combination of a procedure, function, subprogram, program, routine, subroutine, module, software package, class, or instruction, data structure, or program statement. A code segment can be linked to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, transferred, or transmitted via any appropriate means, including memory sharing, message passing, token passing, network transmission, etc.

[0141] Examples of these applications may be used with RANs such as Universal Mobile Telecommunications System (UMTS), Global System for Mobile Communications (GSM), Advanced Mobile Phone Services (AMPS) systems, Narrowband AMPS systems (NAMPS), Total Access Communications System (TACS), Personal Digital Cellular (PDC) systems, U.S. Digital Cellular (USDC) systems, Code Division Multiple Access (CDMA) systems as described in EIA / TIA IS-95, High Rate Packet Data (HRPD) systems, Worldwide Interoperability for Microwave Access (WiMAX), Ultra Mobile Broadband (UMB), and Third Generation Partnership Project LTE (3GPP LTE).

[0142] Methods, apparatus, and computer program products for video and image compression and reconstruction using end-to-end trained coder-decoder are provided as described herein.

[0143] To perform end-to-end learned compression and reconstruction, an apparatus 10, such as the one shown in Figure 1, is provided. The apparatus may be embodied by or communicate with any of the various different types of computing devices, including, for example, a video processing system, an image processing system, or any other system configured to decompress images captured by a snapshot compression detection system. As shown in Figure 1, the exemplary apparatus includes, is associated with, or otherwise communicates with a processor 12, associated memory 14, and a communication interface 16.

[0144] The processor 12 (and / or a coprocessor or any other circuit that assists the processor or is otherwise associated with it) may communicate with the memory device 14 via a bus to pass information between components of the device 10. The memory device may be non-temporary and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer-readable storage medium) having gates configured to store data (e.g., bits) that may be retrieved by a machine (e.g., a computing device such as a processor). The memory device may be configured to store information, data, content, applications, instructions, or the like to enable the device to perform various functions as illustrated in the embodiments of this disclosure. For example, the memory device may be configured to buffer input data for processing by the processor. Additionally or alternatively, the memory device may be configured to store instructions for execution by the processor.

[0145] In some embodiments, the device 10 may be embodied in various computing devices as described above. Nevertheless, in some embodiments, the device may be embodied as a chip or chipset. In other words, the device may comprise one or more physical packages (e.g., a chip) including materials, components, and / or wires on a structural assembly (e.g., a baseboard). The structural assembly can provide physical strength, size preservation, and / or limitations on electrical interaction of the components included in the structural assembly. The device may, in some cases, be configured to implement embodiments of the present invention on a single chip or as a single "system on a chip". Thus, in some cases, the chip or chipset may constitute means for performing one or more operations to provide the functions described herein.

[0146] The processor 12 can be embodied in several different ways. For example, the processor can be embodied as one or more of various hardware processing means, such as a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an associated DSP, or various other circuits including integrated circuits such as ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), microcontroller units (MCUs), hardware accelerators, special-purpose computer chips, or similar. Thus, in some embodiments, the processor can include one or more processing cores configured to operate independently. A multicore processor can enable multiprocessing within a single physical package. As an addition or alternative, the processor can include one or more processors configured in tandem via a bus to enable independent instruction execution, pipeline, and / or multithreading.

[0147] In one exemplary embodiment, the processor 12 may be configured to execute instructions stored in the memory device 14, or otherwise accessible to the processor. Alternatively or additionally, the processor may be configured to perform hardcoded functions. Thus, whether configured by a hardware-based method, a software-based method, or a combination thereof, the processor may represent an entity (e.g., physically embodied in a circuit) that is configured as appropriate to perform the operations according to embodiments of the present disclosure. Thus, for example, when the processor is embodied as an ASIC, FPGA, or similar, the processor may also be specifically configured hardware for performing the operations described herein. Alternatively, as another example, when the processor is embodied as an instruction executor, the instructions may specifically configure the processor to perform the algorithms and / or operations described herein when the instructions are executed. Nevertheless, in some cases, the processor may also be a processor in a particular device (e.g., an image processing system) configured to employ embodiments of the present invention by further configuration of the processor with instructions for performing the algorithms and / or operations described herein. The processor may include, among other things, a clock, an arithmetic logic unit (ALU), and logic gates configured to support the operation of the processor.

[0148] The communication interface 16 may be any means such as a device or circuit embodied in hardware or a combination of hardware and software configured to receive and / or transmit data, such as by receiving frames from a snapshot compression sensing system or an external memory device, and / or by providing reconstruction signals to an imaging system or other type of display for representation or to the external memory device for storage. In this regard, the communication interface may include, for example, an antenna (or a number of antennas) and supporting hardware and / or software to enable communication with a wireless communication network. Additionally or alternatively, the communication interface may include circuitry for interacting with an antenna to cause the transmission of a signal via the antenna or to handle the reception of a signal received via the antenna. In some environments, the communication interface may, or alternatively, support wired communication. Thus, for example, the communication interface may include a communication modem and / or other hardware / software to support communication via cable, digital subscriber line (DSL), universal serial bus (USB), or other mechanisms.

[0149] The processor 12 may be configured to execute instructions of a computer program by performing arithmetic operations, logical operations, and input / output operations of the system. Instructions may be provided to the processor 12 by memory 14.

[0150] The various interfaces of device 10 may include components that interface the processor 12 with an antenna or other input / output components. Naturally, the interfaces and programs stored in memory 14 for defining the special-purpose functions of device 10 will vary depending on the implementation of device 10.

[0151] In one exemplary embodiment, the device 10 may be any known or to be developed device, including but not limited to portable media devices such as cellular phones, laptops, tablets, personal computers, and television sets, as well as multifunction cameras, drones, electric vehicles, and the like.

[0152] Video and image compression codecs are devices or computer programs that encode and / or decode digital data streams, bitstreams, sequences of pictures, signals, or similar related to video and / or images. Still image codecs may conform to standards such as JPEG, GIF, PNG, or similar. Video codecs may conform to standards such as Cinepak, MPEG, MPEG-2, H.264, VP8, H.265, or similar.

[0153] Moving on to Figure 2, an exemplary system for performing an end-to-end codec is provided. In this exemplary configuration, the service device 22 can encode an input image or video (encoding may also include decoding). In one or more embodiments, the service device 22 can transmit the encoded bitstream data resulting from the input image to a consumer device 24. In one exemplary configuration, the bitstream data may be transmitted wirelessly over a network. In one exemplary configuration, the bitstream data may be transmitted through a wired connection between the service device 22 and the consumer device 24. In one or more embodiments, the consumer device 24 can decode the bitstream data received from the service device 22 into one or more reconstructed images or videos. In one or more embodiments, the service device 22 can store the bitstream data in a memory 14. The memory 14 may be located inside or outside the service device 22. In one or more embodiments, the service device 22 can retrieve the bitstream data from the memory 14 and reconstruct the input image. In one or more embodiments, the service device 22 and / or consumer device 24 may be embodied by the apparatus 10.

[0154] Moving on to Figure 3, examples of prior embodiments of codec 300 using neural networks as part of the pipeline are provided. In one or more embodiments, the in-loop filter 310 may include a neural network. In one example, the in-loop filter 310 includes one or more neural network-based in-loop filters and one or more non-neural network-based in-loop filters. In another example, the in-loop filter 310 includes one or more neural network-based in-loop filters. In one or more embodiments, the intra-frame prediction 320 may include one or more neural networks. In one or more embodiments, the inter-frame prediction 330 may include one or more neural networks. In one or more embodiments, the transform and / or inverse transform 340 may include one or more neural networks. In one or more embodiments, the reversible coding (e.g., entropi coding) 350 may include or use a neural network-based probabilistic model. Other neural networks may be used throughout the pipeline in various embodiments.

[0155] Moving on to Figure 4A, an illustration of a preceding embodiment of the video coding pipeline 400 is provided, in which the main components are replaced by neural networks. The video coding pipeline 400 uses an end-to-end trained compression approach with neural networks. Figure 4B illustrates an exemplary pipeline 410 that uses neural networks on both the encoder and decoder sides. In one or more embodiments, the pipeline 410 includes an analysis network 420, quantization means and arithmetic encoders and decoders in block 430, an arithmetic decoder 440, and a synthesis network 450. In one or more embodiments, the analysis network 420 includes an encoder neural network, and the synthesis network 450 includes a decoder neural network. In one or more embodiments, the analysis network 420 and the synthesis network 450 are part of a neural autoencoder architecture. In one or more embodiments, the analysis network 420 is configured to perform a nonlinear transform, and the synthesis network 450 is configured to perform a nonlinear inverse transform.

[0156] In one or more embodiments, input data is analyzed by an analysis network 420, which outputs a new representation of the input data. The new representation may be more compressible. This new representation can then be quantized to discrete numerical values ​​in block 430. The quantized data is then reversibly encoded, for example, by an arithmetic encoder in block 430, thus obtaining a bitstream. On the decoding side, the bitstream is first reversibly decoded, for example, by using an arithmetic decoder 440. The reversibly decoded data is dequantized and then input to a synthesis network 450. The output is reconstructed or decoded data.

[0157] In one or more embodiments in which irreversible compression is performed, the irreversible step may include quantization in the analysis network 420 and block 430.

[0158] In one exemplary form, a training objective function (also called the "training loss") is commonly used to train pipeline 410, and the training objective function may include one or more terms, or a loss term, or simply a loss. In one exemplary form, the training loss includes a reconstruction loss term and a rate loss term. In one exemplary form, the reconstruction loss prompts the system to decode data similar to the input data according to some similarity metric. Examples of reconstruction losses include mean squared error (MSE), multiscale structural similarity (MS-SSIM), loss induced from the use of a pre-trained network, loss induced from the use of a neural network trained simultaneously with an end-to-end pre-trained codec, or similar. An example of loss induced from the use of a pre-trained network is error(f1, f2), where f1 and f2 are features extracted by the pre-trained neural network for the input data and the decoded data, respectively, and error() is an error or distance function such as the L1 norm or L2 norm. An example of loss induced from the use of a neural network simultaneously trained on an end-to-end pre-trained codec is adversarial loss, which is the loss provided by a discriminator neural network trained adversarially on a codec according to the proposed configuration in the context of generative adversarial networks (GANs) and their variants.

[0159] Rate loss can encourage the system to compress (i.e., reduce the number of bits) the output of an encoding stage, such as the output of an arithmetic encoder. In one or more embodiments, when an entropy-based reversible encoder (e.g., an arithmetic encoder) is used, rate loss encourages the output of the analysis network 420 to have low entropy. Examples of rate loss include a differentiable estimate of the entropy, sparsification loss (i.e., a loss that encourages the output of the analysis network 420 or the quantization output to have many zeros, such as the L0 norm, L1 norm, L1 norm divided by the L2 norm), cross-entropy loss applied to the output of a probabilistic model, or similar, where the probabilistic model may be a neural network used to estimate the probability of the next symbol to be encoded by the arithmetic encoder.

[0160] In one or more embodiments, one or more of the reconstruction losses and one or more of the rate losses may be used as a weighted sum. In one or more embodiments, different loss terms are weighted using different weights, which determine how the final system performs in terms of rate distortion loss. For example, if more weight is given to the reconstruction loss than to the rate loss, the system may learn to reconstruct with less compression but with higher accuracy (such as measured by a metric that shows the correlation with the reconstruction loss). In one or more embodiments, these weights are hyperparameters of the training session and may be set manually by the person designing the training session or automatically, for example, by grid search or by using an additional neural network.

[0161] In one or more embodiments, a non-neural network, such as an arithmetic codec, may be used in an end-to-end trained approach.

[0162] Moving on to Figure 5, a neural network-based end-to-end pre-trained video coding system 500 according to a prior embodiment is provided. In one or more embodiments, the system 500 includes an encoder 510, a quantizer 520, a probabilistic model 530, an entropy codec (arithmetic encoder 540 and arithmetic decoder 550), a dequantizer 560, and a decoder 570. In one or more embodiments, the encoder 510 and decoder 570 are two neural networks. In one or more embodiments, the encoder 510 and decoder 570 mainly include neural network components. In one or more embodiments, the probabilistic model 530 mainly includes neural network components. In one or more embodiments, the quantizer 520, dequantizer 560, and entropy codec are not based on neural network components. In one or more alternative embodiments, these components include neural network components.

[0163] In one or more embodiments, the encoder 510 takes video as input and converts the video from its original signal space into a latent representation (also called a latent tensor) which may contain a more compressible representation of the input. In one or more embodiments, for an input image, the latent representation may be a three-dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represents the “channels” containing information at this particular location. In one or more embodiments, where the input image is a 128×128×3RGB image (having a horizontal size of 128 pixels, a vertical size of 128 pixels, and three channels for red, green, and blue color components), and the encoder 510 downsamples the input tensor by 2, expanding the channel dimension to 32 channels, the latent representation includes a tensor with dimensions (or “shape”) of 64×64×32 (i.e., having a horizontal size of 64 elements, a vertical size of 64 elements, and 32 channels). In various embodiments, the order of the different dimensions may vary depending on the convention used. In one exemplary form, for an input image, the channel dimension may be the first dimension. In this example, the above shape of the input tensor may be represented as 3 × 128 × 128 instead of 128 × 128 × 3. In one exemplary form, for an input video, another dimension in the input tensor may be used to represent temporal information. In one exemplary form, the quantizer 520 quantizes the latent representation into discrete values ​​given a predefined set of quantization levels. The output of the quantizer 520 may be called the quantized latent tensor. In one or more embodiments, the probabilistic model 530 and the arithmetic codec components work together to perform lossless compression for the quantized latent representation and generate a bitstream that will be sent to the decoder. Given a symbol to be encoded in the bitstream, the probabilistic model 530 estimates the probability distribution of all possible values ​​for this symbol based on the context constructed from available information of the currently encoded / decoded state, such as already encoded / decoded data. In one or more embodiments, the arithmetic encoder 540 encodes the input symbols into a bitstream using an estimated probability distribution.

[0164] In one or more embodiments, the reverse operation is performed on the decoder side. In one or more embodiments, the arithmetic decoder 550 and the probabilistic model 530 first decode symbols from the bitstream to reconstruct the quantized latent representation. In one or more embodiments, the dequantizer 560 then reconstructs the latent representation with continuous values ​​and passes this to the decoder 570 to reconstruct the input video / image. In one or more embodiments, the probabilistic model 530 is shared between the encoding system and the decoding system. In the exemplary embodiment, a copy of the probabilistic model is used on the encoder side and another exact copy is used on the decoder side.

[0165] In one exemplary form of system 500, the encoder 510, the probabilistic model 530, and the decoder 570 are based on a deep neural network. In one exemplary form, the system is trained in an end-to-end manner by minimizing the following rate-distortion loss function: L = D + λR. In one exemplary form, D is the distortion loss term, R is the rate loss term, and λ is a weight that controls the balance between the two losses. In one exemplary form, the distortion loss term may be the mean squared error (MSE), structural similarity (SSIM), or other metric that evaluates the quality of the reconstructed video. In one or more embodiments, a number of distortion losses, such as a weighted sum of MSE and SSIM, may be used and integrated into D. In one or more embodiments, the rate loss term is the estimated entropy of the quantized latent representation, which indicates the number of bits required to represent the encoded symbol, such as bits per pixel (bpp).

[0166] In one exemplary system for lossless video / image compression, the system may include only a probabilistic model 530, an arithmetic encoder 540, and an arithmetic decoder 550. In one or more embodiments, the system loss function includes only rate loss, since distortion loss is always zero (i.e., there is no loss of information).

[0167] In one or more embodiments, the decoded data may be analyzed by a machine. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, and pedestrian tracking. Illustrative use cases and applications include autonomous vehicles, video surveillance cameras and public security, smart sensor networks, smart TVs and smart advertising, person re-identification, smart traffic monitoring, drones, and / or similar. In one or more embodiments, quality metrics and dedicated algorithms may be used to compress and decompress data for machine use, distinct from those used for compressing and decompressing data for human use. The set of tools and concepts for compressing and decompressing data for machine use is referred to here as machine video coding.

[0168] In one or more embodiments, the receiver-side device has a number of “machines” or neural networks (NN). In one or more embodiments, these machines may be used in specific combinations determined by an orchestrator subsystem. In one or more embodiments, the machines may be used sequentially and / or in parallel based on the output of previously used machines. For example, compressed and then decompressed video may be analyzed by one machine (NN) to detect pedestrians, by another machine (another NN) to detect cars, and by another machine (another NN) to estimate the depth of all pixels in the frame.

[0169] In one or more embodiments, the encoded video data may be stored in a memory device, for example, as a file. In one or more embodiments, the stored file may later be provided to another device. In one or more alternative embodiments, the encoded video data may be streamed from one device to another.

[0170] Moving on to Figure 6A, an illustration of a machine video coding pipeline 600 according to a preceding embodiment is provided. In one or more embodiments, a VCM encoder 610 encodes an input video into a bitstream. In one or more embodiments, the bitrate may be calculated from the bitstream to evaluate the size of the bitstream. In one or more embodiments, a VCM decoder 620 decodes the bitstream output by the VCM encoder 610. In one or more embodiments, the output from the VCM decoder 620 contains machine-decoded data. In one or more embodiments, this data may be decoded or reconstructed video. However, in one or more embodiments, this data may not have the same or similar characteristics as the original video input to the VCM encoder 610. For example, this data may not be readily understandable to humans by simply depicting the data on a screen. In one or more embodiments, the output of the VCM decoder 620 is then input to one or more task neural networks 631-63N. In one or more embodiments, any number of task neural networks may be present. In one or more embodiments, the VCM600 aims to obtain a low bitrate while ensuring that tasks NN631-63N perform well in terms of evaluation metrics associated with each task.

[0171] In one or more embodiments, an end-to-end trained approach may be used to implement machine video coding. In one or more embodiments, the VCM encoder 610 and VCM decoder 620 consist primarily of neural networks. Figure 6B illustrates an example of a pipeline for an end-to-end trained approach according to a prior embodiment. In one or more embodiments, video is input to a neural network encoder 650. In one or more embodiments, the output of the neural network encoder is input to a reversible encoder 670, such as an arithmetic encoder, which outputs a bitstream. In one or more embodiments, the reversible codec may include a probabilistic model 640 in both the reversible encoder 670 and the reversible decoder 680, which predicts the probability of the next symbol to be encoded and decoded. In one or more embodiments, the probabilistic model 640 may be trained, for example, a neural network. In one or more embodiments, the bitstream is input to a reversible decoder 680, such as an arithmetic decoder, on the decoder side, and the output of the reversible decoder 680 is input to a neural network decoder 660. In one or more embodiments, the output of the neural network decoder 660 is machine-decoded data, which can be input to one or more tasks NN631-63N.

[0172] Moving on to Figure 7, we see an example of how an end-to-end trained system 700 can be trained according to a preceding embodiment. For brevity, only one task NN631 is illustrated, but any number of task NNs may be used. In one or more embodiments, the rate loss can be calculated from the output of the probabilistic model 640. In one or more embodiments, the rate loss provides an approximation of the bitrate required to encode the input video data. In one or more embodiments, the task loss can be calculated from the output of task NN631.

[0173] In one or more embodiments, the rate loss and task loss may then be used to train a neural network used in a system such as a neural network encoder 650, a probabilistic model 640, or a neural network decoder 660. In one or more embodiments, training may be carried out by first calculating the gradient for each loss with respect to the neural network that contributes to or influences the calculation of this loss. In one or more embodiments, the gradient may then be used by an optimization method such as Adam to update the trainable parameters of the neural network.

[0174] In one or more embodiments, machine tasks may be performed on the decoder side rather than the encoder side. For example, this may be done because the encoder-side device lacks the ability (computation, power, memory) to run the neural network that performs these tasks. Another example is when some aspect or performance of the task neural network has changed or improved (e.g., different or additional semantic classes, a better neural network architecture) by the time the decoder-side device requires the task results. In one or more embodiments, customization may be required, in which case different clients should run different neural networks to perform these machine learning tasks.

[0175] Moving on to Figure 8, an exemplary Dense Split Attention (DSA) 800 block is provided, in one embodiment, where one type of NN layer is a Resblock 810 containing an NN layer. In one or more embodiments, the DSA 800 is an attention block that estimates one or more attention maps and applies one or more attention maps to one or more data tensors. In one or more embodiments, the attention maps may be vectors, matrices, or tensors. In one example, the attention maps may have values ​​in the range [0,1]. In one or more embodiments, the one or more data tensors may include one or more input tensors to the attention block 800, one or more feature maps extracted within the attention block 800, and / or one or more feature maps extracted outside the attention block 800. In one or more embodiments, the application of one or more attention maps to one or more data tensors may include multiplying the values ​​of one or more attention maps by one or more data tensors, for example, by using element-wise multiplication. In one or more embodiments, other operations may also be considered.

[0176] In one or more embodiments, the DSA block 800 includes extracting features from the input of the DSA block 800 based on at least one or more initial NN layers. In one or more embodiments, the DSA block 800 further includes splitting the extracted features across the channel axis to obtain two split features. In one or more embodiments, the DSA block 800 further includes summing the split features. In one or more embodiments, the DSA block 800 further includes performing a global average operation on the summed features. In one or more embodiments, the DSA block 800 further includes processing the output of the global average operation based on at least one or more NN layers. In one or more embodiments, the DSA block 800 further includes inputting the result of this processing into a softmax operation. In one or more embodiments, the DSA block 800 further includes splitting the result of the softmax operation across the channel axis to obtain two attention tensors. In one or more embodiments, the DSA block 800 further includes multiplying two attention tensors by two previously determined partitioned features to obtain two accompanying partitioned features. In one or more embodiments, the DSA block 800 further includes summing the two accompanying partitioned features. In one or more embodiments, the DSA block 800 further includes chaining the summed accompanying partitioned features with features determined based on at least one or more initial NN layers. In one or more embodiments, the DSA block 800 further includes processing the results of the chaining by at least one or more NN layers. In one or more embodiments, the DSA block 800 further includes summing the output of this processing with the input of the DSA block 800 to obtain the output of the DSA block 800. In one or more embodiments, the global average operation may also be a global pooling (average pooling) operation that calculates the average value of the feature map patches. In one or more embodiments, the global average operation can aggregate the spatial information of the feature map into a single channel to help leverage the inter-channel relationships of the features.

[0177] In one or more embodiments, the DSA block 800 may be used as one of the blocks for an image encoder and an image decoder in an end-to-end trained codec. In the image encoder and image decoder, the DSA block may consist of one or more convolutional layers or one or more transposed convolutional layers.

[0178] Moving on to Figure 9, an end-to-end (e2e) trained intra-frame codec 900 is provided for optimizing rate distortion performance. The exemplary embodiments herein refer to an end-to-end trained intra-frame codec. Nevertheless, at least some embodiments described herein may also apply to an end-to-end trained inter-frame codec used for video compression, or to an end-to-end trained video codec that compresses both intra-frames and inter-frames. The intra-frame codec 900 comprises a first-step codec 905 and a second-step codec 910, where the output of the first-step codec is the input to the second-step codec. In one or more embodiments, the e2e intra-frame codec can process input frames independently and does not use any information from other frames. In one or more exemplary embodiments, YUV is the format of the data input to the first-step codec 905. In embodiments where the YUV format is used, in the YUV format, "Y" represents brightness, or "luma" value, and "UV" represents color, or "chroma" value. In one example, the input image may be a YUV4:4:4 color format image represented as a 256×256×3 three-dimensional array (or tensor), with a horizontal size of 256 pixels and a vertical size of 256 pixels, and the three channels for the Y, U, and V components, respectively. In another example, the input image may be a YUV4:2:0 color format image represented by a combination of a 256×256 matrix for the lumens and a 128×128×2 two-dimensional array (or tensor) for the chromens. Nevertheless, the proposed embodiments can also be extended to other formats such as RGB.

[0179] In one or more embodiments, the end-to-end trained intraframe codec 900 includes components including image encoders 916 and 934, a quantizer, probabilistic models 924 and 942, entropy encoders 920 and 936, entropy decoders 922 and 940, a dequantizer, and image decoders 928 and 938. The entropy encoders 920 and 936 may include reversible encoders, such as arithmetic encoders. In one or more embodiments, the entropy decoders 922 and 940 may include reversible decoders, such as arithmetic decoders. The image encoders 916 and 934, probabilistic models 924 and 942, and image decoders 928 and 938 may include neural network components. In one or more embodiments, the quantizer, entropy codec, and dequantizer may include neural network components.

[0180] In one or more embodiments, the codec includes an encoder and a decoder. For example, the first step codec 905 includes an encoder 912 and a decoder 914. In one or more embodiments, the encoder 912 may include an image encoder 916, a quantizer, a probabilistic model 924, and an entropy encoder 920. In one or more embodiments, the decoder may include an entropy decoder 922, a probabilistic model 924, a dequantizer, and an image decoder 928. In the illustrated embodiments, the probabilistic model 924 in the encoder and the probabilistic model 924 in the decoder are the same probabilistic model. In this context, “same” can refer to two instances of the probabilistic model, one probabilistic model being a copy of another, two identical probabilistic models, or both being embodied in the same instance of the probabilistic model. Similarly, this use of “same” may be used to refer to identical instances, copies, an auxiliary encoder and a quantized latent tensor which are one embodiment. For brevity, identical instances of the probabilistic model and the auxiliary encoder are labeled with the same number in Figure 9.

[0181] In one or more embodiments, the intraframe codec includes two steps for coding input data: a first step codec 905 and a second step codec 910. In one or more embodiments, the first step codec includes an encoder 912 that takes ground truth data as input and outputs a first bitstream. The ground truth data may be data to be compressed. Nevertheless, the codec, such as the first step codec, may use other inputs in addition to the ground truth data, such as instructions for a desired quality of the reconstructed data, or instructions for a desired bitrate of the data to be encoded, or instructions for characteristics of the ground truth data, such as its resolution (in the case of image or video data). In one or more embodiments, the first step codec includes a decoder 914 that takes a first bitstream as input and outputs an initial reconstruction of the input data. In one or more embodiments, the encoder 930 of the second step codec 910 uses residuals, which are calculated based on the ground truth and reconstruction from the first step codec. In one or more embodiments, the probabilistic model 942 of the second step codec 910 uses the reconstruction from the first step codec 905 as an auxiliary input. In one or more embodiments, the encoder 930 of the second step codec 910 outputs a second bitstream. In one or more embodiments, the decoder 932 of the second step codec 910 uses the second bitstream as input and reconstructs the residuals. In one or more embodiments, the reconstructed residuals from the second step codec 910 are added to the reconstruction from the first step codec 910 to obtain the final reconstruction of the ground truth. In one or more embodiments, the sum of the bitstream output by the encoder 912 of the first step codec 905 (i.e., the first bitstream) and the bitstream output by the encoder 930 of the second step codec 910 (i.e., the second bitstream) represents the encoded ground truth.

[0182] This embodiment offers technical advantages over prior embodiments of the coder-decoder. In one or more embodiments, the auxiliary input to the probabilistic model of the second step codec 910 conditions the second step codec based on the initial reconstruction of the input data. This improves the rate distortion performance of the second step codec 910. Consequently, the overall rate distortion performance of the codec 900 (combination of the first step codec 905 and the second step codec 910) is improved.

[0183] In one or more embodiments, the e2e trained intraframe codec 900 may be used as part of a video codec, the intraframe codec may be able to code one or more first frames (e.g., intraframes) of a video independently of any other frames, and another codec, including an interframe codec, may be able to code one or more second frames of a video based on at least one or more third frames, the one or more third frames may include zero or more of the one or more first frames and / or zero or more of the one or more second frames. Nevertheless, in one or more embodiments, the e2e trained intraframe codec 900 may be used to code a subset of all frames that are coded independently of any other frames (i.e., a subset of all intraframes in a video sequence). In one example, the e2e trained intraframe codec 900 is used to code a first subset of all intraframes in a video, the conventional intraframe codec is used to code a second subset of all intraframes in a video, and the conventional interframe codec is used to code interframes in a video.

[0184] In one or more embodiments, in encoder 912, image encoder 916 takes an image as input and converts the image into a latent tensor. In one or more embodiments, given a predefined set of quantization levels, a quantizer quantizes the latent tensor into discrete values ​​to obtain a quantized latent tensor. In one or more embodiments, for each symbol or element in the quantized latent tensor, probabilistic model 924 estimates the probability distribution of all possible values ​​based on the context constructed from the information available in the current encoding / decoding state. In one or more embodiments, arithmetic encoder 920 (reversibly) encodes the input symbol or element into a bitstream, at least based on the estimated probability distribution. In one or more embodiments, this series of steps includes image compression / encoding, and the resulting bitstream represents the compressed or encoded image.

[0185] In one or more embodiments, in decoder 914, the arithmetic decoder 922 and the probabilistic model 924 first decode the bitstream to reconstruct the quantized latent tensor. In one or more embodiments, the probabilistic model 924 used in decoder 914 may be required to be the same probabilistic model 924 used on the encoder side. In one or more embodiments, the dequantizer then reconstructs the latent tensor with continuous values ​​and passes the latent tensor to image decoder 928 to obtain a reconstruction of the input image. In one or more embodiments, the process in decoder 914 represents image decompression / decoding, and the resulting image represents a reconstructed or decoded or decompressed image (these terms may be used as synonyms in at least some of these embodiments).

[0186] In one or more embodiments, the trained intraframe codec 900 may be trained in an end-to-end manner by minimizing D + λR, where D is the distortion loss term, R is the rate loss term, and λ is a weight controlling the balance between the two losses. In one or more embodiments, the applied optimization process results in a trade-off between rate and distortion, where a balance is found between the distortion loss D and the rate loss R. In one or more embodiments, the rate loss R can indicate the bitrate of the encoded image, and the distortion can indicate pixel fidelity distortion, such as mean squared error (MSE), multiscale structural similarity (MS-SSIM), a weighted sum of MSE and MS-SSIM, or other metrics that evaluate the quality of the reconstructed image.

[0187] In one or more embodiments, the distortion may relate to the performance of one or more machine analysis tasks, or the estimated performance of one or more machine analysis tasks. In one or more embodiments, one or more machine analysis tasks may include object detection, image segmentation, instance segmentation, and / or similar. In one or more embodiments, the estimated performance of one or more machine analysis tasks may include distortion calculated based on at least a first set of features extracted from the output of decoder 914 or 932 and a second set of features extracted from the respective ground truth data, the first set of features and the second set of features being output by one or more layers of a pre-trained feature extraction neural network.

[0188] In one or more embodiments, optimization or training may be performed together with respect to the strain loss D and the rate loss R. In one or more embodiments, optimization or training may be performed in two alternating phases, in which the first of the two alternating phases may use only the strain loss D, and in which the second of the two alternating phases may use only the rate loss R.

[0189] As illustrated in Figure 9, in one or more embodiments, the intraframe codec 900 includes a first codec 905, which may be called a first step codec, used to initially code the input image, and a second codec 910, which may be called a second step codec, used to code the residuals of the ground truth (e.g., the difference between the output of the decoder in the first step codec and the respective ground truth data). In one or more embodiments, the input to the first step codec 905 may include the ground truth (e.g., a block or whole image) or a portion of the ground truth (e.g., a masked ground truth). One example of input data to the first step codec 905 is the ground truth. For example, the ground truth may be an image in YUV4:4:4 color format represented as a 256×256×3 multidimensional array or tensor, with a horizontal size of 256 pixels, a vertical size of 256 pixels, and three channels for the Y, U, and V components, respectively. In another example, the ground truth may be an image in YUV4:2:0 color format represented by a combination of a 256×256 matrix for the lumens component and a 128×128×2 two-dimensional array (or tensor) for the chromens component. In yet another example, the ground truth may include only the lumens component, represented as a 256×256×1 one-dimensional array (or tensor). In yet another example, the ground truth may include only the chromens component, represented as a 128×128×2 two-dimensional array (or tensor). Another example of input data to the first step codec 905 is partial ground truth. For example, the input data may be the result of multiplying the ground truth by the mask or output of a Gaussian filter. In one or more exemplary forms, the input to the first step codec 905 may further include one or more pieces of preliminary data, such as block / image resolution.

[0190] In the exemplary configuration depicted in Figure 9, the input data x is ground truth containing both lumen and chroma components, where h × w × 3 represents the size of x, with height h, width w, and number of channels 3. In one or more embodiments, the first step codec 905 is used to initially encode the input data with the encoder 916. In one or more embodiments, the output of the decoder 914 represents the initial reconstruction of the input data.

[0191] In one or more embodiments, the encoder 912 of the first step codec 905 may include a neural encoder 916, a quantizer, a probabilistic model 924, and an entropy encoder 920. The quantizer is not illustrated in Figure 9 but is present in one or more embodiments. In one or more embodiments, the neural encoder 916 comprises a first convolutional layer (which may include "Conv5×5,48,1", where conv represents convolution, 5×5 is the kernel size, 48 is the number of output channels, and 1 is the stride value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. In one or more embodiments, the neural encoder 916 outputs a latent tensor. In one or more embodiments, the latent tensor is converted into a quantized latent tensor by a quantizer. In one or more embodiments, the latent tensor or quantized latent tensor may be input to a probabilistic model 924, where the dimension of the latent tensor represents h / / 16 × w / / 16 × 128, where h / / 16 indicates the height, w / / 16 indicates the width, and 128 indicates the number of channels. In one or more embodiments, the probabilistic model 924 outputs an estimate of the element-wise probability of the (quantized) latent tensor. In one or more embodiments, the probabilistic model 924 may be learned from data by using machine learning techniques—for example, the probabilistic model 924 may be a neural network and may be trained together with other neural networks in the codec. In one or more embodiments, on the encoder 912 side, the output of the probabilistic model 924 is used as one of the inputs to the entropy encoder 920. In one or more embodiments, the entropy encoder may be an arithmetic encoder. In one or more embodiments, the entropy encoder 920 takes in at least a (quantized) latent tensor and the output of the probabilistic model 924 and outputs a bitstream.In one or more embodiments, the latent tensor input to the entropy encoder can be quantized first.

[0192] In one or more embodiments, the decoder 914 of the first step codec 905 may include an entropy decoder 922, a probabilistic model 924, a dequantizer, and a neural decoder 928. In Figure 9, the dequantizer is not illustrated but is present in one or more embodiments. In one or more embodiments, the entropy decoder 922 may also be an arithmetic decoder. In one or more embodiments, the entropy decoder 922 takes in at least a bitstream and the output of the probabilistic model 924 and outputs a (quantized) decoded latent tensor. In one or more embodiments, the probabilistic model 924 may need to be the same probabilistic model available on the encoder 912 side. In one or more embodiments, the decoded latent tensor can undergo dequantization. In one or more embodiments, the decoded latent tensor or the dequantized decoded latent tensor is then input to the neural decoder 928. In one or more embodiments, the neural decoder 928 may include a first transposed convolutional layer ("UpConv5×5,384,2", where UpConv refers to transposed convolution, 5×5 is the kernel size, 384 is the number of output channels, and 2 is the stride value), a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer. In one or more embodiments, the output of the neural decoder 928

[0193]

number

[0194] In one or more embodiments, the input to the second step codec 910 is ground truth (e.g., block or whole image) and initial reconstruction from the first step codec.

[0195]

number

[0196]

number

[0197]

number

[0198]

number

[0199] In the exemplary configuration of the second-step codec 910 depicted in Figure 9, the input data r is initially reconstructed from the ground truth and the first-step codec 905.

[0200]

number

[0201] In one or more embodiments, the encoder 930 of the second step codec 910 may include a neural encoder 934, an auxiliary encoder 944, a quantizer (not shown in Figure 9), a probabilistic model 942, and an entropy encoder 936. In one or more embodiments, similar to the neural encoder 916 of the first step codec 905, the neural encoder 934 of the second step codec 910 may also include a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. In one or more embodiments, the dimension of the latent tensor output by the neural encoder 934 is expressed as h / / 16 × w / / 16 × 128, where the height is h / / 16, the width is w / / 16, and the number of channels is 128.

[0202] In one or more embodiments, the second step codec 910 also includes an auxiliary encoder 944 that generates an auxiliary input to the probabilistic model 942. In one or more embodiments, the auxiliary encoder 944 is used on the encoder 930 side and the decoder 932 side. In one or more embodiments, two copies of the same auxiliary encoder 944 may be created, with the first copy used on the encoder 930 side and the second copy used on the decoder 932 side. In one or more embodiments, different auxiliary encoders are used on the encoder 930 side and the decoder 932 side. In one or more embodiments, the input to the auxiliary encoder 944

[0203]

number

[0204]

number

[0205]

number

[0206]

number

[0207]

number

[0208] In one or more embodiments, the size of the input to the auxiliary encoder 944 may be larger than the size of the input and output of the second step codec 910. For example, when the input image is processed in blocks, the input to the auxiliary encoder 944 may be larger than the size of the block by using data from a larger area of ​​the image reconstructed by the first step codec 905.

[0209] In one or more embodiments, input

[0210]

number

[0211]

number

[0212] In one or more embodiments, the latent tensor (e.g., residual latent tensor) output by the encoder 930 of the second step codec 910 can be input to a probabilistic model 942. In one or more embodiments, the probabilistic model 942 may be a neural network. In one or more embodiments, using auxiliary inputs as preliminary context information, the probabilistic model 942 outputs an estimate of the element-wise probability of the residual latent tensor. In one or more embodiments, on the encoder 930 side, the output of the probabilistic model 942 is used as one of the inputs to the entropy encoder 936. In one or more embodiments, the entropy encoder 936 may be an arithmetic encoder. In one or more embodiments, the entropy encoder 936 takes in at least the (quantized) latent tensor and the output of the probabilistic model 942 and outputs a bitstream representing the encoded residual information. In one or more embodiments, the latent tensor input to the entropy encoder 936 is first quantized.

[0213] In one or more embodiments, the decoder 932 of the second step codec 910 may include an entropy decoder 940, a probabilistic model 942, a dequantizer (not shown), an auxiliary encoder 944, and a neural decoder 938. In one or more embodiments, the entropy decoder 940 may also be an arithmetic decoder. In one or more embodiments, the entropy decoder 940 takes in at least the bitstream output by the encoder 930 and the output of the probabilistic model 942 and outputs a decoded (quantized) latent tensor. In one or more embodiments, the probabilistic model 942 may be the same probabilistic model available on the encoder 930 side. In one or more embodiments, the auxiliary encoder 944 may also be the same auxiliary encoder 944 available on the encoder 930 side. In one or more embodiments, the decoded (quantized) latent tensor can undergo dequantization. In one or more embodiments, after dequantization, the dequantized decoded latent tensor is input to the neural decoder 932. In one or more embodiments, the dequantized decoded latent tensor can be chained with an auxiliary input along the channel dimension. In one or more embodiments, after chaining, the new decoded latent tensor is input to the neural decoder 938. In one or more embodiments, the neural decoder 938 of the second step codec 910 may include a first transposed convolutional layer, a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer. In one or more embodiments, the output of the neural decoder 938 is a reconstruction of the residual components, which may be of size h × w × 3. In one or more embodiments, the reconstructed residual is an initial reconstruction to obtain the final reconstruction.

[0214]

number

[0215] In one or more embodiments, the first step codec 905 may be trained in a first training phase, and the second step codec 910 may be trained in a second training phase by using an initial reconstruction from the first step codec 905 as input, while the second step codec 910 is not trained in the first training phase, and the first step codec 905 is not trained in the second training phase. In one or more embodiments, the first step codec 905 and the second step codec 910 may be trained together (i.e., the second step codec 910 uses the output of the decoder 914 in the first step codec 905 as one of its inputs, and both are trained simultaneously). In one or more embodiments, the first step codec 905 and the second step codec 910 may be trained alternately (for example, the first step codec 905 may be trained during a first number of iterations, then the second step codec 910 may be trained during a second number of iterations, then the first step codec 905 may be trained during a third number of iterations, then the second step codec 910 may be trained during a fourth number of iterations, etc.). Further combinations of these embodiments may be possible, for example, in a first stage, the first step codec 905 and the second step codec 910 may be trained sequentially (for example, the first step codec 905 may be trained first, then the second step codec 910 may be trained), and in a second stage, the first step codec 905 and the second step codec 910 may be trained together or fine-tuned.

[0216] Moving on to Figure 10, an illustration of an exemplary system 1000 in which three or more codecs are used is provided. In one or more embodiments, the e2e intraframe trained codec may include n codecs, where n ≥ 2. In one or more embodiments illustrated in Figure 10, block 1010 represents the first step codec, block 1020 represents the second step codec, and the process performed by block 1020 is repeated one or more times after block 1020, ending with block 10N representing the n step codec. In one or more embodiments, for each codec, the auxiliary data includes the most recent reconstructed ground truth data. In one or more embodiments, the most recent reconstructed ground truth data is derived based on the output of the decoder of the previous codec. In one or more embodiments, the most recent ground truth data is the direct output of the decoder of the previous codec. In one or more embodiments, the most recent reconstructed ground truth data is determined based on output data obtained from the decoder of one or more previous codecs. In one or more embodiments, the most recent reconstructed ground truth data is the sum of the most recent reconstructed ground truth data from one or more previous codecs and the most recent reconstructed residual data.

[0217] In one or more embodiments where the e2e trained codec includes three or more coding steps, for each of the one or more coding steps after the first step coding, a residual is calculated based on ground truth data and a reconstruction determined based on the output of the decoder of the previous step codec, and this residual can be input to the encoder of the current coding step. In one or more embodiments, the probabilistic model of each codec or the auxiliary encoder of each codec can take as input one or more of the following: a reconstruction determined based on the output of the decoder of the previous step codec, the output of the decoder of the previous step codec, and a latent tensor determined by the previous step codec. In one or more embodiments, the codec may include a first step codec, a second step codec, and a third step codec.

[0218] In one or more embodiments, the input to the first step codec is an image representing ground truth data. In one or more embodiments, the decoder of the first step codec outputs a first latent tensor. In one or more embodiments, the decoder of the first step codec outputs a first reconstructed image. In one or more embodiments, a first residual is calculated as the difference between the first reconstructed image and the ground truth data.

[0219] In one or more embodiments, the first residual is input to the second step codec. In one or more embodiments, the auxiliary encoder of the second step codec can take the first latent tensor, the first reconstructed image, features extracted from the first latent tensor, features extracted from the first reconstructed image, or similar as input. In one or more embodiments, the decoder of the second step codec outputs the second latent tensor. In one or more embodiments, the decoder of the second step codec outputs the first reconstructed residual. In one or more embodiments, the first reconstructed residual is added to the first reconstructed image to obtain the second reconstructed image. In one or more embodiments, the second residual is calculated as the difference between the second reconstructed image and the ground truth.

[0220] In one or more embodiments, the second residual is input to the third step codec. In one or more embodiments, the auxiliary encoder of the third step codec can take as input the second latent tensor determined by the second step codec, the second reconstructed image, the first reconstructed residual, features extracted from the second latent tensor, features extracted from the second reconstructed image, features extracted from the first reconstructed residual, or similar. In one or more embodiments, the decoder of the third step codec outputs the second reconstructed residual. In one or more embodiments, the second reconstructed residual is added to the second reconstructed image to obtain the third reconstructed image. In one or more embodiments, the third reconstructed image represents the final reconstructed image of the codec.

[0221] Moving on to Figure 11, an illustrative flowchart is shown for a process 1100 carried out by a device (hereinafter generally referred to as being embodied by a service device 22) that is associated with, or otherwise communicating with, a service device 22, in order to receive residual ground truth data and auxiliary data.

[0222] As shown in block 1110 of Figure 11, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for receiving ground truth data by a first codec. In one or more embodiments, the ground truth data includes an image including brightness data and color data.

[0223] As shown in block 1120 of Figure 11, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating a first bitstream based on ground truth data.

[0224] As shown in block 1130 of Figure 11, the apparatus embodied by the service device 22 generates initial reconstruction data based on a first bitstream, the initial reconstruction data includes means such as a processor 12, a communication interface 16, or the like, which includes the reconstruction of ground truth data.

[0225] As shown in block 1140 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for outputting initial reconstruction data by a first codec.

[0226] As shown in block 1150 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining residual ground truth data based at least on initial reconstruction data and ground truth data. In one or more embodiments, the residual ground truth data includes the difference between the ground truth data and the initial reconstruction data. In one or more embodiments, the residual ground truth data is determined by a first codec.

[0227] As shown in block 1160 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining auxiliary data based at least on a first bitstream or at least on data derived from the first bitstream. In one or more embodiments, the auxiliary data includes one or more of the initial reconstruction data, a first latent tensor, and a first resulting latent tensor.

[0228] As shown in block 1170 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for receiving residual ground truth data and auxiliary data by the second codec. In one or more embodiments, the first and second codecs are trained in an end-to-end manner by reducing at least one of distortion loss and rate loss. In one or more embodiments, the first codec is trained before the second codec, and the second codec is trained at least based on the first codec or at least based on data generated by the first codec. In one or more embodiments, the first and second codecs are trained simultaneously. In one or more embodiments, the first and second codecs are trained at alternating intervals.

[0229] As shown in block 1180 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating a second bitstream based on residual ground truth data and auxiliary data. In one or more embodiments, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining a combined bitstream based at least on the first bitstream and the second bitstream.

[0230] As shown in the optional block 1190 of Figure 11, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes the reconstruction of residual ground truth data. In one or more embodiments, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. In one or more embodiments, the combined reconstruction data is determined by a second codec.

[0231] As shown in optional block 1195 of Figure 11, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for outputting residual reconstruction data by a second codec.

[0232] Moving on to Figure 12, an illustrative flowchart is shown for process 1200, which is carried out by a device (hereinafter generally referred to as being embodied by) associated with, or otherwise communicating with, a service device 22, to encode ground truth data into a first bitstream.

[0233] As shown in block 1210 of Figure 12, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for converting ground truth data into a first latent tensor using a first neural encoder.

[0234] As shown in block 1220 of Figure 12, the apparatus embodied by the service device 22 generates a first quantized latent tensor based on a first latent tensor, using a first quantizer and a first predefined set of quantization levels, wherein the first quantized latent tensor includes at least one symbol or element, and includes means such as a processor 12, a communication interface 16, or the like.

[0235] As shown in block 1230 of Figure 12, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining a first estimated probability distribution of possible values ​​for each symbol or element of at least one symbol or element of a first quantized latent tensor, using a first probabilistic model.

[0236] As shown in block 1240 of Figure 12, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for encoding each symbol or element of at least one symbol or element of a first quantized latent tensor into a first bitstream based at least on a first estimated probability distribution of possible values, using a first entropy encoder.

[0237] Moving on to Figure 13, an illustrative flowchart is shown for a process 1300 carried out by a device (hereinafter generally referred to as being embodied by a service device 22) that is associated with, or otherwise communicating with, a first bitstream to decode the first bitstream into initial reconstruction data.

[0238] As shown in block 1310 of Figure 13, the device embodied by the service device 22 includes a processor 12, a communication interface 16, or similar means for decoding a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor, using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model.

[0239] As shown in block 1320 of Figure 13, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating a first resulting latent tensor using a first dequantizer.

[0240] As shown in block 1330 of Figure 13, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like, for using a first neural decoder to convert the first resulting latent tensor into initial reconstructed data.

[0241] Moving on to Figure 14, an illustrative flowchart is shown for a process 1400 carried out by a device (hereinafter generally referred to as being embodied by a service device 22) that is associated with, or otherwise communicating with, a service device 22, in order to encode the residual ground truth data into a second bitstream.

[0242] As shown in block 1410 of Figure 14, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like, for converting residual ground truth data into a second latent tensor using a second neural encoder.

[0243] As shown in block 1420 of Figure 14, the apparatus embodied by the service device 22 generates a second quantized latent tensor based on a second latent tensor, using a second quantizer and a second predefined set of quantization levels, wherein the second quantized latent tensor includes at least one symbol or element, and includes means such as a processor 12, a communication interface 16, or the like.

[0244] As shown in block 1430 of Figure 14, the device embodied by the service device 22 includes means such as a processor 12 for converting auxiliary data into auxiliary features using an auxiliary encoder, a communication interface 16, or the like.

[0245] As shown in block 1440 of Figure 14, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for inputting auxiliary features into a second probabilistic model.

[0246] As shown in block 1450 of Figure 14, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining a second estimated probability distribution of possible values ​​for each symbol or element of at least one symbol or element of a second quantized latent tensor, using a second probabilistic model.

[0247] As shown in block 1460 of Figure 14, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for encoding each symbol or element of at least one symbol or element of a second quantized latent tensor into a second bitstream based at least on a second estimated probability distribution of possible values, using a second entropy encoder.

[0248] Moving on to Figure 15, an illustrative flowchart is shown for process 1500, which is carried out by a device (hereinafter generally referred to as being embodied by) associated with, or otherwise communicating with, a service device 22, to decode the second bitstream into residual reconstruction data.

[0249] As shown in block 1510 of Figure 15, the device embodied by the service device 22 includes means such as a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a processor 12 for decoding the second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probabilistic model or another probabilistic model identical to the second probabilistic model, a communication interface 16, or the like.

[0250] As shown in block 1520 of Figure 15, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating a second resulting latent tensor using a second dequantizer.

[0251] As shown in block 1530 of Figure 15, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for converting a second resulting latent tensor into residual reconstruction data using a second neural decoder. In one or more embodiments, at least one of the first neural encoder, first neural decoder, first probabilistic model, second neural encoder, second neural decoder, second probabilistic model, and auxiliary encoder includes a neural network component.

[0252] Moving on to Figure 16, an illustrative flowchart is shown for a process 1600 carried out by a device (hereinafter generally referred to as being embodied by a service device 22) that is associated with, or otherwise communicating with, a service device 22, in order to output additional reconstruction data.

[0253] As shown in block 1610 of Figure 16, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining additional residual ground truth data based at least on combined reconstruction data and ground truth data.

[0254] As shown in block 1620 of Figure 16, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for determining additional auxiliary data based at least on the second bitstream or at least on data derived from the second bitstream.

[0255] As shown in block 1630 of Figure 16, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for receiving additional residual ground truth data and additional auxiliary data by different codecs.

[0256] As shown in block 1640 of Figure 16, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for generating additional bitstreams based at least on additional residual ground truth data and additional auxiliary data.

[0257] As shown in block 1650 of Figure 16, the apparatus embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like, which generate additional reconstructed data based on at least an additional bitstream, the additional reconstructed data including the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with combined reconstructed data to form composite reconstructed data.

[0258] As shown in block 1660 of Figure 16, the device embodied by the service device 22 includes means such as a processor 12, a communication interface 16, or the like for outputting at least one of additional reconstructed data and composite reconstructed data by different codecs.

[0259] Moving on to Figure 17, an illustrative flowchart is shown for a process 1700 carried out by a device (hereinafter generally referred to as being embodied by a device) that is associated with, or otherwise communicating with, a consumer device 24, in order to decode the bitstream into reconstructed data.

[0260] As shown in block 1710 of Figure 17, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for receiving the first bitstream.

[0261] As shown in block 1720 of Figure 17, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for generating initial reconstructed data based on a first bitstream.

[0262] As shown in block 1730 of FIG. 17, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for determining auxiliary data based at least on a first bitstream or on data derived from the first bitstream.

[0263] As shown in block 1740 of FIG. 17, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for outputting initial reconstruction data.

[0264] As shown in block 1750 of FIG. 17, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for receiving a second bitstream. In one or more embodiments, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0265] As shown in block 1760 of FIG. 17, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for generating residual reconstruction data based on the second bitstream and the auxiliary data.

[0266] As shown in block 1770 of FIG. 17, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for outputting the residual reconstruction data. In one or more embodiments, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for determining combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. In one or more embodiments, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for...

[0267] Moving on to Figure 18, an illustrative flowchart is shown for a process 1800 carried out by a device (hereinafter generally referred to as being embodied by a device) embodied by a consumer device 24, which is associated with or otherwise communicating with a device, in order to decode the first bitstream into initial reconstructed data.

[0268] As shown in block 1810 of Figure 18, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model.

[0269] As shown in block 1820 of Figure 18, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for generating a first latent tensor using a first dequantizer and based on a first quantized latent tensor.

[0270] As shown in block 1830 of Figure 18, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like, for converting a first latent tensor into initial reconstructed data using a first neural decoder.

[0271] Moving on to Figure 19, an illustrative flowchart is shown for a process 1900 carried out by a device (hereinafter generally referred to as being embodied by a device) embodied by a consumer device 24, which is associated with or otherwise communicating with a device, in order to decode the second bitstream into residual reconstruction data.

[0272] As shown in block 1910 of Figure 19, the apparatus embodied by consumer device 24 includes means such as a processor 12 for decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model, a communication interface 16, or the like.

[0273] As shown in block 1920 of Figure 19, the apparatus embodied by consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for generating a second latent tensor using a second dequantizer and based on a second quantized latent tensor.

[0274] As shown in block 1930 of Figure 19, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for converting a second latent tensor into residual reconstruction data using a second neural decoder. In one or more embodiments, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0275] Moving on to Figure 20, an illustrative flowchart is shown for a process 2000 carried out by a device (hereinafter generally referred to as being embodied by a device) associated with, or otherwise communicating with, a consumer device 24, in order to decode an additional bitstream into additional reconstructed data.

[0276] As shown in block 2010 of Figure 20, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for receiving additional bitstreams.

[0277] As shown in block 2020 of Figure 20, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like, which generate additional reconstructed data based on an additional bitstream, and which is operable to combine the additional reconstructed data with the combined reconstructed data to form composite reconstructed data.

[0278] As shown in block 2030 of Figure 20, the apparatus embodied by the consumer device 24 includes means such as a processor 12, a communication interface 16, or the like for outputting at least one of additional reconstructed data and composite reconstructed data.

[0279] Figures 11–20 illustrate flowcharts depicting methods according to exemplary embodiments of the present disclosure. It will be understood that each block of the flowchart and combinations of blocks in the flowchart may be implemented by various means such as hardware, firmware, processors, circuits, and / or other communication devices associated with the execution of software containing one or more computer program instructions. For example, one or more of the steps described above may be embodied by computer program instructions. In this regard, computer program instructions that embody the steps described above may be stored in a memory device 14 of an apparatus employing an embodiment and executed by a processor 12. Naturally, any such computer program instructions may be loaded into a computer or other programmable device (e.g., hardware) to produce a machine, and as a result, the resulting computer or other programmable device performs the functions specified in the flowchart blocks. These computer program instructions may also be stored in computer-readable memory, instructing the computer or other programmable device to function in a particular manner, and as a result, the instructions stored in computer-readable memory produce a product, and the execution of the instructions performs the functions specified in the flowchart blocks. Computer program instructions may also be loaded into a computer or other programmable device to cause a series of actions to be performed on the computer or other programmable device, in order to produce a computer execution process such that instructions executed on the computer or other programmable device provide actions to perform the functions specified in the flowchart block.

[0280] Therefore, the blocks in a flowchart support combinations of means for performing a specified function, and combinations of actions for performing a specified function. It will also be understood that one or more blocks in a flowchart, and combinations of blocks in a flowchart, can be executed by a special-purpose hardware-based computer system, or by a combination of special-purpose hardware and computer instructions, to perform a specified function.

[0281] Many of the modifications and other embodiments described herein will serve as a reminder to those skilled in the art to which this disclosure applies, who have benefited from the teachings presented in the foregoing description and the accompanying drawings. It should therefore be understood that this disclosure is not to be limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0282] Furthermore, while the foregoing description and associated drawings illustrate exemplary forms in the context of specific exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative embodiments without departing from the scope of the appended claims. In this regard, different combinations of elements and / or functions than those explicitly described above are also conceivable, for example, as may be described in part of the appended claims. Certain terms are used herein, but only in a general and descriptive sense and not for limitation.

Claims

1. The first codec receives the ground truth data, To generate a first bitstream based on ground truth data, The process involves generating initial reconstruction data based on a first bitstream, wherein the initial reconstruction data includes the reconstruction of ground truth data. The first codec outputs the initial reconstruction data, Determine the residual ground truth data based at least on the initial reconstruction data and ground truth data, Determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream, The second codec receives residual ground truth data and auxiliary data, To generate a second bitstream based on residual ground truth data and auxiliary data. A computer execution method, including...

2. The process involves generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes the reconstruction of residual ground truth data. The residual reconstruction data is output by the second codec. The computer execution method according to claim 1, further comprising:

3. Determine the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. A computer execution method according to claim 1 or 2, further comprising:

4. Determining the combined bitstream based at least on the first bitstream and the second bitstream. A computer execution method according to any one of claims 1 to 3, further comprising:

5. The computer execution method according to any one of claims 1 to 4, wherein the residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

6. The computer execution method according to any one of claims 1 to 5, wherein the ground truth data includes an image including brightness data and color data.

7. To generate the first bitstream, Converting ground truth data into a first latent tensor using a first neural encoder, Using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor is generated based on a first latent tensor, wherein the first quantized latent tensor contains at least one symbol or element. For each symbol or element of at least one symbol or element of the first quantized latent tensor, a first estimated probability distribution of possible values ​​is determined using a first probabilistic model, Using a first entropy encoder, each symbol or element of at least one symbol or element of a first quantized latent tensor is encoded into a first bitstream based at least on a first estimated probability distribution of possible values. A computer execution method according to any one of claims 1 to 6, including the method described in any one of claims 1 to 6.

8. Generating initial reconstruction data is Decoding a first bitstream into a first quantized latent tensor or the same quantized latent tensor as the first quantized latent tensor using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model, Using the first dequantizer, generate the first resulting latent tensor, Using the first neural decoder, the resulting latent tensor is converted to the initial reconstructed data. A computer execution method according to any one of claims 1 to 7, including the method described in any one of claims 1 to 7.

9. To generate a second bitstream, Using a second neural encoder, the residual ground truth data is converted into a second latent tensor, Using a second quantizer and a second predefined set of quantization levels, generate a second quantized latent tensor based on a second latent tensor, wherein the second quantized latent tensor contains at least one symbol or element. Using an auxiliary encoder to convert auxiliary data into auxiliary features, Inputting auxiliary features into a second probabilistic model, For each symbol or element of at least one symbol or element of the second quantized latent tensor, a second probabilistic model is used to determine a second estimated probability distribution of possible values, Using a second entropy encoder, each symbol or element of at least one symbol or element of the second quantized latent tensor is encoded into a second bitstream based at least on a second estimated probability distribution of possible values. A computer execution method according to any one of claims 1 to 8, including the method described in any one of claims 1 to 8.

10. Generating residual reconstruction data is Decoding a second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor, using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probabilistic model or another probabilistic model identical to the second probabilistic model, Using a second dequantizer, generate a second resulting latent tensor, Using a second neural decoder, the second resulting latent tensor is converted into residual reconstruction data. A computer execution method according to any one of claims 1 to 9, including the method described in any one of claims 1 to 9.

11. A computer execution method according to any one of claims 7 to 10, wherein at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

12. The computer execution method according to any one of claims 1 to 11, wherein the first codec and the second codec are trained in an end-to-end manner by reducing at least one of distortion loss and rate loss.

13. The computer execution method according to claim 12, wherein the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

14. The computer execution method according to claim 12, wherein the first codec and the second codec are trained simultaneously.

15. The computer execution method according to claim 12, wherein the first codec and the second codec are trained at alternating intervals.

16. The computer execution method according to any one of claims 1 to 15, wherein the auxiliary data includes one or more of the initial reconstruction data, a first latent tensor, and a first resulting latent tensor.

17. A computer execution method according to any one of claims 1 to 16, wherein residual ground truth data is determined by a first codec.

18. A computer execution method according to any one of claims 3 to 17, wherein the combined reconstructed data is determined by a second codec.

19. Based at least on the combined reconstructed data and ground truth data, determine additional residual ground truth data, Determining additional auxiliary data based at least on the second bitstream or at least on data derived from the second bitstream, Different codecs allow for the reception of additional residual ground truth data and additional auxiliary data. To generate additional bitstreams based at least on additional residual ground truth data and additional auxiliary data, The process involves generating additional reconstructed data based on an additional bitstream, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. Different codecs will output at least one of the following: additional reconstructed data and composite reconstructed data. A computer execution method according to any one of claims 3 to 18, further comprising:

20. Receiving the first bitstream, The process involves generating initial reconstruction data based on the first bitstream, Determining auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream, Outputting initial reconstruction data, Receiving a second bitstream, Based on the second bitstream and auxiliary data, residual reconstruction data is generated, Output residual reconstruction data and A computer execution method, including...

21. Determine the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. The computer execution method according to claim 20, further comprising:

22. The computer execution method according to claim 20 or 21, wherein the first bitstream and the second bitstream are received as part of a combined bitstream.

23. Generating initial reconstruction data is Decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model, Using a first dequantizer and based on a first quantized latent tensor, generating a first latent tensor, Using the first neural decoder, convert the first latent tensor into the initial reconstructed data. A computer execution method according to any one of claims 20 to 22, including the method described in any one of claims 20 to 22.

24. Generating residual reconstruction data is Decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model, Using a second dequantizer and based on a second quantized latent tensor, generate a second latent tensor, Using a second neural decoder, the second latent tensor is converted into residual reconstruction data. A computer execution method according to any one of claims 20 to 23, including the method described in any one of claims 20 to 23.

25. A computer execution method according to any one of claims 20 to 24, wherein at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

26. Receiving an additional bitstream, The process involves generating additional reconstructed data based on an additional bitstream, wherein the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. Output at least one of the following: additional reconstructed data and composite reconstructed data. A computer execution method according to any one of claims 21 to 25, further comprising:

27. It is a device, At least one processor, When executed by at least one processor, at least, The first codec receives the ground truth data. To generate a first bitstream based on ground truth data, The process involves generating initial reconstructed data based on a first bitstream, wherein the initial reconstructed data includes the reconstruction of ground truth data. Outputting initial reconstruction data using the first codec, Determine the residual ground truth data based at least on the initial reconstruction data and ground truth data. Determining auxiliary data based at least on a first bitstream or at least on data derived from a first bitstream, The second codec receives residual ground truth data and auxiliary data, and To generate a second bitstream based on residual ground truth data and auxiliary data. A memory containing instructions that cause the device to perform an action and A device equipped with the following features.

28. At least one memory and computer program code are located in at least one processor. The process involves generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes the reconstruction of residual ground truth data. The residual reconstruction data is output by the second codec. The apparatus according to claim 27, further configured to cause the apparatus to perform the following.

29. At least one memory and computer program code are located in at least one processor. Determine the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. The apparatus according to claim 27 or 28, further configured to cause the apparatus to perform the following.

30. At least one memory and computer program code are located in at least one processor. Determining the combined bitstream based at least on the first bitstream and the second bitstream. The apparatus according to any one of claims 27 to 29, further configured to cause the apparatus to perform the following.

31. The apparatus according to any one of claims 27 to 30, wherein the residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

32. The apparatus according to any one of claims 27 to 31, wherein the ground truth data includes an image including brightness data and color data.

33. At least one memory and computer program code are located in at least one processor. Converting ground truth data into a first latent tensor using a first neural encoder, Using a first quantizer and a first predefined set of quantization levels, a first quantized latent tensor is generated based on a first latent tensor, wherein the first quantized latent tensor contains at least one symbol or element. For each symbol or element of at least one symbol or element of the first quantized latent tensor, a first estimated probability distribution of possible values ​​is determined using a first probabilistic model, Using a first entropy encoder, each symbol or element of at least one symbol or element of a first quantized latent tensor is encoded into a first bitstream based at least on a first estimated probability distribution of possible values. The apparatus according to claims 27 to 32, further configured to cause the apparatus to perform the following.

34. At least one memory and computer program code are located in at least one processor. Decoding a first bitstream into a first quantized latent tensor or the same quantized latent tensor as the first quantized latent tensor using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model, Using the first dequantizer, generate the first resulting latent tensor, Using the first neural decoder, the resulting latent tensor is converted to the initial reconstructed data. The apparatus according to any one of claims 27 to 33, further configured to cause the apparatus to perform the following.

35. At least one memory and computer program code are located in at least one processor. Using a second neural encoder, the residual ground truth data is converted into a second latent tensor, Using a second quantizer and a second predefined set of quantization levels, generate a second quantized latent tensor based on a second latent tensor, wherein the second quantized latent tensor contains at least one symbol or element. Using an auxiliary encoder to convert auxiliary data into auxiliary features, Inputting auxiliary features into a second probabilistic model, For each symbol or element of at least one symbol or element of the second quantized latent tensor, a second probabilistic model is used to determine a second estimated probability distribution of possible values, Using a second entropy encoder, each symbol or element of at least one symbol or element of the second quantized latent tensor is encoded into a second bitstream based at least on a second estimated probability distribution of possible values. The apparatus according to any one of claims 27 to 34, further configured to cause the apparatus to perform the following.

36. At least one memory and computer program code are located in at least one processor. Decode the second bitstream into a second quantized latent tensor or the same quantized latent tensor as the second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model or a copy of the second probabilistic model, Using a second dequantizer, generate a second resulting latent tensor, Using a second neural decoder, the second resulting latent tensor is converted into residual reconstruction data. The apparatus according to any one of claims 27 to 35, further configured to cause the apparatus to perform the following.

37. The apparatus according to any one of claims 33 to 36, wherein at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

38. The apparatus according to any one of claims 27 to 37, wherein the first codec and the second codec are trained in an end-to-end manner by reducing at least one of distortion loss and rate loss.

39. The apparatus according to claim 38, wherein the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

40. The apparatus according to claim 38, wherein the first codec and the second codec are trained simultaneously.

41. The apparatus according to claim 38, wherein the first codec and the second codec are trained at alternating intervals.

42. The apparatus according to any one of claims 27 to 41, wherein the auxiliary data includes one or more of the initial reconstruction data, a first latent tensor, and a first resulting latent tensor.

43. The apparatus according to any one of claims 27 to 42, wherein residual ground truth data is determined by a first codec.

44. The apparatus according to any one of claims 29 to 43, wherein the combined reconstructed data is determined by a second codec.

45. At least one memory and computer program code are located in at least one processor. Based at least on the combined reconstructed data and ground truth data, determine additional residual ground truth data, Determining additional auxiliary data based at least on the second bitstream or at least on data derived from the second bitstream, Different codecs allow for the reception of additional residual ground truth data and additional auxiliary data. To generate additional bitstreams based at least on additional residual ground truth data and additional auxiliary data, The process involves generating additional reconstructed data based on an additional bitstream, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. Outputting additional reconstructed data using different codecs The apparatus according to any one of claims 29 to 44, further configured to cause the apparatus to perform the following.

46. It is a device, At least one processor, When executed by at least one processor, at least, Receiving the first bitstream, To generate initial reconstruction data based on the first bitstream, Determining auxiliary data based at least on a first bitstream or at least on data derived from a first bitstream, Output the initial reconstruction data. Receiving a second bitstream, Based on the second bitstream and auxiliary data, generate residual reconstruction data, and Output residual reconstruction data. A memory containing instructions that cause the device to perform an action and A device equipped with the following features.

47. At least one memory and computer program code are located in at least one processor. Determine the combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data. The apparatus according to claim 46, further configured to cause the apparatus to perform the following.

48. The apparatus according to claim 46 or 47, wherein the first bitstream and the second bitstream are received as part of a combined bitstream.

49. At least one memory and computer program code are located in at least one processor. Decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model, Using a first dequantizer and based on a first quantized latent tensor, generating a first latent tensor, Converting the first latent tensor to the initial reconstructed data using the first neural decoder and The apparatus according to any one of claims 46 to 48, further configured to cause the apparatus to perform the following.

50. At least one memory and computer program code are located in at least one processor. Decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model, Using a second dequantizer and based on a second quantized latent tensor, generate a second latent tensor, Convert the second latent tensor into residual reconstruction data using a second neural decoder. The apparatus according to any one of claims 46 to 49, further configured to cause the apparatus to perform the following.

51. The apparatus according to any one of claims 46 to 50, wherein at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

52. At least one memory and computer program code are located in at least one processor. Receiving an additional bitstream, The process involves generating additional reconstructed data based on an additional bitstream, wherein the additional reconstructed data is operable to combine with the combined reconstructed data to form composite reconstructed data. Outputting additional reconstruction data The apparatus according to any one of claims 46 to 51, further configured to cause the apparatus to perform the following.

53. A non-temporary computer-readable storage medium comprising a computer instruction, when executed by the device, causing the device to perform the method according to any one of claims 1 to 19.

54. A non-temporary computer-readable storage medium comprising a computer instruction, when executed by the device, causing the device to perform the method according to any one of claims 20 to 26.

55. A means for receiving ground truth data by the first codec, Means for generating a first bitstream based on ground truth data, Means for generating initial reconstructed data based on a first bitstream, wherein the initial reconstructed data includes the reconstruction of ground truth data. A means for outputting initial reconstruction data using a first codec, A means for determining residual ground truth data based at least on initial reconstruction data and ground truth data, Means for determining auxiliary data based at least on a first bitstream or at least on data derived from a first bitstream, A means for receiving residual ground truth data and auxiliary data by a second codec, Means for generating a second bitstream based on residual ground truth data and auxiliary data, A device equipped with the following features.

56. Means for generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes the reconstruction of residual ground truth data. A means for outputting residual reconstruction data using a second codec and The apparatus according to claim 55, further comprising:

57. Means for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. The apparatus according to claim 55 or 56, further comprising:

58. Means for determining a combined bitstream based at least on a first bitstream and a second bitstream. The apparatus according to any one of claims 55 to 57, further comprising the above.

59. The apparatus according to any one of claims 55 to 58, wherein the residual ground truth data includes the difference between the ground truth data and the initial reconstruction data.

60. The apparatus according to any one of claims 55 to 59, wherein the ground truth data includes an image including brightness data and color data.

61. A means for converting ground truth data into a first latent tensor using a first neural encoder, Means for generating a first quantized latent tensor based on a first latent tensor, using a first quantizer and a first predefined set of quantization levels, wherein the first quantized latent tensor includes at least one symbol or element. A means for determining a first estimated probability distribution of possible values ​​using a first probabilistic model for each symbol or element of at least one symbol or element of a first quantized latent tensor, A means for encoding each symbol or element of at least one symbol or element of a first quantized latent tensor into a first bitstream based at least on a first estimated probability distribution of possible values, using a first entropy encoder. The apparatus according to any one of claims 55 to 60, further comprising the above.

62. A means for decoding a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor, using a first entropy decoder and a first probabilistic model or a copy of the first probabilistic model, A means for generating a first resulting latent tensor using a first dequantizer, A means for converting the first resulting latent tensor into initial reconstructed data using a first neural decoder, The apparatus according to any one of claims 55 to 61, further comprising the above.

63. A means for converting residual ground truth data into a second latent tensor using a second neural encoder, Means for generating a second quantized latent tensor based on a second latent tensor, using a second quantizer and a second predefined set of quantization levels, wherein the second quantized latent tensor includes at least one symbol or element. A means for converting auxiliary data into auxiliary features using an auxiliary encoder, A means for inputting auxiliary features into a second probabilistic model, A means for determining a second estimated probability distribution of possible values ​​using a second probabilistic model for each symbol or element of at least one symbol or element of a second quantized latent tensor, A means for encoding each symbol or element of at least one symbol or element of a second quantized latent tensor into a second bitstream based at least on a second estimated probability distribution of possible values, using a second entropy encoder. The apparatus according to any one of claims 55 to 62, further comprising the above.

64. A means for decoding a second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor, using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probabilistic model or another probabilistic model identical to the second probabilistic model, A means for generating a second resulting latent tensor using a second dequantizer, A means for using a second neural decoder to convert the second resulting latent tensor into residual reconstruction data, The apparatus according to any one of claims 55 to 63, further comprising the above.

65. The apparatus according to any one of claims 61 to 64, wherein at least one of a first neural encoder, a first neural decoder, a first probabilistic model, a second neural encoder, a second neural decoder, a second probabilistic model, and an auxiliary encoder includes a neural network component.

66. The apparatus according to any one of claims 55 to 65, wherein the first codec and the second codec are trained in an end-to-end manner by reducing at least one of distortion loss and rate loss.

67. The apparatus according to claim 66, wherein the first codec is trained before the second codec, and the second codec is trained at least on the first codec or at least on data generated by the first codec.

68. The apparatus according to claim 66, wherein the first codec and the second codec are trained simultaneously.

69. The apparatus according to claim 66, wherein the first codec and the second codec are trained at alternating intervals.

70. The apparatus according to any one of claims 55 to 69, wherein the auxiliary data includes one or more of the initial reconstruction data, a first latent tensor, and a first resulting latent tensor.

71. The apparatus according to any one of claims 55 to 70, wherein residual ground truth data is determined by a first codec.

72. The apparatus according to any one of claims 57 to 71, wherein the combined reconstructed data is determined by a second codec.

73. A means for determining additional residual ground truth data, based at least on combined reconstructed data and ground truth data, Means for determining additional auxiliary data based at least on a second bitstream or at least on data derived from a second bitstream, Means for receiving additional residual ground truth data and additional auxiliary data via different codecs, Means for generating additional bitstreams based at least on additional residual ground truth data and additional auxiliary data, Means for generating additional reconstructed data based on additional bitstreams, wherein the additional reconstructed data includes the reconstruction of additional residual ground truth data, and the additional reconstructed data is operable to combine with combined reconstructed data to form composite reconstructed data. Means for outputting at least one of additional reconstructed data and composite reconstructed data using different codecs The apparatus according to any one of claims 57 to 72, further comprising:

74. A means for receiving the first bitstream, Means for generating initial reconstruction data based on a first bitstream, Means for determining auxiliary data based at least on a first bitstream or at least on data derived from a first bitstream, A means for outputting initial reconstruction data, A means for receiving the second bitstream, A means for generating residual reconstruction data based on a second bitstream and auxiliary data, Means for outputting residual reconstruction data and A device equipped with the following features.

75. Means for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. The apparatus according to claim 74, further comprising:

76. The apparatus according to claim 74 or 75, wherein the first bitstream and the second bitstream are received as part of a combined bitstream.

77. A means for decoding a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model, Means for generating a first latent tensor using a first dequantizer and based on a first quantized latent tensor, A means for converting a first latent tensor to initial reconstructed data using a first neural decoder, The apparatus according to any one of claims 74 to 76, further comprising:

78. A means for decoding a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model, Means for generating a second latent tensor using a second dequantizer and based on a second quantized latent tensor, A means for converting a second latent tensor into residual reconstruction data using a second neural decoder, The apparatus according to any one of claims 74 to 77, further comprising:

79. The apparatus according to any one of claims 74 to 78, wherein at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

80. A means of receiving additional bitstreams, Means for generating additional reconstructed data based on additional bitstreams, wherein the additional reconstructed data is operable to combine with combined reconstructed data to form composite reconstructed data. Means for outputting at least one of additional reconstructed data and composite reconstructed data The apparatus according to any one of claims 74 to 79, further comprising:

Images