Method for intra codec self adjustment in end-to-end learning codec

Abstract

A method, an apparatus, and a computer program product are provided. In the context of a method, the method receives real data by a first codec. The method generates a first bitstream based on real data. The method generates initial reconstructed data based on the first bitstream, wherein the initial reconstructed data includes a reconstruction of the real data. The method outputs initial reconstructed data by a first codec. The method determines residual true data based at least on the initial reconstructed data and the true data. The method determines auxiliary data based at least on the first bitstream, or at least on data derived from the first bitstream. The method receives, by a second codec, residual true data and auxiliary data. The method generates a second bitstream based on the residual true data and the auxiliary data.

Description

Technical Field

[0001] The example implementations relate to video and image encoder-decoders (codecs), and more specifically to end-to-end learned encoder-decoders. Background Technology

[0002] Neural networks (NNs) are increasingly used in various devices, such as mobile phones. NNs are used for image and video analysis and processing, social media data analysis, device usage analysis, and other applications. A neural network is a computational graph consisting of several computational layers. Each layer can consist of one or more units that perform basic computations. One unit is connected to one or more other units, and this connection is typically associated with weights used to scale the signal through that connection. The weights are values ​​that can be learned from the training data. Furthermore, batch normalization layer parameters can be learned from the training data.

[0003] Feedforward neural networks have no feedback loops, where each layer receives input from the previous layer and provides its output to the next layer. Initial layers, closer to the input data, extract low-level semantic features, such as edges and textures in an image, while intermediate and final layers extract higher-level features. Furthermore, one or more layers can perform tasks such as classification, semantic segmentation, object detection, denoising, style transfer, and super-resolution. Recurrent neural networks have feedback loops, making the neural network stateful. Recurrent neural networks are able to remember information or states.

[0004] Neural networks can learn attributes from input data in a supervised or unsupervised manner. This ability is a result of training algorithms or meta-neural networks that provide training signals. Training algorithms modify the properties of the neural network so that the output is as close as possible to the desired output. For example, for a neural network classifying objects in an image, the network output can be used to derive the object's class or category index. Training can minimize or reduce the error in the output, i.e., the loss. Examples of losses include mean squared error and cross-entropy. Training can be an iterative process, where in each iteration, the algorithm modifies the neural network's weights to gradually reduce the network's loss. The ultimate goal of training a neural network is to enable it to learn the properties of the data distribution from a finite training dataset, or to "generalize" to data not used for training. Validation datasets not used for training are typically used to check the performance of the neural network.

[0005] When examining the performance of a neural network, the training set error should be minimized. Otherwise, the network is underfitting. If the network is generalizing, the validation set error should be minimized and should not be significantly higher than the training set error. Otherwise, the network is overfitting (i.e., the network has memorized the properties of the training set and only performs well on that set).

[0006] In the case of image codecs, neural network-based codecs can be used to compress and decompress data such as images. A neural network-based image codec can include a component called an autoencoder, which comprises a neural encoder and a neural decoder. The neural encoder takes the image as input and generates a latent tensor that may require fewer bits than the input image. The latent tensor can be quantized and losslessly compressed to obtain a bitstream representing the encoded image. The neural decoder takes the bitstream and reconstructs the image that was input to the neural encoder.

[0007] The neural encoder and decoder are trained to minimize the combination of bit rate and distortion. Distortion can be based on mean squared error (MSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), etc. These distortion measures are related to the quality of human visual perception, so improving them can improve the visual quality of decoded images perceived by humans.

[0008] A video codec consists of a neural encoder that converts the input video into a compressed representation suitable for storage / transmission and a neural decoder that decompresses the compressed video representation back into a visual form. The neural encoder typically discards some information from the original video to represent it in a more compact form (at a lower bit rate). Hybrid video codecs can encode video information in two stages. First, the pixel values ​​of a given image block are predicted. This can be done, for example, by a motion compensation component (i.e., finding and indicating the region in one of the previously encoded video frames that corresponds to the height of the block being encoded) or by a spatial component (i.e., using the pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error (i.e., the difference between the predicted pixel block and the original pixel block) is encoded. This can be achieved, for example, by transforming the differences in pixel values ​​using a specified transform (e.g., discrete cosine transform), quantization coefficients, and entropy encoding the quantization coefficients. By varying the fidelity of the quantization process, the neural encoder controls the balance between the precision of pixel rendering (pixel quality) and the size of the final encoded video representation (file size or transmission bit rate).

[0009] Inter-frame prediction (also known as temporal prediction, motion-compensated, or motion-compensated prediction) utilizes temporal redundancy. Inter-frame prediction uses previously decoded images as prediction sources. In contrast, intra-frame prediction uses adjacent pixels within two identical images. Intra-frame prediction can be performed in the spatial domain to predict sample values ​​or in the transform domain to predict transform coefficients. Intra-frame prediction is typically used in intra-coding where no prediction is applied.

[0010] The result of the encoding process is a set of encoded parameters, such as motion vectors and quantization transform coefficients. Entropy coding of the parameters can be more efficient if they are predicted first from spatially or temporally adjacent parameters. For example, motion vectors can be predicted based on spatially adjacent motion vectors, and only the difference relative to the motion vector predictor can be encoded. The prediction of encoded parameters and intra-frame prediction can be collectively referred to as intra-image prediction.

[0011] The decoder reconstructs the video input to the encoder by applying a prediction component similar to that of the encoder. In doing so, the decoder uses motion or spatial information created by the encoder and stored in a compressed representation to form a predicted representation of pixel blocks. The decoder performs prediction error decoding (the inverse operation of prediction error encoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying prediction and prediction error decoding, the decoder adds the prediction and prediction error signals (pixel values) to form the output video frame. The decoder and encoder can also apply additional filtering to improve the quality of the output video before passing it to a display or storing it as a prediction reference for upcoming frames in a video sequence.

[0012] In a typical video encoder-decoder, motion information is indicated by motion vectors associated with each motion-compensated image block. Each motion vector represents the displacement of the corresponding image block in the picture to be encoded or decoded and the predicted source block in one of the previously encoded or decoded images. To efficiently represent motion vectors, they are typically differentially encoded relative to block-specific predicted motion vectors. In a typical video codec, predicted motion vectors are created in a predefined manner. For example, they can be created by calculating the median of the encoded or decoded motion vectors of neighboring blocks. Another method for creating motion vector predictions is to generate a list of candidate predictions based on neighboring and / or juxtaposed blocks in a time-referenced image and signal the selected candidate as the motion vector predictor. Furthermore, reference indices from previously encoded or decoded images can be predicted. Reference indices can be predicted based on neighboring and / or juxtaposed blocks in a time-referenced image. Additionally, typical efficient video codecs employ an additional motion information encoding / decoding mechanism. This mechanism is often called a merge mode, where all motion field information (including motion vectors and corresponding reference image indices for each available list of reference images) is predicted and used without modification or correction. Similarly, motion field information is predicted using motion field information from adjacent and / or juxtaposed blocks in a time-referenced image. The information of the motion field to be used is signaled to a candidate motion field list with available motion field information from adjacent and / or juxtaposed blocks.

[0013] In a typical video codec, the motion-compensated prediction residuals are first transformed using a transform kernel such as DCT, and then encoded. This is done to reduce the correlation between prediction residuals and provide more efficient coding. A typical video encoder uses a Lagrange cost function to find the optimal coding pattern (e.g., the desired macroblock pattern and associated motion vectors). This is a costly function that uses a weighting factor λ to correlate the image distortion caused by lossy coding methods with the amount of information needed to represent the pixel values ​​in an image region. The cost can be expressed by the equation C = D + λR, where C is the Lagrange cost to be minimized, D is the image distortion (such as mean square error) considering both pattern and motion vectors, and R is the number of bits required in the decoder to reconstruct the image patch (including the amount of data representing candidate motion vectors).

[0014] In some cases, Supplemental Enhancement Information (SEI) messages can be used. In other cases, SEI NAL units consisting of a prefix SEI NAL unit and a suffix SEI NAL unit are used. A prefix SEI NAL unit can begin a picture unit, and a suffix SEI NAL unit can end a picture unit. An SEI unit contains one or more SEI messages that are not essential for output picture decoding but can be helpful in related processes such as picture output timing, post-processing of decoded pictures, rendering, error detection, error hiding, and resource preservation.

[0015] Image and video codecs can use a set of filters to enhance the visual quality of predicted visual content. These filters can be applied in-loop, out-of-loop, or both. In the case of in-loop filters, a filter applied to a block in the currently coded frame will affect the encoding of another block in the same frame and / or another frame predicted based on the currently coded frame. In-loop filters may affect bitrate and / or visual quality. Enhanced blocks will result in smaller residuals (the difference between the original block and the predicted and filtered blocks), thus requiring fewer bits for encoding. After the frame is reconstructed, out-of-loop filters are applied to the frame. In this case, the filtered visual content does not become a source of prediction and may only affect the visual quality of the frame output by the decoder.

[0016] The embodiments described herein disclose improved video and image codecs that improve rate distortion performance in video and image compression using end-to-end learned codecs. Compared to the preceding embodiments, the embodiments of this disclosure provide lower bitrates and higher luminance PSNR for the resulting reconstructed images. The embodiments of this disclosure achieve better rate distortion performance in reconstructed images. Summary of the Invention

[0017] A method, apparatus, and computer program product for end-to-end learning of intra-frame codecs are disclosed. By using end-to-end learning with multiple codecs, the bit rate can be reduced and the PSNR improved.

[0018] In one example embodiment, a method is provided that includes receiving real data by a first codec. The method further includes generating a first bitstream based on the real data. The method further includes generating initial reconstructed data based on the first bitstream, wherein the initial reconstructed data includes a reconstruction of the real data. The method further includes outputting the initial reconstructed data by the first codec. The method further includes determining residual real data based at least on the initial reconstructed data and the real data. The method further includes determining auxiliary data based at least on the first bitstream, or at least on data derived from the first bitstream. The method further includes receiving the residual real data and the auxiliary data by a second codec. The method further includes generating a second bitstream based on the residual real data and the auxiliary data.

[0019] The method in the example embodiment further includes generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of the original residual data. The method also includes outputting the residual reconstruction data by a second codec.

[0020] The method in the example embodiment also includes determining combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0021] The method in the example embodiment further includes determining a combined bit stream based at least on a first bit stream and a second bit stream.

[0022] In one example embodiment, the residual ground truth data includes the difference between the ground truth data and the initial reconstructed data.

[0023] In one example embodiment, the real data includes an image that includes brightness data and color data.

[0024] The method in the example embodiment further includes converting real data into a first latent tensor using a first neural encoder. The method further includes generating a first quantized latent tensor using a first quantizer and a first predefined set of quantization levels, at least based on the first latent tensor, wherein the first quantized latent tensor includes at least one symbol or element. The method further includes determining a first estimated probability distribution of possible values ​​for a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor using a first probability model. The method further includes encoding a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor into a first bitstream using a first entropy encoder, at least based on the first estimated probability distribution of possible values.

[0025] The method in the example embodiment further includes decoding the first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor using a first entropy decoder and a first probability model or a copy of the first probability model. The method also includes using a first dequantizer to generate a first resulting latent tensor. The method further includes using a first neural decoder to convert the first resulting latent tensor into initial reconstructed data.

[0026] The method in the example embodiment further includes converting residual ground data into a second latent tensor using a second neural encoder. The method further includes generating a second quantized latent tensor using a second quantizer and a second predefined set of quantization levels, at least based on the second latent tensor, wherein the second quantized latent tensor includes at least one symbol or element. The method further includes converting auxiliary data into auxiliary features using an auxiliary encoder. The method further includes inputting the auxiliary features into a second probabilistic model. The method further includes determining a second estimated probability distribution of possible values ​​for a corresponding symbol or element in at least one symbol or element of the second quantized latent tensor using the second probabilistic model. The method further includes encoding a corresponding symbol or element in at least one symbol or element of the second quantized latent tensor into a second bitstream using a second entropy encoder, at least based on the second estimated probability distribution of possible values.

[0027] The method in the example embodiment further includes decoding the second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probabilistic model or another probabilistic model identical to the second probabilistic model. The method further includes using a second dequantizer to generate a second resulting latent tensor. The method further includes using a second neural decoder to convert the second resulting latent tensor into residual reconstructed data.

[0028] In one example embodiment, at least one of the first neural encoder, the first neural decoder, the first probabilistic model, the second neural encoder, the second neural decoder, the second probabilistic model, and the auxiliary encoder includes a neural network component.

[0029] In one example embodiment, the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

[0030] In one example embodiment, the first codec is trained before the second codec, and the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

[0031] In one example embodiment, the first codec and the second codec are trained simultaneously.

[0032] In one example embodiment, the first codec and the second codec are trained at alternating intervals.

[0033] In one example embodiment, the auxiliary data includes one or more of the initial reconstruction data, the first potential tensor, and the first result potential tensor.

[0034] In one example embodiment, the residual true data is determined by a first codec.

[0035] In one example embodiment, the combined reconstructed data is determined by a second codec.

[0036] The method in the example embodiment further includes determining additional residual real data based at least on the combined reconstructed data and the real data. The method further includes determining additional auxiliary data based at least on a second bitstream, or at least on data derived from the second bitstream. The method further includes receiving the additional residual real data and the additional auxiliary data by different codecs. The method further includes generating an additional bitstream based at least on the additional residual real data and the additional auxiliary data. The method further includes generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual real data, and wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data. The method further includes outputting at least one of the additional reconstructed data and the composite reconstructed data by different codecs.

[0037] In one example embodiment, a method is provided that includes receiving a first bit stream. The method further includes generating initial reconstructed data based on the first bit stream. The method further includes determining auxiliary data based at least on the first bit stream, or at least on data derived from the first bit stream. The method further includes outputting the initial reconstructed data. The method further includes receiving a second bit stream. The method further includes generating residual reconstructed data based on the second bit stream and the auxiliary data. The method further includes outputting the residual reconstructed data.

[0038] The method in the example embodiment also includes determining combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0039] In one example embodiment, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0040] The method in the example embodiment further includes decoding the first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. The method also includes generating the first latent tensor using a first dequantizer and based on the first quantized latent tensor. The method further includes converting the first latent tensor into initial reconstructed data using a first neural decoder.

[0041] The method in the example embodiment further includes decoding the second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. The method also includes using a second dequantizer and generating a second latent tensor based on the second quantized latent tensor. The method further includes using a second neural decoder to convert the second latent tensor into residual reconstructed data.

[0042] In one example embodiment, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0043] The method in the example embodiment further includes receiving an additional bitstream. The method also includes generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data is operable to be combined with composite reconstructed data to form composite reconstructed data. The method further includes outputting at least one of the additional reconstructed data and the composite reconstructed data.

[0044] In one example embodiment, an apparatus is provided, comprising at least one processor and at least one memory including computer program code configured, together with the at least one processor, such that the apparatus receives real data by a first codec. The at least one memory and the computer program code are further configured to generate a first bitstream based on the real data. The at least one memory and the computer program code are further configured to generate initial reconstructed data based on the first bitstream, wherein the initial reconstructed data includes a reconstruction of the real data. The at least one memory and the computer program code are further configured to output the initial reconstructed data by the first codec. The at least one memory and the computer program code are further configured to determine residual real data based at least on the initial reconstructed data and the real data. The at least one memory and the computer program code are further configured to determine auxiliary data based at least on the first bitstream, or at least on data derived from the first bitstream. The at least one memory and the computer program code are further configured to receive the residual real data and the auxiliary data by a second codec. The at least one memory and the computer program code are further configured to generate a second bitstream based on the residual real data and the auxiliary data.

[0045] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to cause the apparatus to generate residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of the original residual data. The at least one memory and computer program code are also configured to output the residual reconstruction data by a second codec.

[0046] In one example embodiment, at least one memory and computer program code are also configured, together with at least one processor, such that the apparatus determines combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0047] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, such that the device determines a combined bit stream based at least on a first bit stream and a second bit stream.

[0048] In one example embodiment, the residual ground truth data includes the difference between the ground truth data and the initial reconstructed data.

[0049] In one example embodiment, the real data includes an image that includes brightness data and color data.

[0050] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to cause the device to convert real data into a first latent tensor using a first neural encoder. The at least one memory and computer program code are also configured to generate a first quantized latent tensor using a first quantizer and a first predefined set of quantization levels, at least based on the first latent tensor, wherein the first quantized latent tensor includes at least one symbol or element. The at least one memory and computer program code are further configured to: for a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor, use a first probability model to determine a first estimated probability distribution of possible values. The at least one memory and computer program code are also configured to encode a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor into a first bitstream using a first entropy encoder, at least based on the first estimated probability distribution of possible values.

[0051] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to decode a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor using a first entropy decoder and a first probability model or a copy of the first probability model. At least one memory and computer program code are also configured to generate a first result latent tensor using a first dequantizer. At least one memory and computer program code are further configured to convert the first result latent tensor into initial reconstructed data using a first neural decoder.

[0052] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to cause the device to convert residual ground data into a second latent tensor using a second neural encoder. At least one memory and computer program code are further configured to generate a second quantized latent tensor using a second quantizer and a second predefined set of quantization levels, at least based on the second latent tensor, wherein the second quantized latent tensor includes at least one symbol or element. At least one memory and computer program code are further configured to convert auxiliary data into auxiliary features using an auxiliary encoder. At least one memory and computer program code are further configured to input the auxiliary features into a second probabilistic model. At least one memory and computer program code are further configured to: for a corresponding symbol or element among at least one symbol or element of the second quantized latent tensor, use the second probabilistic model to determine a second estimated probability distribution of possible values. At least one memory and computer program code are further configured to encode a corresponding symbol or element among at least one symbol or element of the second quantized latent tensor into a second bitstream using a second entropy encoder, at least based on the second estimated probability distribution of possible values.

[0053] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to decode the second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probability model or a copy of the second probability model. At least one memory and computer program code are also configured to generate a second resulting latent tensor using a second dequantizer. At least one memory and computer program code are further configured to convert the second resulting latent tensor into residual reconstructed data using a second neural decoder.

[0054] In one example embodiment, at least one of the first neural encoder, the first neural decoder, the first probabilistic model, the second neural encoder, the second neural decoder, the second probabilistic model, and the auxiliary encoder includes a neural network component.

[0055] In one example embodiment, the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

[0056] In one example embodiment, the first codec is trained before the second codec, and the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

[0057] In one example embodiment, the first codec and the second codec are trained simultaneously.

[0058] In one example embodiment, the first codec and the second codec are trained at alternating intervals.

[0059] In one example embodiment, the auxiliary data includes one or more of the initial reconstruction data, the first potential tensor, and the first result potential tensor.

[0060] In one example embodiment, the residual true data is determined by a first codec.

[0061] In one example embodiment, the combined reconstructed data is determined by a second codec.

[0062] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, such that the apparatus determines additional residual real data based at least on combined reconstructed data and real data. At least one memory and computer program code are further configured to determine additional auxiliary data based at least on a second bitstream, or at least on data derived from the second bitstream. At least one memory and computer program code are further configured to receive the additional residual real data and the additional auxiliary data by different codecs. At least one memory and computer program code are further configured to generate an additional bitstream based at least on the additional residual real data and the additional auxiliary data. At least one memory and computer program code are further configured to generate additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual real data, and wherein the additional reconstructed data is operable to be combined with combined reconstructed data to form composite reconstructed data. At least one memory and computer program code are further configured to output the additional reconstructed data by different codecs.

[0063] In one example embodiment, an apparatus is provided, comprising at least one processor and at least one memory including computer program code configured, together with the at least one processor, such that the apparatus receives a first bitstream. The at least one memory and the computer program code are further configured to generate initial reconstruction data based on the first bitstream. The at least one memory and the computer program code are also configured to determine auxiliary data based at least on the first bitstream, or at least on data derived from the first bitstream. The at least one memory and the computer program code are further configured to output the initial reconstruction data. The at least one memory and the computer program code are also configured to receive a second bitstream. The at least one memory and the computer program code are further configured to generate residual reconstruction data based on the second bitstream and the auxiliary data. The at least one memory and the computer program code are also configured to output the residual reconstruction data.

[0064] In one example embodiment, at least one memory and computer program code are also configured, together with at least one processor, such that the apparatus determines combined reconstruction data based at least on initial reconstruction data and residual reconstruction data.

[0065] In one example embodiment, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0066] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to decode a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. At least one memory and computer program code are also configured to generate the first latent tensor using a first dequantizer and based on the first quantized latent tensor. At least one memory and computer program code are further configured to convert the first latent tensor into initial reconstructed data using a first neural decoder.

[0067] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to decode a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. At least one memory and computer program code are also configured to generate a second latent tensor using a second dequantizer and based on the second quantized latent tensor. At least one memory and computer program code are further configured to convert the second latent tensor into residual reconstructed data using a second neural decoder.

[0068] In one example embodiment, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0069] In one example embodiment, at least one memory and computer program code are further configured, together with at least one processor, to cause the device to receive an additional bit stream. The at least one memory and computer program code are also configured to generate additional reconstructed data based on the additional bit stream, wherein the additional reconstructed data is operable to be combined with composite reconstructed data to form composite reconstructed data. The at least one memory and computer program code are also configured to output the additional reconstructed data.

[0070] In one example embodiment, a non-transitory computer-readable storage medium is provided, comprising computer instructions that, when executed by a device, cause the device to receive real data by a first codec. The non-transitory computer-readable storage medium also includes computer instructions configured to generate a first bitstream based on the real data upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to generate initial reconstructed data based on the first bitstream upon execution, wherein the initial reconstructed data includes a reconstruction of the real data. The non-transitory computer-readable storage medium also includes computer instructions configured to output the initial reconstructed data by the first codec upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to determine residual real data based at least on the initial reconstructed data and the real data upon execution. The non-transitory computer-readable storage medium also includes computer instructions configured to determine auxiliary data based at least on the first bitstream or at least on data derived from the first bitstream upon execution. The non-transitory computer-readable storage medium also includes computer instructions configured to receive residual real data and auxiliary data by a second codec upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to generate a second bitstream based on the residual real data and auxiliary data upon execution.

[0071] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to generate residual reconstruction data based on a second bitstream at execution time, wherein the residual reconstruction data includes a reconstruction of the original residual data. The non-transitory computer-readable storage medium also includes computer instructions configured to output the residual reconstruction data by a second codec at execution time.

[0072] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to determine combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data when executed.

[0073] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to determine a combined bit stream at least based on a first bit stream and a second bit stream when executed.

[0074] In one example embodiment, the residual ground truth data includes the difference between the ground truth data and the initial reconstructed data.

[0075] In one example embodiment, the real data includes an image that includes brightness data and color data.

[0076] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to, upon execution, convert real data into a first latent tensor using a first neural encoder. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, generate a first quantized latent tensor using a first quantizer and a first predefined set of quantization levels, at least based on the first latent tensor, wherein the first quantized latent tensor includes at least one symbol or element. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution: for a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor, use a first probability model to determine a first estimated probability distribution of possible values. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, encode a corresponding symbol or element among at least one symbol or element of the first quantized latent tensor into a first bitstream using a first entropy encoder, at least based on the first estimated probability distribution of possible values.

[0077] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to, upon execution, decode a first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor using a first entropy decoder and a first probability model or a copy of the first probability model. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, use a first dequantizer to generate a first result latent tensor. The non-transitory computer-readable storage medium further includes computer instructions configured to, upon execution, use a first neural decoder to convert the first result latent tensor into initial reconstructed data.

[0078] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to, upon execution, convert residual ground data into a second latent tensor using a second neural encoder. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, generate a second quantized latent tensor based at least on the second latent tensor using a second quantizer and a second predefined set of quantization levels, wherein the second quantized latent tensor includes at least one symbol or element. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, convert auxiliary data into auxiliary features using an auxiliary encoder. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, input auxiliary features into a second probabilistic model. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution: for a corresponding symbol or element in at least one symbol or element of the second quantized latent tensor, use the second probabilistic model to determine a second estimated probability distribution of possible values. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, encode, at least one symbol or element of a second quantized potential tensor into a second bitstream using a second entropy encoder based at least on a second estimated probability distribution of possible values.

[0079] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to, upon execution, decode the second bitstream into a second quantized latent tensor or a quantized latent tensor identical to the second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probability model or a copy of the second probability model. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, use a second dequantizer to generate a second result latent tensor. The non-transitory computer-readable storage medium further includes computer instructions configured to, upon execution, use a second neural decoder to convert the second result latent tensor into residual reconstructed data.

[0080] In one example embodiment, at least one of the first neural encoder, the first neural decoder, the first probabilistic model, the second neural encoder, the second neural decoder, the second probabilistic model, and the auxiliary encoder includes a neural network component.

[0081] In one example embodiment, the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

[0082] In one example embodiment, the first codec is trained before the second codec, and the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

[0083] In one example embodiment, the first codec and the second codec are trained simultaneously.

[0084] In one example embodiment, the first codec and the second codec are trained at alternating intervals.

[0085] In one example embodiment, the auxiliary data includes one or more of the initial reconstruction data, the first potential tensor, and the first result potential tensor.

[0086] In one example embodiment, the residual true data is determined by a first codec.

[0087] In one example embodiment, the combined reconstructed data is determined by a second codec.

[0088] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to determine additional residual real data based at least on the combined reconstructed data and the real data when executed. The non-transitory computer-readable storage medium also includes computer instructions configured to determine additional auxiliary data based at least on a second bitstream, or at least on data derived from the second bitstream, when executed. The non-transitory computer-readable storage medium also includes computer instructions configured to receive the additional residual real data and the additional auxiliary data by different codecs when executed. The non-transitory computer-readable storage medium also includes computer instructions configured to generate an additional bitstream based at least on the additional residual real data and the additional auxiliary data when executed. The non-transitory computer-readable storage medium also includes computer instructions configured to generate additional reconstructed data based on the additional bitstream when executed, wherein the additional reconstructed data includes a reconstruction of the additional residual real data, and wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data. The non-transitory computer-readable storage medium also includes computer instructions configured to output additional reconstructed data by different codecs upon execution.

[0089] In one example embodiment, a non-transitory computer-readable storage medium is provided, comprising computer instructions that, when executed by a device, cause the device to receive a first bitstream. The non-transitory computer-readable storage medium also includes computer instructions configured to generate initial reconstruction data based on the first bitstream upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to determine auxiliary data upon execution, based at least on the first bitstream or at least on data derived from the first bitstream. The non-transitory computer-readable storage medium also includes computer instructions configured to output the initial reconstruction data upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to receive a second bitstream upon execution. The non-transitory computer-readable storage medium also includes computer instructions configured to generate residual reconstruction data based on the second bitstream and the auxiliary data upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to output the residual reconstruction data upon execution.

[0090] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to determine combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data when executed.

[0091] In one example embodiment, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0092] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to, upon execution, decode a first bitstream into a first quantized latent tensor using a first entropy decoder and a first probabilistic model. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, use a first dequantizer and generate a first latent tensor based on the first quantized latent tensor. The non-transitory computer-readable storage medium also includes computer instructions configured to, upon execution, use a first neural decoder to convert the first latent tensor into initial reconstructed data.

[0093] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to decode a second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model at execution. The non-transitory computer-readable storage medium also includes computer instructions configured to generate a second latent tensor based on the second quantized latent tensor at execution. The non-transitory computer-readable storage medium further includes computer instructions configured to convert the second latent tensor into residual reconstructed data using a second neural decoder at execution.

[0094] In one example embodiment, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0095] The non-transitory computer-readable storage medium of the example embodiment also includes computer instructions configured to receive an additional bit stream upon execution. The non-transitory computer-readable storage medium further includes computer instructions configured to generate additional reconstructed data based on the additional bit stream upon execution, wherein the additional reconstructed data is operable to be combined with composite reconstructed data to form composite reconstructed data. The non-transitory computer-readable storage medium also includes computer instructions configured to output the additional reconstructed data upon execution.

[0096] In one example embodiment, an apparatus is provided, including components for receiving real data by a first codec. The apparatus also includes components for generating a first bitstream based on the real data. The apparatus further includes components for generating initial reconstructed data based on the first bitstream, wherein the initial reconstructed data includes a reconstruction of the real data. The apparatus also includes components for outputting the initial reconstructed data by the first codec. The apparatus further includes components for determining residual real data based at least on the initial reconstructed data and the real data. The apparatus further includes components for determining auxiliary data based at least on the first bitstream, or at least on data derived from the first bitstream. The apparatus also includes components for receiving the residual real data and the auxiliary data by a second codec. The apparatus further includes components for generating a second bitstream based on the residual real data and the auxiliary data.

[0097] The apparatus of the example embodiment further includes components for generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of the original residual data. The apparatus also includes components for outputting the residual reconstruction data by a second codec.

[0098] The apparatus of the example embodiment also includes a component for determining combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0099] The apparatus of the example embodiment also includes components for determining a combined bit stream based at least on a first bit stream and a second bit stream.

[0100] In one example embodiment, the residual ground truth data includes the difference between the ground truth data and the initial reconstructed data.

[0101] In one example embodiment, the real data includes an image that includes brightness data and color data.

[0102] The apparatus of the example embodiment further includes components for converting real data into a first latent tensor using a first neural encoder. The apparatus also includes components for generating a first quantized latent tensor using a first quantizer and a first predefined set of quantization levels, at least based on the first latent tensor, wherein the first quantized latent tensor includes at least one symbol or element. The apparatus further includes components for determining a first estimated probability distribution of possible values ​​for a corresponding symbol or element among the at least one symbol or element of the first quantized latent tensor using a first probability model. The apparatus further includes components for encoding a corresponding symbol or element among the at least one symbol or element of the first quantized latent tensor into a first bitstream using a first entropy encoder, at least based on the first estimated probability distribution of possible values.

[0103] The apparatus of the example embodiment further includes means for decoding the first bitstream into a first quantized latent tensor or a quantized latent tensor identical to the first quantized latent tensor using a first entropy decoder and a first probability model or a copy of the first probability model. The apparatus also includes means for generating a first result latent tensor using a first dequantizer. Finally, the apparatus includes means for converting the first result latent tensor into initial reconstructed data using a first neural decoder.

[0104] The apparatus of the example embodiment further includes components for converting residual ground data into a second latent tensor using a second neural encoder. The apparatus also includes components for generating a second quantized latent tensor using a second quantizer and a second predefined set of quantization levels, at least based on the second latent tensor, wherein the second quantized latent tensor includes at least one symbol or element. The apparatus further includes components for converting auxiliary data into auxiliary features using an auxiliary encoder. The apparatus further includes components for inputting the auxiliary features into a second probabilistic model. The apparatus further includes components for using the second probabilistic model to determine a second estimated probability distribution of possible values ​​for a corresponding symbol or element in at least one symbol or element of the second quantized latent tensor. The apparatus further includes components for encoding a corresponding symbol or element in at least one symbol or element of the second quantized latent tensor into a second bitstream using a second entropy encoder, at least based on the second estimated probability distribution of possible values.

[0105] The apparatus of the example embodiment further includes components for decoding the second bitstream into a second quantization latent tensor or a quantization latent tensor identical to the second quantization latent tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and a second probability model or another probability model identical to the second probability model. The apparatus also includes components for generating a second result latent tensor using a second dequantizer. The apparatus further includes components for converting the second result latent tensor into residual reconstructed data using a second neural decoder.

[0106] In one example embodiment, at least one of the first neural encoder, the first neural decoder, the first probabilistic model, the second neural encoder, the second neural decoder, the second probabilistic model, and the auxiliary encoder includes a neural network component.

[0107] In one example embodiment, the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

[0108] In one example embodiment, the first codec is trained before the second codec, and the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

[0109] In one example embodiment, the first codec and the second codec are trained simultaneously.

[0110] In one example embodiment, the first codec and the second codec are trained at alternating intervals.

[0111] In one example embodiment, the auxiliary data includes one or more of the initial reconstruction data, the first potential tensor, and the first result potential tensor.

[0112] In one example embodiment, the residual true data is determined by a first codec.

[0113] In one example embodiment, the combined reconstructed data is determined by a second codec.

[0114] The apparatus of the example embodiment further includes components for determining additional residual real data based at least on the combined reconstructed data and the real data. The apparatus also includes components for determining additional auxiliary data based at least on a second bitstream, or at least on data derived from the second bitstream. The apparatus further includes components for receiving the additional residual real data and the additional auxiliary data by different codecs. The apparatus further includes components for generating an additional bitstream based at least on the additional residual real data and the additional auxiliary data. The apparatus further includes components for generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual real data, and wherein the additional reconstructed data is operable for combining with the combined reconstructed data to form composite reconstructed data. The apparatus further includes components for outputting at least one of the additional reconstructed data and the composite reconstructed data by different codecs.

[0115] In one example embodiment, an apparatus is provided, including components for receiving a first bit stream. The apparatus further includes components for generating initial reconstructed data based on the first bit stream. The apparatus also includes components for determining auxiliary data based at least on the first bit stream, or at least on data derived from the first bit stream. The apparatus further includes components for outputting the initial reconstructed data. The apparatus also includes components for receiving a second bit stream. The apparatus further includes components for generating residual reconstructed data based on the second bit stream and the auxiliary data. The apparatus further includes components for outputting the residual reconstructed data.

[0116] The apparatus of the example embodiment also includes a component for determining combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

[0117] In one example embodiment, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0118] The apparatus of the example embodiment further includes components for decoding the first bitstream into a first quantized latent tensor using a first entropy decoder and a first probability model. The apparatus also includes components for generating the first latent tensor using a first dequantizer and based on the first quantized latent tensor. The apparatus further includes components for converting the first latent tensor into initial reconstructed data using a first neural decoder.

[0119] The apparatus of the example embodiment further includes components for decoding the second bitstream into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probabilistic model. The apparatus also includes components for generating the second latent tensor using a second dequantizer and based on the second quantized latent tensor. Finally, the apparatus includes components for converting the second latent tensor into residual reconstructed data using a second neural decoder.

[0120] In one example embodiment, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0121] The apparatus of the example embodiment further includes components for receiving additional bitstreams. The apparatus also includes components for generating additional reconstructed data based on the additional bitstreams, wherein the additional reconstructed data is operable to be combined with composite reconstructed data to form composite reconstructed data. The apparatus further includes components for outputting at least one of the additional reconstructed data and the composite reconstructed data. Attached Figure Description

[0122] After a general description of certain exemplary embodiments of this disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in the drawings:

[0123] Figure 1 This is a block diagram of an example communication system in which example embodiments of the present disclosure may be deployed;

[0124] Figure 2 This is a block diagram of a system that includes both service equipment and user equipment, and is designed to perform functions according to the example embodiments herein;

[0125] Figure 3 This is an illustration of an encoder-decoder that uses a neural network as a component of a pipeline according to the foregoing embodiments;

[0126] Figure 4A This is an illustration of a video coding pipeline utilizing a neural network according to the foregoing embodiments;

[0127] Figure 4B This is an illustration of a video coding pipeline utilizing neural networks on the encoding and decoding sides according to the foregoing embodiments;

[0128] Figure 5 This is an illustration of a video coding system based on end-to-end neural network learning according to the foregoing embodiments;

[0129] Figure 6A This is a diagram of the VCM pipeline according to the foregoing embodiment;

[0130] Figure 6B This is a diagram of a VCM pipeline using an end-to-end learning method according to the foregoing embodiments;

[0131] Figure 7 This is a diagram of the training end-to-end learning system according to the foregoing embodiments;

[0132] Figure 8 This is an illustration of an example densely split attention block according to an example embodiment of the present disclosure;

[0133] Figure 9 This is an illustration of an intra-frame codec learned end-to-end according to an exemplary embodiment of the present disclosure;

[0134] Figure 10 This is a block diagram of a system having more than two codecs according to an exemplary embodiment of the present disclosure;

[0135] Figure 11 It is a flowchart illustrating the operations performed to generate a bitstream based on real data, such as... Figure 2 The operation performed by the device;

[0136] Figure 12 It is a flowchart illustrating the operations performed to encode the first bitstream, such as by Figure 2 The operation performed by the device;

[0137] Figure 13 This is a flowchart illustrating the operations performed to generate initial reconstruction data, such as... Figure 2 The operation performed by the device;

[0138] Figure 14 This is a flowchart illustrating the operations performed to encode a second bitstream, such as those performed by... Figure 2 The operation performed by the device;

[0139] Figure 15 This is a flowchart illustrating the operations performed to generate residual reconstruction data, such as those performed by... Figure 2 The operation performed by the device;

[0140] Figure 16 This is a flowchart illustrating the operations performed to output additional reconstructed data, such as... Figure 2 The operation performed by the device;

[0141] Figure 17 This is a flowchart illustrating the operations performed to output residual reconstruction data, such as... Figure 2 The operation performed by the device;

[0142] Figure 18 This is a flowchart illustrating the operations performed to generate initial reconstruction data, such as... Figure 2 The operation performed by the device;

[0143] Figure 19 This is a flowchart illustrating the operations performed to generate residual reconstruction data, such as those performed by... Figure 2 The operation performed by the device; and

[0144] Figure 20 This is a flowchart illustrating the operations performed to output additional reconstructed data, such as... Figure 2 The operation performed by the device. Detailed Implementation

[0145] Some embodiments of the invention will now be described more fully with reference to the accompanying drawings, which illustrate some, but not all, embodiments of the invention. In fact, various embodiments of the invention may be embodied in many different forms and should not be construed as limited to the embodiments described herein; rather, these embodiments are provided so that this disclosure meets applicable legal requirements. The same reference numerals consistently refer to the same elements. As used herein, the terms “data,” “content,” “information,” and similar terms are used interchangeably to refer to data that can be transmitted, received, and / or stored according to embodiments of the invention. Therefore, the use of any such terms should not be considered as limiting the spirit and scope of the embodiments of the invention.

[0146] Although the terms first, second, etc., may be used herein to describe various elements, these elements should not be limited by these terms. These terms are used only to distinguish one element from another. For example, without departing from the scope of this disclosure, a first element may be referred to as a second element, and similarly, a second element may be referred to as a first element. As used herein, the term "and / or" includes any and all combinations of one or more of the related listed items.

[0147] When an element is described as "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there may be an intermediate element. In contrast, when an element is described as "directly connected" or "directly coupled" to another element, there is no intermediate element. Other terms used to describe the relationship between elements should be interpreted in a similar way (e.g., "between" and "directly between", "adjacent" and "next to", etc.).

[0148] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “the” used herein include the plural forms as well. It should also be understood that the terms “comprises,” “comprising,” “includes,” and / or “including”, when used herein, specify the presence of the stated feature, integral, step, operation, element, and / or component, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof.

[0149] It should also be noted that in some alternative implementations, the above functions / actions may not occur in the order shown in the diagram. For example, depending on the functions / behaviors involved, two diagrams shown consecutively may actually be executed concurrently, or sometimes in reverse order.

[0150] Specific details are set forth in the following description to provide a thorough understanding of the exemplary embodiments. However, those skilled in the art will understand that the exemplary embodiments can be practiced without these specific details. For example, a system may be illustrated in block diagrams so as not to obscure the exemplary embodiments with unnecessary detail. In other instances, well-known processes, structures, and techniques may be shown without unnecessary detail to avoid confusing the exemplary embodiments.

[0151] Furthermore, as used herein, the term "circuit system" can refer to one or more or all of the following: (a) a purely hardware circuit implementation (such as implementations in analog and / or digital circuit systems); (b) a combination of circuitry and software, such as (if applicable): (i) a combination of (one or more) analog and / or digital hardware circuitry with software / firmware, and (ii) any portion of (one or more) hardware processors (including (one or more) digital signal processors), software, and (one or more) memory having software, which work together to enable a device (such as a mobile phone or server) to perform various functions; and (c) (one or more) hardware circuitry and / or (one or more) processors, such as (one or more) microprocessors or portions thereof, which require software (e.g., firmware) to operate, but may be absent when operation is not required. This definition of "circuit system" applies to all uses of the term in this application, including in any claim. As another example, as used herein, the term "circuit system" also covers implementations of hardware circuitry or processors (or processors) alone or portions thereof and their accompanying software and / or firmware. For example, if applicable to a particular claim element, the term "circuit system" also includes baseband integrated circuits or processor integrated circuits for mobile phones, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0152] Furthermore, as used in this paper, the terms "model," "neural network," "neural net," and "network" are used interchangeably. Additionally, the weights of a neural network can be referred to as learnable parameters or parameters.

[0153] Furthermore, as used herein, the terms “machine” and “task neural network” are used interchangeably to refer to any process or algorithm (whether or not it learns from data) that analyzes or processes data for a specific task.

[0154] Furthermore, as used herein, the terms “receiver side” and “decoder side” refer to physical or abstract entities or devices that may contain one or more machines and that may run on some encoded and ultimately decoded video representation encoded by another physical or abstract entity or device (“encoder-side device”).

[0155] Furthermore, as used herein, the terms "intra-frame," "frame," and "image" are used interchangeably. These terms can refer to at least a portion of the input data and at least a portion of the output data of an intra-frame codec learned end-to-end (e2e). In one or more embodiments, these terms refer to an image as a data type. However, the proposed embodiments can be extended to other types of data, such as video, audio, etc.

[0156] As defined herein, “computer-readable storage medium” refers to a physical storage medium (e.g., a volatile or non-volatile memory device) that can be distinguished from “computer-readable transmission medium” (which refers to an electromagnetic signal).

[0157] In the following description, illustrative embodiments are described using the actions and symbolic representations of reference operations (e.g., in the form of flowcharts, diagrams, data flow graphs, structure diagrams, block diagrams, etc.). These operations can be implemented as program modules or functional processes, including routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types, and can be implemented using existing hardware at existing network elements. Such existing hardware may include one or more central processing units (CPUs), digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), computers, etc.

[0158] Although flowcharts can describe operations as a sequential process, many operations can be executed in parallel, concurrently, or simultaneously. Furthermore, the order of operations can be rearranged. When a process completes its operations, it can be terminated, but there may also be other steps not included in the diagram. A process can correspond to a method, function, procedure, subroutine, subroutine, etc. When a process corresponds to a function, its termination can correspond to the function returning to the calling function or the main function.

[0159] As disclosed herein, the term "storage medium" or "computer-readable storage medium" can refer to one or more devices for storing data, including read-only memory (ROM), random access memory (RAM), magnetic RAM, core memory, disk storage media, optical storage media, flash memory devices, and / or other tangible machine-readable media for storing information. The term "computer-readable medium" can include, but is not limited to, portable or fixed storage devices, optical storage devices, and a variety of other media capable of storing, containing, or carrying one or more instructions and / or data.

[0160] Furthermore, the example embodiments can be implemented using hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof. When implemented in software, firmware, middleware, or microcode, program code or code segments that perform the necessary tasks can be stored in a machine or computer-readable medium, such as a computer-readable storage medium. When implemented in software, one or more processors will perform the necessary tasks.

[0161] A code segment can represent a procedure, function, subroutine, program, routine, subroutine, module, software package, class, or any combination of instructions, data structures, or program statements. A code segment can be coupled to another code segment or hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc., can be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network transmission, etc.

[0162] Example implementations can be used in conjunction with RANs such as: Universal Mobile Telecommunications System (UMTS); Global System for Mobile Communications (GSM); Advanced Mobile Telephone Services (AMPS) systems; Narrowband AMPS systems (NAMPS); Total Access Communications System (TACS); Personal Digital Cellular (PDC) systems; U.S. Digital Cellular (USDC) systems; Code Division Multiple Access (CDMA) systems as described in EIA / TIA IS-95; High-Speed ​​Packet Data (HRPD) systems; Global Microwave Interconnection Access (WiMAX); Ultra Mobile Broadband (UMB); and 3GPP LTE.

[0163] As described herein, a method, apparatus, and computer program product for video and image compression and reconstruction using an end-to-end learned encoder-decoder are provided.

[0164] To perform end-to-end learning compression and reconstruction, for example, such as... Figure 1The device 10 shown can be embodied in or communicate with any of a variety of different types of computing devices, including, for example, video processing systems, image processing systems, or any other system configured to decompress images captured by a snapshot compression sensing system. Figure 1 As shown, the apparatus of the example embodiment includes a processor 12, an associated memory 14, and a communication interface 16, which are associated with or otherwise communicate with.

[0165] Processor 12 (and / or coprocessor, or auxiliary processor, or any other circuitry otherwise associated with the processor) may communicate with memory device 14 via a bus for transferring information between components of device 10. The memory device may be non-transitory and may include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer-readable storage medium) including gates configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device such as a processor). The memory device may be configured to store information, data, content, applications, instructions, etc., enabling the device to perform various functions according to exemplary embodiments of this disclosure. For example, the memory device may be configured to buffer input data for processor processing. Additionally or alternatively, the memory device may be configured to store instructions for processor execution.

[0166] In some embodiments, device 10 may be embodied in various computing devices as described above. However, in some embodiments, the device may be embodied as a chip or chipset. In other words, the device may include one or more physical packages (e.g., chips), including materials, components, and / or wires on structural components (e.g., substrates). Structural components may provide physical strength, save size, and / or limit electrical interactions for the component circuitry systems included thereon. Thus, in some cases, the device may be configured to implement embodiments of the invention on a single chip or as a single "system-on-a-chip." Thus, in some cases, a chip or chipset may constitute components for performing one or more operations to provide the functions described herein.

[0167] Processor 12 can be embodied in a variety of different ways. For example, a processor can be embodied as one or more of a variety of hardware processing units, such as a coprocessor, microprocessor, controller, digital signal processor (DSP), processing element with or without an accompanying DSP, or various other circuit systems including integrated circuits, such as, for example, ASIC (Application-Specific Integrated Circuit), FPGA (Field-Programmable Gate Array), microcontroller unit (MCU), hardware accelerator, application-specific computer chip, etc. Therefore, in some embodiments, the processor may include one or more processing cores configured to execute independently. Multi-core processors can implement multiprocessing within a single physical package. Additionally or alternatively, the processor may include one or more processors configured in series via a bus to implement independent execution of instructions, pipelined, and / or multithreaded.

[0168] In one example embodiment, processor 12 may be configured to execute instructions stored in memory device 14 or otherwise accessible to the processor. Alternatively or additionally, the processor may be configured to perform hard-coded functions. Thus, whether configured by hardware or software methods, or by a combination thereof, the processor may represent an entity (e.g., physically embodied in a circuit system) capable of performing operations according to embodiments of this disclosure when configured accordingly. Thus, for example, when the processor is embodied as an ASIC, FPGA, etc., the processor may be hardware specifically configured to perform the operations described herein. Alternatively, as another example, when the processor is embodied as an instruction executor, the instructions may specifically configure the processor to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, the processor may be the processor of a particular device (e.g., an image processing system) configured to further configure the processor to employ embodiments of the invention by executing instructions of the algorithms and / or operations described herein. The processor may include clocks, arithmetic logic units (ALUs), and logic gates, etc., configured to support processor operation.

[0169] Communication interface 16 can be any component, such as a device or circuitry system embodied in hardware or a combination of hardware and software, configured to receive and / or transmit data, such as by receiving frames from a snapshot compression sensing system or external memory device, and / or to provide reconstructed signals to an imaging system or other type of display for presentation, or to an external memory device for storage. In this regard, for example, the communication interface may include an antenna (or antennas) and supporting hardware and / or software for communicating with a wireless communication network. Additionally or alternatively, the communication interface may include circuitry for interacting with the antenna(s) to transmit signals via the antenna(s) or to process signals received via the antenna(s). In some environments, the communication interface may alternatively or additionally support wired communication. Thus, for example, the communication interface may include a communication modem and / or other hardware / software for supporting communication via cable, digital subscriber line (DSL), universal serial bus (USB), or other mechanisms.

[0170] Processor 12 can be configured to execute instructions of a computer program by performing arithmetic, logical, and input / output operations of the system. Instructions can be provided to processor 12 by memory 14.

[0171] Various interfaces of device 10 may include components that interface processor 12 with antennas or other input / output components. As will be understood, the interfaces and programs stored in memory 14 that articulate the specific functions of device 10 will vary depending on the implementation of device 10.

[0172] In one example embodiment, device 10 can be any known or under-development device, including but not limited to mobile phones, laptops, tablets, personal computers, portable media devices (such as televisions), multi-function cameras, drones, electric vehicles, etc.

[0173] Video and image compression codecs are devices or computer programs that encode and / or decode digital data streams, bit streams, image sequences, signals, etc., associated with video and / or images. Still image codecs may conform to standards such as JPEG, GIF, and PNG. Video codecs may conform to standards such as Cinepak, MPEG, MPEG-2, H.264, VP8, and H.265.

[0174] Now go to Figure 2An example system for implementing an end-to-end codec is provided. In this example embodiment, the serving device 22 can perform encoding of an input image or video (wherein encoding may further include performing decoding). In one or more embodiments, the serving device 22 can send encoded bitstream data generated from the input image to a consumer device 24. In one example embodiment, the bitstream data can be transmitted wirelessly via a network. In one example embodiment, the bitstream data can be transmitted between the serving device 22 and the consumer device 24 via a wired connection. In one or more embodiments, the consumer device 24 can decode the bitstream data received from the serving device 22 into one or more reconstructed images or videos. In one or more embodiments, the serving device 22 can store the bitstream data in a memory 14. The memory 14 can be internal or external to the serving device 22. In one or more embodiments, the serving device 22 is capable of retrieving the bitstream data from the memory 14 and reconstructing the input image. In one or more embodiments, the serving device 22 and / or the consumer device 24 can be embodied by means 10.

[0175] Now go to Figure 3 Examples of previous embodiments of a codec 300 using a neural network as part of a pipeline are provided. In one or more embodiments, the in-loop filter 310 may include a neural network. In one example, the in-loop filter 310 includes one or more neural network-based in-loop filters and one or more non-neural network-based in-loop filters. In another example, the in-loop filter 310 includes only one or more neural network-based in-loop filters. In one or more embodiments, intra-frame prediction 320 may include one or more neural networks. In one or more embodiments, inter-frame prediction 330 may include one or more neural networks. In one or more embodiments, transform and / or inverse transform 340 may include one or more neural networks. In one or more embodiments, lossless coding (e.g., entropy coding) 350 may include or use a neural network-based probabilistic model. In various embodiments, other neural networks may be used throughout the pipeline.

[0176] Now go to Figure 4A The illustration provides an illustration of a previous embodiment of a video coding pipeline 400, in which the main components are replaced by neural networks. The video coding pipeline 400 uses a compression method with end-to-end learning of neural networks. Figure 4BAn example pipeline 410 using neural networks on both the encoder and decoder sides is illustrated. In one or more embodiments, pipeline 410 includes an analysis network 420, a quantization component and arithmetic encoder in box 430, an arithmetic decoder 440, and a synthesis network 450. In one or more embodiments, analysis network 420 includes an encoder neural network, and synthesis network 450 includes a decoder neural network. In one or more embodiments, analysis network 420 and synthesis network 450 are part of a neural autoencoder architecture. In one or more embodiments, analysis network 420 is configured to perform a nonlinear transformation, and synthesis network 450 is configured to perform an inverse nonlinear transformation.

[0177] In one or more embodiments, analysis network 420 analyzes the input data and outputs a new representation of the input data. This new representation can be more compressible. Then, in block 430, this new representation can be quantized into a discrete number of values. Then, in block 430, the quantized data is losslessly encoded, for example, by an arithmetic encoder, to obtain a bitstream. On the decoding side, the bitstream is first losslessly decoded, for example, by using an arithmetic decoder 440. The losslessly decoded data is dequantized and then input to synthesis network 450. The output is reconstructed or decoded data.

[0178] In one or more embodiments of performing lossy compression, the lossy step may include analyzing the quantization in network 420 and block 430.

[0179] In one example embodiment, to train pipeline 410, a training objective function (also known as a “training loss”) is typically used, which may include one or more terms, loss terms, or simply loss. In one example embodiment, the training loss includes a reconstruction loss term and a rate loss term. In one example embodiment, the reconstruction loss prompts the system to decode data similar to the input data based on some similarity metric. Examples of reconstruction losses include mean squared error (MSE), multi-scale structural similarity (MS-SSIM), losses generated using a pre-trained network, losses generated using a neural network trained simultaneously with an end-to-end learned encoder-decoder, etc. An example of a loss generated using a pre-trained network is error(f1, f2), where f1 and f2 are, respectively, features extracted by the pre-trained neural network for the input and decoded data, and error() is an error or distance function, such as the L1 norm or L2 norm. An example of a loss generated using a neural network trained simultaneously with an end-to-end learned encoder-decoder is adversarial loss, which is the loss provided by a discriminator neural network trained adversarially with respect to the encoder-decoder, based on the settings proposed in the context of generative adversarial networks (GANs) and their variants.

[0180] Rate loss can cause the system to compress (i.e., reduce the number of bits) the output of the coding level, such as the output of an arithmetic encoder. In one or more embodiments, when using an entropy-based lossless encoder (e.g., an arithmetic encoder), rate loss causes the output of the analysis network 420 to have low entropy. Examples of rate loss include differentiable estimates of entropy, sparsity loss (i.e., losses that cause the output or quantized output of the analysis network 420 to have many zeros, such as L0 norm, L1 norm, L1 norm divided by L2 norm), cross-entropy loss applied to the output of a probabilistic model (where the probabilistic model can be a neural network used to estimate the probability of the next symbol to be encoded by the arithmetic encoder), etc.

[0181] In one or more embodiments, one or more reconstruction losses may be used, and one or more rate losses may be used as a weighted sum. In one or more embodiments, different loss terms are weighted with different weights, and these weights determine how the final system performs with respect to rate distortion loss. For example, if more weight is given to the reconstruction loss relative to the rate loss, the system can learn to compress less but reconstruct with higher accuracy (as measured by metrics associated with the reconstruction loss). In one or more embodiments, these weights are hyperparameters of the training session and may be set manually by the person designing the training session, or automatically, for example, through grid search or by using additional neural networks.

[0182] In one or more embodiments, non-neural networks can be used for end-to-end learning methods, such as arithmetic codecs.

[0183] Now go to Figure 5 A video coding system 500 based on neural networks for end-to-end learning, according to the foregoing embodiments, is provided. In one or more embodiments, system 500 includes an encoder 510, a quantizer 520, a probabilistic model 530, an entropy codec (arithmetic encoder 540 and arithmetic decoder 550), an inverse quantizer 560, and a decoder 570. In one or more embodiments, encoder 510 and decoder 570 are two neural networks. In one or more embodiments, encoder 510 and decoder 570 primarily include neural network components. In one or more embodiments, probabilistic model 530 primarily includes neural network components. In one or more embodiments, quantizer 520, inverse quantizer 560, and entropy codec are not based on neural network components. In one or more alternative embodiments, these components do indeed include neural network components.

[0184] In one or more embodiments, encoder 510 takes video as input and transforms the video from its original signal space into a latent representation (also called a latent tensor), which may include a more compressible representation of the input. In one or more embodiments, in the case of an input image, the latent representation may be a three-dimensional tensor, where two dimensions represent vertical and horizontal spatial dimensions, and a third dimension represents a "channel" containing information at that particular location. In one or more embodiments, where the input image is a 128×128×3 RGB image (where the horizontal dimension is 128 pixels, the vertical dimension is 128 pixels, and the red, green, and blue components are 3 channels), and encoder 510 downsamples the input tensor to half its original size and expands the channel dimension to 32 channels, the latent representation includes a tensor with dimensions (or "shape") of 64×64×32 (i.e., a horizontal dimension of 64 elements, a vertical dimension of 64 elements, and 32 channels). In various embodiments, the order of the different dimensions may vary depending on the conventions used. In one example embodiment, for an input image, the channel dimension may be the first dimension. In this example, the aforementioned shape of the input tensor can be represented as 3×128×128 instead of 128×128×3. In an example embodiment with input video, another dimension in the input tensor can be used to represent temporal information. In one example embodiment, given a predefined set of quantization levels, quantizer 520 quantizes the latent representation into discrete values. The output of quantizer 520 can be referred to as the quantized latent tensor. In one or more embodiments, probabilistic model 530 and arithmetic codec components work together to perform lossless compression on the quantized latent representation and generate a bitstream to be sent to the decoder side. Given a symbol to be encoded into the bitstream, probabilistic model 530 estimates the probability distribution of all possible values ​​for that symbol based on a context constructed from available information (such as data already encoded / decoded) from the current encoding / decoding state. In one or more embodiments, arithmetic encoder 540 uses the estimated probability distribution to encode the input symbol into a bitstream.

[0185] In one or more embodiments, the reverse operation is performed on the decoder side. In one or more embodiments, the arithmetic decoder 550 and the probabilistic model 530 first decode symbols from the bitstream to recover the quantized latent representation. In one or more embodiments, the dequantizer 560 then reconstructs the latent representation with continuous values ​​and passes it to the decoder 570 to recover the input video / image. In one or more embodiments, the probabilistic model 530 is shared between the encoding and decoding systems. In one example embodiment, a copy of the probabilistic model is used on the encoder side, and another exact copy is used on the decoder side.

[0186] In an example embodiment of system 500, encoder 510, probabilistic model 530, and decoder 570 are based on deep neural networks. In one example embodiment, the system is trained end-to-end by minimizing the following rate distortion loss function: L = D + λR In one example embodiment, D It is a distortion loss item. R It is a rate loss term. λ These are weights used to control the balance between the two losses. In one example embodiment, the distortion loss term can be mean squared error (MSE), structural similarity (SSIM), or other metrics used to evaluate the quality of the reconstructed video. In one or more embodiments, multiple distortion losses can be used and integrated into D, such as a weighted sum of MSE and SSIM. In one or more embodiments, the rate loss term is an estimated entropy that quantizes the latent representation, indicating the number of bits required to represent the coded symbols, for example, bits per pixel (bpp).

[0187] In an example system for lossless video / image compression, the system may consist only of a probabilistic model 530, an arithmetic encoder 540, and an arithmetic decoder 550. In one or more embodiments, the system loss function includes only rate loss, since distortion loss is always zero (i.e., no information loss).

[0188] In one or more embodiments, the decoded data can be analyzed by a machine. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. Example use cases and applications include autonomous vehicles, video surveillance cameras and public safety, smart sensor networks, smart TVs and smart advertising, people re-identification, smart traffic monitoring, drones, etc. In one or more embodiments, different quality metrics and specialized algorithms for compressing and decompressing data for machine consumption can be used to compress and decompress data for human consumption. A set of tools and concepts for compressing and decompressing data for machine consumption is referred to herein as machine video coding.

[0189] In one or more embodiments, the receiver-side device has multiple "machines" or neural networks (NNs). In one or more embodiments, these multiple machines can be used in a specific combination determined by the orchestrator subsystem. In one or more embodiments, multiple machines can be used sequentially and / or in parallel based on the output of previously used machines. For example, compressed and then decompressed video can be analyzed by one machine (NN) for detecting pedestrians, another machine (another NN) for detecting cars, and yet another machine (another NN) for estimating the depth of all pixels in a frame.

[0190] In one or more embodiments, the encoded video data may be stored in a memory device, such as as a file. In one or more embodiments, the stored file may later be provided to another device. In one or more alternative embodiments, the encoded video data may be streamed from one device to another.

[0191] Now go to Figure 6A This document provides an illustration of a video encoding pipeline for a machine 600 according to a prior embodiment. In one or more embodiments, a VCM encoder 610 encodes an input video into a bitstream. In one or more embodiments, a bitrate can be calculated from the bitstream to evaluate the size of the bitstream. In one or more embodiments, a VCM decoder 620 decodes the bitstream output by the VCM encoder 610. In one or more embodiments, the output from the VCM decoder 620 includes decoded data for the machine. In one or more embodiments, this data may be decoded or reconstructed video. However, in one or more embodiments, this data may not have the same or similar characteristics as the original video input to the VCM encoder 610. For example, humans may not easily understand this data by simply rendering it onto a screen. In one or more embodiments, the output of the VCM decoder 620 is then input to one or more task neural networks 631-63N. In one or more embodiments, any number of task neural networks may be present. In one or more embodiments, the aim of the VCM 600 is to achieve a low bitrate while ensuring that the task NNs 631-63N perform well in terms of the evaluation metrics associated with each task.

[0192] In one or more embodiments, an end-to-end learning approach can be used to implement video encoding for the machine. In one or more embodiments, the VCM encoder 610 and VCM decoder 620 are primarily composed of neural networks. Figure 6BAn example pipeline of an end-to-end learning method according to the foregoing embodiments is illustrated. In one or more embodiments, video is input to a neural network encoder 650. In one or more embodiments, the output of the neural network encoder is input to a lossless encoder 670, such as an arithmetic encoder, which outputs a bitstream. In one or more embodiments, the lossless codec may include a probabilistic model 640 in both the lossless encoder 670 and the lossless decoder 680, which predicts the probability of the next symbol to be encoded and decoded. In one or more embodiments, the probabilistic model 640 may be learned, for example, it may be a neural network. In one or more implementations, on the decoder side, the bitstream is input to a lossless decoder 680, such as an arithmetic decoder, the output of which is input to a neural network decoder 660. In one or more embodiments, the output of the neural network decoder 660 is machine-decoded data that may be input to one or more tasks NN 631-63N.

[0193] Now go to Figure 7 This illustrates how an end-to-end learning system 700 is trained according to the preceding embodiments. For simplicity, only one task NN 631 is illustrated, but any number of task NNs can be used. In one or more embodiments, a rate loss can be calculated based on the output of the probabilistic model 640. In one or more embodiments, the rate loss provides an approximation of the bit rate required to encode the input video data. In one or more embodiments, a task loss can be calculated based on the output of the task NN 631.

[0194] In one or more embodiments, the rate loss and task loss can then be used to train neural networks used in the system, such as a neural network encoder 650, a probabilistic model 640, and a neural network decoder 660. In one or more embodiments, training can be performed by first calculating the gradient of each loss relative to the neural network that contributes to or influences the calculation of that loss. In one or more embodiments, the gradients are then used by an optimization method (such as Adam) to update the trainable parameters of the neural network.

[0195] In one or more embodiments, the machine tasks may be performed on the decoder side rather than the encoder side. For example, this could be because the encoder-side device lacks the capability (computation, power, memory) to run the neural networks that perform these tasks. As another example, this could be because some aspects or performance of the task neural network have changed or improved when the decoder-side device requires the task results (e.g., different or additional semantic classes, better neural network architecture). In one or more embodiments, there may be a need for customization, where different clients will run different neural networks to perform these machine learning tasks.

[0196] Now go to Figure 8 An example dense split attention (DSA) block 800 is provided according to an example embodiment, wherein one type of NN layer is a Resblock 810 including NN layers. In one or more embodiments, the DSA 800 is an attention block that estimates one or more attention maps and applies the one or more attention maps to one or more data tensors. In one or more embodiments, the attention map can be a vector, matrix, or tensor. In one example, the attention map can have values ​​in the range [0, 1]. In one or more embodiments, the one or more data tensors can include one or more input tensors to the attention block 800, one or more feature maps extracted within the attention block 800, and / or one or more feature maps extracted outside the attention block 800. In one or more embodiments, applying one or more attention maps to one or more data tensors can include, for example, multiplying the values ​​of one or more attention maps by one or more data tensors using element-wise multiplication. Other operations may also be considered in one or more embodiments.

[0197] In one or more embodiments, DSA block 800 includes extracting features from its input based at least on one or more initial neural network layers. In one or more embodiments, DSA block 800 further includes splitting the extracted features across a channel axis to obtain two split features. In one or more embodiments, DSA block 800 further includes summing the split features. In one or more embodiments, DSA block 800 further includes performing a global averaging operation on the summed features. In one or more embodiments, DSA block 800 further includes processing the output of the global averaging operation based at least on one or more neural network layers. In one or more embodiments, DSA block 800 further includes inputting the result of this processing into a Softmax operation. In one or more embodiments, DSA block 800 further includes splitting the result of the Softmax operation across a channel axis to obtain two attention tensors. In one or more embodiments, DSA block 800 further includes multiplying the two attention tensors with two previously determined split features to obtain two attention split features. In one or more embodiments, DSA block 800 further includes summing the split features of the two attentions. In one or more embodiments, DSA block 800 further includes concatenating the summed attention split features with features determined at least based on one or more initial neural network layers. In one or more embodiments, the DSA block 800 further includes processing the cascaded results through at least one or more neural network layers. In one or more embodiments, the DSA block 800 further includes adding the output of the processing to the input of the DSA block 800 to obtain the output of the DSA block 800. In one or more embodiments, the global averaging operation may be a global pooling (average pooling) operation used to compute the average value of patches of a feature map. In one or more embodiments, the global averaging operation may aggregate spatial information of the feature map into a single channel to help utilize inter-channel relationships of the features.

[0198] In one or more embodiments, the DSA block 800 can be used as one of the blocks in an image encoder and image decoder in an end-to-end learning codec. In the image encoder and image decoder, the DSA block can follow one or more convolutional layers or one or more transposed convolutional layers.

[0199] Now go to Figure 9An intra-frame codec 900 for end-to-end (e2e) learning is provided for optimizing rate distortion performance. The example embodiments described herein relate to an end-to-end learning intra-frame codec. However, at least some of the embodiments described herein can also be applied to end-to-end learning inter-frame codecs for video compression, or to end-to-end learning video codecs for compressing both intra-frame and inter-frame frames. The intra-frame codec 900 includes a first-step codec 905 and a second-step codec 910, wherein the output of the first-step codec is the input of the second-step codec. In one or more embodiments, the e2e intra-frame codec can process input frames independently without using any information from other frames. In one or more example embodiments, YUV is the format used for data input to the first-step codec 905. In embodiments using the YUV format, 'Y' represents luminance or 'luma' value; 'UV' represents color or 'chroma' value. In one example, the input image can be a YUV 4:4:4 color format image, represented as a 256×256×3 three-dimensional array (or tensor), with 256 pixels horizontally and 256 pixels vertically, and three channels corresponding to the Y, U, and V components. In another example, the input image can be a YUV 4:2:0 color format image, represented by a combination of a 256×256 matrix for the luminance component and a 128×128×2 two-dimensional array (or tensor) for the chrominance component. However, the proposed embodiments can also be extended to other formats, such as RGB.

[0200] In one or more embodiments, the end-to-end learned intra-frame codec 900 includes components including image encoders 916 and 934, a quantizer, probabilistic models 924 and 942, an entropy encoder 920 and 936, an entropy decoder 922 and 940, an inverse quantizer, and image decoders 928 and 938. Entropy encoders 920 and 936 may include lossless encoders, such as arithmetic encoders. In one or more embodiments, entropy decoders 922 and 940 may include lossless decoders, such as arithmetic decoders. Image encoders 916 and 934, probabilistic models 924 and 942, and image decoders 928 and 938 may include neural network components. In one or more embodiments, the quantizer, entropy encoder / decoder, and inverse quantizer may include neural network components.

[0201] In one or more embodiments, the codec includes an encoder and a decoder. For example, a first-step codec 905 includes an encoder 912 and a decoder 914. In one or more embodiments, the encoder 912 may include an image encoder 916, a quantizer, a probability model 924, and an entropy encoder 920. In one or more embodiments, the decoder may include an entropy decoder 922, a probability model 924, an inverse quantizer, and an image decoder 928. In the illustrated embodiments, the probability model 924 in the encoder and the probability model 924 in the decoder are the same probability model. In this context, "same" can refer to two instances of the probability model, one probability model being a copy of another probability model; it can refer to two identical probability models; or it can refer to both probability models being embodied in the same instance of the probability model. Similarly, this use of "same" can be used to refer to the auxiliary encoder and the quantization latent tensor, which are the same instance, copy, or embodiment. For simplicity, the same instance of the probability model and the auxiliary encoder is referred to as... Figure 9 The same numbers are used to mark them.

[0202] In one or more embodiments, the intra-frame codec includes two steps for encoding input data: a first-step codec 905 and a second-step codec 910. In one or more embodiments, the first-step codec includes an encoder 912 that takes real data as input and outputs a first bitstream. The real data may be data to be compressed. However, in addition to real data, the codec (such as the first-step codec) may also take other inputs, such as an indication of the desired quality of the reconstructed data, an indication of the desired bit rate of the encoded data, or an indication of the characteristics of the real data, such as its resolution (in the case of image or video data). In one or more embodiments, the first-step codec includes a decoder 914 that takes the first bitstream as input and outputs an initial reconstruction of the input data. In one or more embodiments, the encoder 930 of the second-step codec 910 acquires a residual, wherein the residual is calculated based on the real data and the reconstruction of the first-step codec. In one or more embodiments, the probabilistic model 942 of the second-step codec 910 takes the reconstruction of the first-step codec 905 as an auxiliary input. In one or more embodiments, the encoder 930 of the second-step codec 910 outputs a second bitstream. In one or more embodiments, the decoder 932 of the second-step codec 910 takes the second bitstream as input and reconstructs the residual. In one or more embodiments, the reconstructed residual from the second-step codec 910 is added to the reconstruction from the first-step codec 910 to obtain the true final reconstruction. In one or more embodiments, the sum of the bitstream output by the encoder 912 of the first-step codec 905 (i.e., the first bitstream) and the bitstream output by the encoder 930 of the second-step codec 910 (i.e., the second bitstream) represents the encoded truth.

[0203] This embodiment provides technical advantages over previous encoder-decoder embodiments. In one or more embodiments, by utilizing the auxiliary input of the probabilistic model of the second-step encoder-decoder 910, the second-step encoder-decoder is conditioned on the initial reconstruction of the input data, which improves the rate distortion performance of the second-step encoder-decoder 910. Therefore, the rate distortion performance of the entire encoder-decoder 900 (the combination of the first-step encoder-decoder 905 and the second-step encoder-decoder 910) is improved.

[0204] In one or more embodiments, the e2e-learned intra-frame codec 900 can be used as part of a video codec, wherein the intra-frame codec can encode one or more first frames (e.g., intra-frames) of the video independently of any other frame, and another codec including an inter-frame codec can encode one or more second frames of the video based at least on one or more third frames, wherein the one or more third frames can include zero or more of the first frames and / or zero or more of the second frames. However, in one or more embodiments, the e2e-learned intra-frame codec 900 can be used to encode a subset of all frames encoded independently of any other frame (i.e., a subset of all intra-frames in the video sequence). In one example, the e2e-learned intra-frame codec 900 is used to encode a first subset of all intra-frames in the video, a conventional intra-frame codec is used to encode a second subset of all intra-frames in the video, and a conventional inter-frame codec is used to encode all inter-frames in the video.

[0205] In one or more embodiments, at encoder 912, image encoder 916 takes an image as input and converts the image into a latent tensor. In one or more embodiments, given a predefined set of quantization levels, a quantizer quantizes the latent tensor into discrete values ​​to obtain a quantized latent tensor. In one or more embodiments, for each symbol or element in the quantized latent tensor, probability model 924 estimates the probability distribution of all possible values ​​based on a context constructed from available information from the current encoding / decoding state. In one or more embodiments, arithmetic encoder 920 encodes the input symbols or elements (losslessly) into a bitstream based at least on the estimated probability distribution. In one or more embodiments, this series of steps includes image compression / encoding, and the resulting bitstream represents a compressed or encoded image.

[0206] In one or more embodiments, at decoder 914, arithmetic decoder 922 and probabilistic model 924 first decode the bitstream to recover the quantized latent tensor. In one or more embodiments, the probabilistic model 924 used at decoder 914 may need to be the same as the probabilistic model 924 used on the encoder side. In one or more embodiments, the dequantizer then reconstructs the latent tensor with continuous values ​​and passes it to image decoder 928 to obtain a reconstruction of the input image. In one or more embodiments, the process at decoder 914 describes image decompression / decoding, and the resulting image represents a reconstructed, decoded, or decompressed image (these terms may be used synonymously in at least some of these embodiments).

[0207] In one or more embodiments, the learned intra-frame codec 900 can be enabled by... D + λR Minimize is trained in an end-to-end manner, where D It is a distortion loss item. R It is a rate loss term. λ These are weights used to control the balance between these two losses. In one or more embodiments, the applied optimization process results in a rate distortion tradeoff, where in distortion... D With rate loss R Finding a balance between them. In one or more embodiments, rate loss... R It can indicate the bit rate of the encoded image, and distortion can indicate pixel fidelity distortion, such as mean squared error (MSE), multi-scale structural similarity (MS-SSIM), multiple distortion losses (such as a weighted sum of MSE and MS-SSIM), or other metrics used to evaluate the quality of the reconstructed image.

[0208] In one or more embodiments, distortion may be related to the performance of one or more machine analysis tasks or the estimated performance of one or more machine analysis tasks. In one or more embodiments, the one or more machine analysis tasks may include object detection, image segmentation, instance segmentation, etc. In one or more embodiments, the estimated performance of one or more machine analysis tasks may include distortion calculated based at least on a first set of features extracted from the output of decoder 914 or 932 and a second set of features extracted from corresponding real data, wherein the first set of features and the second set of features are output from one or more layers of a pre-trained feature extraction neural network.

[0209] In one or more embodiments, optimization or training can be performed jointly for the distortion loss D and the rate loss R. In one or more embodiments, optimization or training can be performed in two alternating phases, wherein in the first phase of these two alternating phases, only the distortion loss D can be used, and in the second phase of these two alternating phases, only the rate loss R can be used.

[0210] like Figure 9 As shown, in one or more embodiments, the intra-frame codec 900 includes a first codec 905 and a second codec 910. The first codec 905 is used to initially encode the input image and may be referred to as the first-step codec, and the second codec 910 is used to encode the ground truth residual (e.g., the difference between the output of the decoder in the first-step codec and the corresponding ground truth data) and may be referred to as the second-step codec. In one or more embodiments, the input to the first-step codec 905 may include ground truth (e.g., a block or the entire image) or a portion of ground truth (e.g., masked ground truth). An example of input data entering the first-step codec 905 is ground truth. For example, ground truth may be an image in YUV 4:4:4 color format, represented as a 256×256×3 multi-dimensional array or tensor, with a horizontal dimension of 256 pixels, a vertical dimension of 256 pixels, and three channels correspondingly for the Y, U, and V components. In another example, the real data could be an image in YUV 4:2:0 color format, represented by a combination of a 256×256 matrix for the luminance component and a 128×128×2 two-dimensional array (or tensor) for the chrominance component. In another example, the real data could consist only of the luminance component, represented as a 256×256×1 one-dimensional array (or tensor). In yet another example, the real data could consist only of the chrominance component, represented as a 128×128×2 two-dimensional array (or tensor). Another example of input data to the first-step codec 905 is partial real data. For example, the input data could be the result of multiplying the real data by a mask, or it could be the output of a Gaussian filter. Furthermore, in one or more example embodiments, the input to the first-step codec 905 could include one or more additional data, such as block / image resolution.

[0211] exist Figure 9 In the example embodiment shown, the input data x It is the true representation of both luminance and chrominance components, among which h × w × 3 describe x The dimensions, height is h Width is w The number of channels is 3 In one or more embodiments, the first-step codec 905 is used to initially encode the input data using the encoder 916. In one or more embodiments, the output of the decoder 914 represents the initial reconstruction of the input data.

[0212] In one or more embodiments, the encoder 912 of the first-step encoder-decoder 905 may include a neural encoder 916, a quantizer, a probability model 924, and an entropy encoder 920. The quantizer... Figure 9 Not shown, but present in one or more embodiments. In one or more embodiments, the neural encoder 916 may include a first convolutional layer ('Conv5×5, 48,1', where conv denotes convolution, 5×5 is the kernel size, 48 is the number of output channels, and 1 is the stride value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. In one or more embodiments, the neural encoder 916 outputs a latent tensor. In one or more embodiments, the latent tensor is converted into a quantized latent tensor by a quantizer. In one or more embodiments, the latent tensor or the quantized latent tensor may be input to a probabilistic model 924, the dimension of which describes... h / / 16 × w / / 16 × 128 ,in h / / 16 Indicates altitude, w / / 16 Indicates width, 128 The number of channels is indicated. In one or more embodiments, the probabilistic model 924 outputs (quantizes) an estimate of the probability of each element of the latent tensor. In one or more embodiments, the probabilistic model 924 can be learned from data using machine learning techniques; for example, the probabilistic model 924 can be a neural network and can be jointly trained with other neural networks in the codec. In one or more embodiments, on the encoder 912 side, the output of the probabilistic model 924 is used as one of the inputs to the entropy encoder 920. In one or more embodiments, the entropy encoder can be an arithmetic encoder. In one or more embodiments, the entropy encoder 920 at least acquires (quantizes) the latent tensor and the output of the probabilistic model 924, and outputs a bitstream. In one or more embodiments, the latent tensor input to the entropy encoder can be quantized first.

[0213] In one or more embodiments, the decoder 914 of the first-step codec 905 may include an entropy decoder 922, a probability model 924, an inverse quantizer, and a neural decoder 928. Figure 9In this embodiment, a dequantizer is not shown, but exists in one or more embodiments. In one or more embodiments, the entropy decoder 922 may be an arithmetic decoder. In one or more embodiments, the entropy decoder 922 acquires at least the bitstream and the output of the probability model 924, and outputs a (quantized) decoded latent tensor. In one or more embodiments, the probability model 924 may need to be the same as the probability model available on the encoder 912 side. In one or more embodiments, the decoded latent tensor may be dequantized. In one or more embodiments, the decoded latent tensor or the dequantized decoded latent tensor is then input to the neural decoder 928. In one or more embodiments, the neural decoder 928 may include a first transposed convolutional layer ('UpConv5×5, 384, 2', where UpConv refers to transposed convolution, 5×5 is the kernel size, 384 is the number of output channels, and 2 is the stride value), a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and convolutional layers. In one or more embodiments, the output of the neural decoder 928 Input data x The initial reconstruction, where the size can be h × w × 3 .

[0214] In one or more embodiments, the input to the second-step codec 910 may include at least the real (e.g., a block or the entire image) and the initial reconstruction from the first-step codec. The residual between them. For example, the input to the second-step codec 910 may include the real data and the initial reconstruction from the first-step codec 905. The residual between them. In one or more embodiments, the real image may be a YUV 4:4:4 color format image, represented as a 256×256×3 multidimensional array or tensor, with a horizontal dimension of 256 pixels, a vertical dimension of 256 pixels, and three channels correspondingly for the Y, U, and V components. In one or more embodiments, the output of the first-step codec 905 This represents the true initial reconstruction, where the dimensions can be 256×256×3. In one or more embodiments, the residual can be the true and the reconstructed. The difference between them can also be 256×256×3 in size. In one or more embodiments, the input to the second-step codec may also include one or more additional data, such as block / image resolution.

[0215] exist Figure 9 In the example embodiment of the second-step codec 910 shown, the input data rIt is the initial reconstruction of the real and first step codec 905. The residual between them. In one or more embodiments, h × w × 3 describe r The dimensions, height is h Width is w The number of channels is 3. In one or more embodiments, the second-step codec 910 is used to encode the true residual information. In one or more embodiments, the bitstream output by the encoder 930 of the second-step codec represents the encoded residual information, and the output of the decoder 932 represents the reconstructed residual. In one or more embodiments, the bitstream from the first-step codec 905 and the bitstream from the second-step codec 910 represent the encoded true information, and the combination (e.g., sum) of the initial reconstruction from the first-step codec 905 and the reconstructed residual from the second-step codec 910 represents the final reconstructed true information.

[0216] In one or more embodiments, the encoder 930 of the second-step encoder-decoder 910 may include a neural encoder 934, an auxiliary encoder 944, and a quantizer ( Figure 9 (Not shown in the diagram) , probability model 942 and entropy encoder 936. In one or more embodiments, similar to the neural encoder 916 of the first-step encoder 905, the neural encoder 934 of the second-step encoder 910 may further include a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. In one or more embodiments, the dimension of the latent tensor output by the neural encoder 934 is determined by... h / / 16 × w / / 16 × 128 Description, where the height is h / / 16 Width is w / / 16 The number of channels is 128.

[0217] In one or more embodiments, the second-step encoder-decoder 910 further includes an auxiliary encoder 944 for generating auxiliary inputs to the probability model 942. In one or more embodiments, the auxiliary encoder 944 is used on both the encoder 930 side and the decoder 932 side. In one or more embodiments, two copies of the same auxiliary encoder 944 may be created, with a first copy used on the encoder 930 side and a second copy used on the decoder 932 side. In one or more embodiments, different auxiliary encoders are used on the encoder 930 side and the decoder 932 side. In one or more embodiments, the input to the auxiliary encoder 944... This is the output of the first-step encoder 905, which is the initial reconstruction of the real data. In one or more embodiments, it is the input of the auxiliary encoder 944. It is a masked version of the initial reconstruction of the real data, which can be obtained via a masking operation performed on the initially reconstructed real data. In one or more embodiments, the masking operation can mask (e.g., set to zero or other predetermined values) some elements of the initial reconstructed truth value, such as elements not present in the residual latent tensor encoded or decoded by the entropy encoder 936 or entropy decoder 940. In one or more embodiments, the input The residual information can first be fed into a predictive neural network to predict residual information, and the predicted residual information can be used as input to an auxiliary encoder 944. In one or more embodiments, the input to the auxiliary encoder 944... This includes the latent tensor output by the entropy decoder 922 of the first-step codec 905, or the dequantized latent tensor output by the dequantizer of the first-step codec 905. In one or more embodiments, the auxiliary encoder 944 may have the same architecture as the neural encoder 934 of the second-step codec 910. However, any suitable architecture for extracting features from an image may be suitable. In one or more embodiments, the output of the auxiliary encoder 944 may be input to the probabilistic model 942 as additional contextual information. In one or more embodiments, the additional contextual information provided to the probabilistic model 942 may include the latent tensor from the first-step codec 905 or the dequantized latent tensor from the first-step codec 905. In one or more embodiments, where the intra-frame codec includes separate luma and chroma codecs, in addition to the possible auxiliary inputs described above, the input of the auxiliary encoder 944... It may also include the reconstructed luminance component, the reconstructed chrominance component, or the corresponding latent tensor. In one or more embodiments where the initial reconstruction consists of sufficient information from the input data, the auxiliary input of the probabilistic model 944 can improve the accuracy of the estimated probability density function of the elements in the quantized latent tensor to be encoded by the entropy encoder 936 or decoded by the entropy decoder 940. This more accurate estimation of the probabilistic model can bring significant performance gains to the encoding of residual information.

[0218] In one or more embodiments, the size of the input to the auxiliary encoder 944 may be larger than the size of the input and output of the second-stage codec 910. For example, when the input image is processed in blocks, data is obtained from a larger region of the reconstructed image by the first-stage codec 905, and the input to the auxiliary encoder 944 may be larger than the block size.

[0219] In one or more embodiments, the input This can be used in super-prior networks. In one or more embodiments, the super-prior network is an additional compression network that provides auxiliary information µ and σ as prior information about parameters of entropy. In one or more embodiments, the auxiliary information µ and σ includes super-prior information. In one or more embodiments, the distribution of elements in the residual information can be similar to the distribution of elements in the initial true reconstruction, which is related to the inputs of the initial true reconstruction. It can be used alone or together with the latent tensor of residual information as input to a super-prior network.

[0220] In one or more embodiments, the latent tensor (e.g., a residual latent tensor) output by the encoder 930 of the second-step encoder-decoder 910 can be input to the probabilistic model 942. In one or more embodiments, the probabilistic model 942 can be a neural network. In one or more embodiments, utilizing auxiliary inputs as additional contextual information, the probabilistic model 942 outputs an estimate of the probability of each element of the residual latent tensor. In one or more embodiments, on the encoder 930 side, the output of the probabilistic model 942 is used as one of the inputs to the entropy encoder 936. In one or more embodiments, the entropy encoder 936 can be an arithmetic encoder. In one or more embodiments, the entropy encoder 936 acquires at least the (quantized) latent tensor and the output of the probabilistic model 942, and outputs a bitstream representing the encoded residual information. In one or more embodiments, the latent tensor input to the entropy encoder 936 is first quantized.

[0221] In one or more embodiments, the decoder 932 of the second-step encoder-decoder 910 may include an entropy decoder 940, a probability model 942, a dequantizer (not shown), an auxiliary encoder 944, and a neural decoder 938. In one or more embodiments, the entropy decoder 940 may be an arithmetic decoder. In one or more embodiments, the entropy decoder 940 acquires at least the bitstream output by the encoder 930 and the output of the probability model 942, and outputs a decoded (quantized) latent tensor. In one or more embodiments, the probability model 942 may need to be the same as the probability model available on the encoder 930 side. In one or more embodiments, the auxiliary encoder 944 may also need to be the same as the auxiliary encoder 944 available on the encoder 930 side. In one or more embodiments, the decoded (quantized) latent tensor may be dequantized. In one or more embodiments, after dequantization, the dequantized decoded latent tensor is input to the neural decoder 932. In one or more embodiments, the dequantized decoded latent tensor may be concatenated with the auxiliary input along the channel dimension. In one or more embodiments, after concatenation, the new decoded latent tensor is input to the neural decoder 938. In one or more embodiments, the neural decoder 938 of the second-step encoder-decoder 910 may include a first transposed convolutional layer, a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and convolutional layers. In one or more embodiments, the output of the neural decoder 938 is a reconstruction of the residual components, the size of which may be... h × w × 3 In one or more embodiments, the reconstructed residuals are compared with the initial reconstruction. Add them together to obtain the final reconstruction.

[0222] In one or more embodiments, in a first training phase, a first-step codec 905 can be trained, and in a second training phase, a second-step codec 910 can be trained by taking the initial reconstruction from the first-step codec 905 as input, wherein in the first training phase, the second-step codec 910 is not trained, and in the second training phase, the first-step codec 905 is not trained. In one or more embodiments, the first-step codec 905 and the second-step codec 910 can be trained together (i.e., the second-step codec 910 takes the output of decoder 914 in the first-step codec 905 as one of its inputs and trains both simultaneously). In one or more embodiments, the first-step codec 905 and the second-step codec 910 can be trained alternately (e.g., the first-step codec 905 is trained for a first number of iterations, then the second-step codec 910 is trained for a second number of iterations, then the first-step codec 905 is trained for a third number of iterations, then the second-step codec 910 is trained for a fourth number of iterations, and so on). Combinations of these embodiments are also possible, wherein, for example, in a first phase, a first-step codec 905 and a second-step codec 910 are trained sequentially (e.g., the first-step codec 905 is trained first, and then the second-step codec 910 is trained), and in a second phase, the first-step codec 905 and the second-step codec 910 are jointly trained or fine-tuned.

[0223] Now go to Figure 10 An illustration of an example system 1000 is provided, in which more than two codecs are used. In one or more embodiments, the codecs for e2e intra-frame learning may include n codecs, where n>=2. Figure 10 In one or more embodiments shown, block 1010 represents the first step codec, and block 1020 represents the second step codec. Following block 1020, the process performed by block 1020 is repeated one or more times, ultimately forming block 10N representing an n-step codec. In one or more embodiments, in each codec, auxiliary data includes the most recently reconstructed ground truth data. In one or more embodiments, the most recently reconstructed ground truth data is derived based on the output of the decoder of a previous codec. In one or more embodiments, the most recently reconstructed ground truth data is the direct output of the decoder of a previous codec. In one or more embodiments, the most recently reconstructed ground truth data is determined based on output data obtained from the decoders of one or more previous codecs. In one or more embodiments, the most recently reconstructed ground truth data is the sum of the previously most recently reconstructed ground truth data and the most recently reconstructed residual data from one or more previous codecs.

[0224] In one or more embodiments of an e2e learning codec comprising more than two encoding steps, for each of the one or more encoding steps, after the first encoding step, a residual is computed based on real data and a reconstruction determined based on the output of the decoder of the previous codec, and this residual can be input to the encoder of the current encoding step. In one or more embodiments, the probabilistic model of each codec, or the auxiliary encoder of each codec, can take one or more of the following as input: the reconstruction determined based on the output of the decoder of the previous codec, the output of the decoder of the previous codec, and the latent tensor determined by the previous codec. In one or more embodiments, the codec may include a first-step codec, a second-step codec, and a third-step codec.

[0225] In one or more embodiments, the input to the first-step codec is an image representing real data. In one or more embodiments, the decoder of the first-step codec outputs a first latent tensor. In one or more embodiments, the decoder of the first-step codec outputs a first reconstructed image. In one or more embodiments, a first residual is calculated as the difference between the first reconstructed image and the real data.

[0226] In one or more embodiments, a first residual is input to a second-step encoder / decoder. In one or more embodiments, the auxiliary encoder of the second-step encoder / decoder may take a first latent tensor, a first reconstructed image, features extracted from the first latent tensor, features extracted from the first reconstructed image, etc., as input. In one or more embodiments, the decoder of the second-step encoder / decoder outputs a second latent tensor. In one or more embodiments, the decoder of the second-step encoder / decoder outputs a first reconstruction residual. In one or more embodiments, the first reconstruction residual is added to the first reconstructed image to obtain a second reconstructed image. In one or more embodiments, the second residual is calculated as the difference between the second reconstructed image and the ground truth value.

[0227] In one or more embodiments, the second residual is input to the third-step codec. In one or more embodiments, the auxiliary encoder of the third-step codec may take the second latent tensor determined by the second-step codec, the second reconstructed image, the first reconstruction residual, features extracted from the second latent tensor, features extracted from the second reconstructed image, and features extracted from the first reconstruction residual as input. In one or more embodiments, the decoder of the third-step codec outputs the second reconstruction residual. In one or more embodiments, the second reconstruction residual is added to the second reconstructed image to obtain the third reconstructed image. In one or more embodiments, the third reconstructed image represents the final reconstructed image of the codec.

[0228] Now go to Figure 11The diagram illustrates an example flowchart of a process 1100 performed by a device embodied in, associated with, or otherwise communicating with (hereinafter generally referred to as embodied in by the service device 22) to receive residual real data and auxiliary data.

[0229] like Figure 11 As shown in box 1110, the apparatus embodied by service device 22 includes components for receiving real data by a first codec, such as processor 12, communication interface 16, etc. In one or more embodiments, the real data includes an image that includes brightness data and color data.

[0230] like Figure 11 As shown in box 1120, the apparatus embodied by the service device 22 includes components for generating a first bit stream based on real data, such as processor 12, communication interface 16, etc.

[0231] like Figure 11 As shown in box 1130, the apparatus embodied by the service device 22 includes components for generating initial reconstruction data based on a first bit stream, such as processor 12, communication interface 16, etc., wherein the initial reconstruction data includes a reconstruction of the real data.

[0232] like Figure 11 As shown in box 1140, the apparatus embodied by the service device 22 includes components for outputting initial reconstruction data from the first codec, such as processor 12, communication interface 16, etc.

[0233] like Figure 11 As shown in block 1150, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for determining residual true data based at least on initial reconstructed data and true data. In one or more embodiments, the residual true data includes the difference between the true data and the initial reconstructed data. In one or more embodiments, the residual true data is determined by a first codec.

[0234] like Figure 11 As shown in block 1160, the apparatus embodied by service device 22 includes components for determining auxiliary data based at least on a first bitstream, or at least on data derived from the first bitstream, such as processor 12, communication interface 16, etc. In one or more embodiments, the auxiliary data includes one or more of initial reconstruction data, a first latent tensor, and a first result latent tensor.

[0235] like Figure 11As shown in block 1170, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for receiving residual real data and auxiliary data by the second codec. In one or more embodiments, the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss. In one or more embodiments, the first codec is trained before the second codec, and wherein the second codec is trained at least based on the first codec or at least based on data generated by the first codec. In one or more embodiments, the first codec and the second codec are trained simultaneously. In one or more embodiments, the first codec and the second codec are trained at alternating intervals.

[0236] like Figure 11 As shown in block 1180, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for generating a second bitstream based on residual real data and auxiliary data. In one or more embodiments, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for determining a combined bitstream based at least on the first bitstream and the second bitstream.

[0237] like Figure 11 As shown in optional box 1190, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for generating residual reconstruction data based on a second bitstream, wherein the residual reconstruction data includes a reconstruction of the residual true data. In one or more embodiments, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data. In one or more embodiments, the combined reconstruction data is determined by a second codec.

[0238] like Figure 11 As shown in optional box 1195, the apparatus embodied by service device 22 includes components for outputting residual reconstruction data from the second codec, such as processor 12, communication interface 16, etc.

[0239] Now go to Figure 12 The diagram illustrates an example flowchart of a process 1200 performed by a device embodied in, associated with, or otherwise communicating with (hereinafter generally referred to as embodied in by the service device 22) to encode real data into a first bit stream.

[0240] like Figure 12 As shown in box 1210, the apparatus embodied by the service device 22 includes components for converting real data into a first potential tensor using a first neural encoder, such as processor 12, communication interface 16, etc.

[0241] like Figure 12 As shown in box 1220, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for generating a first quantization potential tensor using a first quantizer and a first predefined set of quantization levels, at least based on a first potential tensor, wherein the first quantization potential tensor includes at least one symbol or element.

[0242] like Figure 12 As shown in box 1230, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for determining a first estimated probability distribution of possible values ​​using a first probability model for at least one symbol or element in the first quantized potential tensor.

[0243] like Figure 12 As shown in box 1240, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for encoding corresponding symbols or elements of at least one symbol or element of the first quantized potential tensor into a first bit stream using a first entropy encoder, at least based on a first estimated probability distribution of possible values.

[0244] Now go to Figure 13 The diagram illustrates an example flowchart of process 1300 performed by a device embodied in, associated with, or otherwise communicating with service device 22 (hereinafter generally referred to as embodied in service device 22) to decode a first bitstream into initial reconstructed data.

[0245] like Figure 13 As shown in box 1310, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for decoding the first bitstream into a first quantization potential tensor or the same quantization potential tensor as the first quantization potential tensor using a first entropy decoder and a first probability model or a copy of the first probability model.

[0246] like Figure 13 As shown in box 1320, the apparatus embodied by the service device 22 includes components for generating a first result potential tensor using a first dequantizer, such as processor 12, communication interface 16, etc.

[0247] like Figure 13 As shown in box 1330, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for converting a first result potential tensor into initial reconstruction data using a first neural decoder.

[0248] Now go to Figure 14The diagram illustrates an example flowchart of process 1400 performed by a device embodied in, associated with, or otherwise communicating with service device 22 (hereinafter generally referred to as embodied in service device 22) to encode residual real data into a second bit stream.

[0249] like Figure 14 As shown in box 1410, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for converting residual real data into a second potential tensor using a second neural encoder.

[0250] like Figure 14 As shown in box 1420, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for generating a second quantization potential tensor using a second quantizer and a second predefined set of quantization levels, at least based on a second potential tensor, wherein the second quantization potential tensor includes at least one symbol or element.

[0251] like Figure 14 As shown in box 1430, the apparatus embodied by the service device 22 includes components for converting auxiliary data into auxiliary features using an auxiliary encoder, such as processor 12, communication interface 16, etc.

[0252] like Figure 14 As shown in box 1440, the apparatus embodied by service device 22 includes components for inputting auxiliary features into a second probability model, such as processor 12, communication interface 16, etc.

[0253] like Figure 14 As shown in box 1450, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for determining a second estimated probability distribution of possible values ​​using a second probability model for at least one symbol or element in the second quantization potential tensor.

[0254] like Figure 14 As shown in box 1460, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for encoding a corresponding symbol or element from at least one symbol or element of the second quantized potential tensor into a second bitstream using a second entropy encoder, at least based on a second estimated probability distribution of possible values.

[0255] Now go to Figure 15 The diagram illustrates an example flowchart of a process 1500 performed by a device embodied in, associated with, or otherwise communicating with (hereinafter generally referred to as embodied in by the service device 22) to decode a second bitstream into residual reconstruction data.

[0256] like Figure 15 As shown in box 1510, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for decoding the second bitstream into a second quantization potential tensor or a quantization potential tensor that is the same as the second quantization potential tensor using a second entropy decoder, an auxiliary encoder or another auxiliary encoder that is the same as the auxiliary encoder, and a second probability model or another probability model that is the same as the second probability model.

[0257] like Figure 15 As shown in box 1520, the apparatus embodied by the service device 22 includes components for generating a second result potential tensor using a second dequantizer, such as processor 12, communication interface 16, etc.

[0258] like Figure 15 As shown in box 1530, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for converting a second result latent tensor into residual reconstruction data using a second neural decoder. In one or more embodiments, at least one of the first neural encoder, first neural decoder, first probabilistic model, second neural encoder, second neural decoder, second probabilistic model, and auxiliary encoder includes a neural network component.

[0259] Now go to Figure 16 The diagram illustrates an example flowchart of process 1600 performed by a device embodied in, associated with, or otherwise communicating with service device 22 (hereinafter generally referred to as embodied in service device 22) to output additional reconstruction data.

[0260] like Figure 16 As shown in box 1610, the apparatus embodied by the service device 22 includes components for determining additional residual real data based at least on the combined reconstructed data and real data, such as processor 12, communication interface 16, etc.

[0261] like Figure 16 As shown in box 1620, the apparatus embodied by the service device 22 includes components for determining additional auxiliary data based at least on the second bit stream or at least on data derived from the second bit stream, such as processor 12, communication interface 16, etc.

[0262] like Figure 16 As shown in box 1630, the apparatus embodied by service device 22 includes components such as processor 12, communication interface 16, etc., for receiving additional residual real data and additional auxiliary data by different codecs.

[0263] like Figure 16As shown in box 1640, the apparatus embodied by the service device 22 includes components for generating an additional bitstream based at least on additional residual real data and additional auxiliary data, such as processor 12, communication interface 16, etc.

[0264] like Figure 16 As shown in block 1650, the apparatus embodied by service device 22 includes components for generating additional reconstruction data based at least on additional bit streams, such as processor 12, communication interface 16, etc., wherein the additional reconstruction data includes reconstructions of additional residual real data, and wherein the additional reconstruction data is operable to be combined with composite reconstruction data to form composite reconstruction data.

[0265] like Figure 16 As shown in box 1660, the apparatus embodied by the service device 22 includes components such as processor 12, communication interface 16, etc., for outputting at least one of additional reconstruction data and composite reconstruction data from different codecs.

[0266] Now go to Figure 17 The diagram illustrates an example flowchart of process 1700 performed by a device embodied in, associated with, or otherwise communicating with consumer device 24 (hereinafter generally referred to as embodied in consumer device 24) to decode a bitstream into reconstructed data.

[0267] like Figure 17 As shown in box 1710, the apparatus embodied by consumer device 24 includes components for receiving the first bit stream, such as processor 12, communication interface 16, etc.

[0268] like Figure 17 As shown in box 1720, the apparatus embodied by consumer device 24 includes components for generating initial reconstruction data based on a first bit stream, such as processor 12, communication interface 16, etc.

[0269] like Figure 17 As shown in box 1730, the apparatus embodied by consumer device 24 includes components for determining auxiliary data based at least on a first bit stream or at least on data derived from the first bit stream, such as processor 12, communication interface 16, etc.

[0270] like Figure 17 As shown in box 1740, the device embodied by consumer device 24 includes components for outputting initial reconstruction data, such as processor 12, communication interface 16, etc.

[0271] like Figure 17As shown in block 1750, the apparatus embodied by consumer device 24 includes components for receiving a second bitstream, such as processor 12, communication interface 16, etc. In one or more embodiments, the first bitstream and the second bitstream are received as part of a combined bitstream.

[0272] like Figure 17 As shown in box 1760, the apparatus embodied by consumer device 24 includes components for generating residual reconstruction data based on a second bitstream and auxiliary data, such as processor 12, communication interface 16, etc.

[0273] like Figure 17 As shown in block 1770, the apparatus embodied by consumer device 24 includes components for outputting residual reconstruction data, such as processor 12, communication interface 16, etc. In one or more embodiments, the apparatus embodied by consumer device 24 includes components for determining combined reconstruction data based at least on initial reconstruction data and residual reconstruction data, such as processor 12, communication interface 16, etc. In one or more embodiments, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc. for...

[0274] Now go to Figure 18 The diagram illustrates an example flowchart of process 1800 performed by a device embodied in, associated with, or otherwise communicating with consumer device 24 (hereinafter generally referred to as embodied in consumer device 24) to decode a first bitstream into initial reconstructed data.

[0275] like Figure 18 As shown in box 1810, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc., for decoding a first bitstream into a first quantization potential tensor using a first entropy decoder and a first probability model.

[0276] like Figure 18 As shown in box 1820, the apparatus embodied by consumer device 24 includes components for generating a first potential tensor using a first dequantizer and based on a first quantization potential tensor, such as processor 12, communication interface 16, etc.

[0277] like Figure 18 As shown in box 1830, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc., for converting a first potential tensor into initial reconstructed data using a first neural decoder.

[0278] Now go to Figure 19The diagram illustrates an example flowchart of a process 1900 performed by a device embodied in, associated with, or otherwise communicating with consumer device 24 (hereinafter generally referred to as embodied in consumer device 24) to decode a second bitstream into residual reconstruction data.

[0279] like Figure 19 As shown in box 1910, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc., for decoding a second bitstream into a second quantization potential tensor using a second entropy decoder, an auxiliary encoder, and a second probability model.

[0280] like Figure 19 As shown in box 1920, the apparatus embodied by consumer device 24 includes components for generating a second potential tensor using a second dequantizer and based on a second quantization potential tensor, such as processor 12, communication interface 16, etc.

[0281] like Figure 19 As shown in box 1930, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc., for converting a second latent tensor into residual reconstruction data using a second neural decoder. In one or more embodiments, at least one of the first neural decoder, the first probabilistic model, the second neural decoder, and the second probabilistic model includes a neural network component.

[0282] Now go to Figure 20 The diagram illustrates an example flowchart of a process 2000 performed by a device embodied in, associated with, or otherwise communicating with consumer device 24 (hereinafter generally referred to as embodied in consumer device 24) to decode an additional bitstream into additional reconstructed data.

[0283] like Figure 20 As shown in box 2010, the apparatus embodied by consumer device 24 includes components for receiving additional bit streams, such as processor 12, communication interface 16, etc.

[0284] like Figure 20 As shown in box 2020, the apparatus embodied by consumer device 24 includes components such as processor 12, communication interface 16, etc., for generating additional reconstruction data based on additional bit streams, wherein the additional reconstruction data is operable to be combined with composite reconstruction data to form composite reconstruction data.

[0285] like Figure 20 As shown in box 2030, the apparatus embodied by consumer device 24 includes components for outputting at least one of additional reconstruction data and composite reconstruction data, such as processor 12, communication interface 16, etc.

[0286] Figures 11-20 A flowchart illustrating a method according to an exemplary embodiment of the present disclosure is shown. It should be understood that each block of the flowchart and combinations of blocks in the flowchart can be implemented in various ways, such as hardware, firmware, processors, circuit systems, and / or other communication devices associated with the execution of software including one or more computer program instructions. For example, one or more processes in the above-described process can be implemented by computer program instructions. In this regard, computer program instructions embodying the above-described processes can be stored by memory device 14 of the apparatus employing the embodiment and executed by processor 12. As will be understood, any such computer program instructions can be loaded into a computer or other programmable device (e.g., hardware) to produce a machine such that the resulting computer or other programmable device performs the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable storage medium that can instruct the computer or other programmable device to operate in a particular manner, such that the instructions stored in the computer programmable storage medium produce an article of art whose execution performs the functions specified in the flowchart blocks. The computer program instructions can also be loaded into a computer or other programmable device to cause a series of operations to be performed on the computer or other programmable device, thereby producing a computer-implemented process, such that the instructions executing on the computer or other programmable device provide operations for implementing the functions specified in the flowchart blocks.

[0287] Therefore, the boxes in a flowchart support combinations of components used to perform a specified function and combinations of operations used to perform a specific function. It will also be understood that one or more boxes in a flowchart, as well as combinations of boxes in a flowchart, can be implemented by a dedicated hardware-based computer system or a combination of dedicated hardware and computer instructions to perform the specified function.

[0288] With the help of the teachings presented in the foregoing description and the accompanying drawings, those skilled in the art to which this disclosure pertains will be able to conceive of many modifications and other embodiments described herein. Therefore, it should be understood that this disclosure is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims.

[0289] Furthermore, although the foregoing description and related figures describe exemplary embodiments in the context of certain example combinations of elements and / or functions, it should be understood that alternative embodiments may provide different combinations of elements and / or functions without departing from the scope of the appended claims. In this regard, for example, as described in some of the appended claims, combinations of elements and functions different from those explicitly described above may also be contemplated. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes.

Claims

1. A computer-implemented method, comprising: The first codec receives the actual data; Based on the real data, a first bitstream is generated; Initial reconstruction data is generated based on the first bitstream, wherein the initial reconstruction data includes the reconstruction of the real data; The initial reconstruction data is output by the first codec; The residual true data is determined at least based on the initial reconstruction data and the true data; Auxiliary data is determined based at least on the first bit stream, or at least on data derived from the first bit stream; The residual real data and the auxiliary data are received by the second codec; as well as A second bitstream is generated based on the residual real data and the auxiliary data.

2. The computer-implemented method according to claim 1, further comprising: Residual reconstruction data is generated based on the second bit stream, wherein the residual reconstruction data includes the reconstruction of the actual residual data; as well as The residual reconstruction data is output by the second codec.

3. The computer-implemented method according to any one of claims 1 to 2, further comprising: The combined reconstruction data is determined based at least on the initial reconstruction data and the residual reconstruction data.

4. The computer-implemented method according to any one of claims 1 to 3, further comprising: The combined bit stream is determined based at least on the first bit stream and the second bit stream.

5. The computer-implemented method according to any one of claims 1 to 4, wherein the residual true data comprises: The difference between the real data and the initial reconstructed data.

6. The computer-implemented method according to any one of claims 1 to 5, wherein the real data includes an image, the image including brightness data and color data.

7. The computer-implemented method according to any one of claims 1 to 6, wherein generating the first bitstream comprises: The real data is converted into a first latent tensor using a first neural encoder; Using a first quantizer and a first predefined set of quantization levels, a first quantization potential tensor is generated based at least on the first potential tensor, wherein the first quantization potential tensor includes at least one symbol or element; For a given symbol or element among the at least one symbol or element of the first quantized potential tensor, a first probability model is used to determine a first estimated probability distribution of possible values; as well as Using a first entropy encoder, the corresponding symbol or element of the at least one symbol or element of the first quantized potential tensor is encoded into the first bitstream based at least on a first estimated probability distribution of the possible values.

8. The computer-implemented method according to any one of claims 1 to 7, wherein generating the initial reconstruction data comprises: Using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream is decoded into the first quantization potential tensor or a quantization potential tensor that is the same as the first quantization potential tensor. Use the first dequantizer to generate the first result potential tensor; as well as The first result potential tensor is converted into the initial reconstructed data using a first neural decoder.

9. The computer-implemented method according to any one of claims 1 to 8, wherein generating the second bitstream comprises: The residual real data is converted into a second latent tensor using a second neural encoder. A second quantization potential tensor is generated using a second quantizer and a second predefined set of quantization levels, at least based on the second potential tensor, wherein the second quantization potential tensor includes at least one symbol or element; The auxiliary data is converted into auxiliary features using an auxiliary encoder; The auxiliary features are input into the second probability model; For a given symbol or element among the at least one symbol or element of the second quantized potential tensor, a second probability model is used to determine a second estimated probability distribution of possible values; as well as Using a second entropy encoder, the corresponding symbol or element from at least one symbol or element of the second quantized potential tensor is encoded into the second bitstream based at least on a second estimated probability distribution of the possible values.

10. The computer-implemented method according to any one of claims 1 to 9, wherein generating the residual reconstruction data comprises: The second bitstream is decoded into the second quantization potential tensor, or the same quantization potential tensor, using the second entropy decoder, the auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and the second probability model or another probability model identical to the second probability model. Use a second dequantizer to generate the second result potential tensor; as well as The second result potential tensor is converted into the residual reconstruction data using a second neural decoder.

11. The computer-implemented method according to any one of claims 7 to 10, wherein at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises: Neural network components.

12. The computer-implemented method according to any one of claims 1 to 11, wherein the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

13. The computer-implemented method of claim 12, wherein the first codec is trained prior to the second codec, and wherein the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

14. The computer-implemented method of claim 12, wherein the first codec and the second codec are trained simultaneously.

15. The computer-implemented method of claim 12, wherein the first codec and the second codec are trained at alternating intervals.

16. The computer-implemented method according to any one of claims 1 to 15, wherein the auxiliary data includes one or more of the following: the initial reconstruction data, the first latent tensor, and the first result latent tensor.

17. The computer-implemented method according to any one of claims 1 to 16, wherein the residual true data is determined by the first codec.

18. The computer-implemented method according to any one of claims 3 to 17, wherein the combined reconstructed data is determined by the second codec.

19. The computer-implemented method according to any one of claims 3 to 18, further comprising: The additional residual true data is determined based at least on the combined reconstructed data and the true data; Additional auxiliary data are determined based at least on the second bitstream, or at least on data derived from the second bitstream; The additional residual real data and the additional auxiliary data are received by different codecs; The additional bitstream is generated based at least on the additional residual real data and the additional auxiliary data; Additional reconstructed data is generated based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual true data, and wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and The different codecs output at least one of the additional reconstruction data and the composite reconstruction data.

20. A computer-implemented method, comprising: Receive the first bit stream; Based on the first bit stream, generate initial reconstruction data; Auxiliary data is determined based at least on the first bit stream, or at least on data derived from the first bit stream; Output the initial reconstruction data; Receive the second bit stream; Based on the second bitstream and the auxiliary data, residual reconstruction data is generated; as well as Output the residual reconstruction data.

21. The computer-implemented method according to claim 20, further comprising: The combined reconstruction data is determined based at least on the initial reconstruction data and the residual reconstruction data.

22. The computer-implemented method according to any one of claims 20 to 21, wherein the first bit stream and the second bit stream are received as part of a combined bit stream.

23. The computer-implemented method according to any one of claims 20 to 22, wherein generating initial reconstruction data comprises: The first bitstream is decoded into a first quantized potential tensor using a first entropy decoder and a first probability model; The first potential tensor is generated using the first dequantizer and based on the first quantized potential tensor; as well as The first potential tensor is converted into the initial reconstructed data using a first neural decoder.

24. The computer-implemented method according to any one of claims 20 to 23, wherein generating residual reconstruction data comprises: The second bitstream is decoded into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probability model. The second dequantizer is used and the second potential tensor is generated based on the second quantized potential tensor; as well as The second potential tensor is converted into the residual reconstruction data using a second neural decoder.

25. The computer-implemented method according to any one of claims 20 to 24, wherein at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises: Neural network components.

26. The computer-implemented method according to any one of claims 21 to 25, further comprising: Receive additional bit stream; Additional reconstructed data is generated based on the additional bitstream, wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and Output at least one of the additional reconstruction data and the composite reconstruction data.

27. An apparatus comprising: At least one processor; as well as At least one memory storing instructions, which, when executed by the at least one processor, cause the means to perform at least the following: The first codec receives the actual data; Based on the real data, a first bitstream is generated; Initial reconstruction data is generated based on the first bitstream, wherein the initial reconstruction data includes the reconstruction of the real data; The initial reconstruction data is output by the first codec; The residual true data is determined at least based on the initial reconstruction data and the true data; Auxiliary data is determined based at least on the first bit stream, or at least on data derived from the first bit stream; The residual real data and the auxiliary data are received by the second codec; as well as A second bitstream is generated based on the residual real data and the auxiliary data.

28. The apparatus of claim 27, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: Residual reconstruction data is generated based on the second bitstream, wherein the residual reconstruction data includes a reconstruction of the actual residual data; and The residual reconstruction data is output by the second codec.

29. The apparatus according to any one of claims 27 to 28, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The combined reconstruction data is determined based at least on the initial reconstruction data and the residual reconstruction data.

30. The apparatus according to any one of claims 27 to 29, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The combined bit stream is determined based at least on the first bit stream and the second bit stream.

31. The apparatus according to any one of claims 27 to 30, wherein the residual true data comprises: The difference between the real data and the initial reconstructed data.

32. The apparatus according to any one of claims 27 to 31, wherein the real data includes an image, the image including brightness data and color data.

33. The apparatus of claims 27 to 32, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The real data is converted into a first latent tensor using a first neural encoder; Using a first quantizer and a first predefined set of quantization levels, a first quantization potential tensor is generated based at least on the first potential tensor, wherein the first quantization potential tensor includes at least one symbol or element; For a corresponding symbol or element among the at least one symbol or element of the first quantized potential tensor, a first probability model is used to determine a first estimated probability distribution of possible values; and Using a first entropy encoder, the corresponding symbol or element of the at least one symbol or element of the first quantized potential tensor is encoded into the first bitstream based at least on a first estimated probability distribution of the possible values.

34. The apparatus according to any one of claims 27 to 33, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: Using a first entropy decoder and the first probability model or a copy of the first probability model, the first bit stream is decoded into the first quantization potential tensor or a quantization potential tensor that is the same as the first quantization potential tensor. Use the first dequantizer to generate the first result potential tensor; as well as The first result potential tensor is converted into the initial reconstructed data using a first neural decoder.

35. The apparatus according to any one of claims 27 to 34, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The residual real data is converted into a second latent tensor using a second neural encoder. A second quantization potential tensor is generated using a second quantizer and a second predefined set of quantization levels, at least based on the second potential tensor, wherein the second quantization potential tensor includes at least one symbol or element; The auxiliary data is converted into auxiliary features using an auxiliary encoder; The auxiliary features are input into the second probability model; For a given symbol or element among the at least one symbol or element of the second quantized potential tensor, a second probability model is used to determine a second estimated probability distribution of possible values; as well as Using a second entropy encoder, the corresponding symbol or element from at least one symbol or element of the second quantized potential tensor is encoded into the second bitstream based at least on a second estimated probability distribution of the possible values.

36. The apparatus according to any one of claims 27 to 35, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: Using the second entropy decoder, the auxiliary encoder, and the second probability model or a copy of the second probability model, the second bitstream is decoded into the second quantization potential tensor, or a quantization potential tensor identical to the second quantization potential tensor; Use a second dequantizer to generate the second result potential tensor; as well as The second result potential tensor is converted into the residual reconstruction data using a second neural decoder.

37. The apparatus according to any one of claims 33 to 36, wherein at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises: Neural network components.

38. The apparatus according to any one of claims 27 to 37, wherein the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

39. The apparatus of claim 38, wherein the first codec is trained prior to the second codec, and wherein the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

40. The apparatus of claim 38, wherein the first codec and the second codec are trained simultaneously.

41. The apparatus of claim 38, wherein the first codec and the second codec are trained at alternating intervals.

42. The apparatus according to any one of claims 27 to 41, wherein the auxiliary data includes one or more of the following: the initial reconstruction data, the first latent tensor, and the first result latent tensor.

43. The apparatus according to any one of claims 27 to 42, wherein the residual true data is determined by the first codec.

44. The apparatus according to any one of claims 29 to 43, wherein the combined reconstructed data is determined by the second codec.

45. The apparatus according to any one of claims 29 to 44, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The additional residual true data is determined based at least on the combined reconstructed data and the true data; Additional auxiliary data are determined based at least on the second bitstream, or at least on data derived from the second bitstream; The additional residual real data and the additional auxiliary data are received by different codecs; The additional bitstream is generated based at least on the additional residual real data and the additional auxiliary data; Additional reconstructed data is generated based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual true data, and wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and The additional reconstructed data is output by different codecs.

46. ​​An apparatus comprising: At least one processor; as well as At least one memory storing instructions, which, when executed by the at least one processor, cause the means to perform at least the following: Receive the first bit stream; Based on the first bit stream, generate initial reconstruction data; Auxiliary data is determined based at least on the first bit stream, or at least on data derived from the first bit stream; Output the initial reconstruction data; Receive the second bit stream; Based on the second bitstream and the auxiliary data, residual reconstruction data is generated; as well as Output the residual reconstruction data.

47. The apparatus of claim 46, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The combined reconstruction data is determined based at least on the initial reconstruction data and the residual reconstruction data.

48. The apparatus according to any one of claims 46 to 47, wherein the first bit stream and the second bit stream are received as part of a combined bit stream.

49. The apparatus according to any one of claims 46 to 48, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The first bitstream is decoded into a first quantized potential tensor using a first entropy decoder and a first probability model; The first potential tensor is generated using the first dequantizer and based on the first quantized potential tensor; as well as The first potential tensor is converted into the initial reconstructed data using a first neural decoder.

50. The apparatus according to any one of claims 46 to 49, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: The second bitstream is decoded into a second quantized latent tensor using a second entropy decoder, an auxiliary encoder, and a second probability model. The second dequantizer is used and the second potential tensor is generated based on the second quantized potential tensor; as well as The second potential tensor is converted into the residual reconstruction data using a second neural decoder.

51. The apparatus according to any one of claims 46 to 50, wherein at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises: Neural network components.

52. The apparatus according to any one of claims 46 to 51, wherein the at least one memory and the computer program code are further configured, together with the at least one processor, such that the apparatus: Receive additional bit stream; Additional reconstructed data is generated based on the additional bitstream, wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and Output the additional reconstruction data.

53. A non-transitory computer-readable storage medium comprising computer instructions that, when executed by a device, cause the device to perform the method according to any one of claims 1 to 19.

54. A non-transitory computer-readable storage medium comprising computer instructions that, when executed by a device, cause the device to perform the method according to any one of claims 20 to 26.

55. An apparatus comprising: A component used to receive real data by the first codec; A component used to generate a first bitstream based on the real data; A component for generating initial reconstruction data based on the first bitstream, wherein the initial reconstruction data includes a reconstruction of the real data; Components for outputting the initial reconstructed data by the first codec; Components for determining residual true data based at least on the initial reconstruction data and the true data; A component for determining auxiliary data based at least on the first bit stream, or at least on data derived from the first bit stream; A component for receiving the residual real data and the auxiliary data by a second codec; as well as A component for generating a second bitstream based on the residual real data and the auxiliary data.

56. The apparatus of claim 55, further comprising: A component for generating residual reconstruction data based on the second bit stream, wherein the residual reconstruction data includes a reconstruction of the actual residual data; as well as A component used to output the residual reconstruction data by the second codec.

57. The apparatus according to any one of claims 55 to 56, further comprising: Components used to determine combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

58. The apparatus according to any one of claims 55 to 57, further comprising: A component for determining a combined bit stream based at least on the first bit stream and the second bit stream.

59. The apparatus according to any one of claims 55 to 58, wherein the residual true data includes the difference between the true data and the initial reconstructed data.

60. The apparatus according to any one of claims 55 to 59, wherein the real data includes an image, the image including luminance data and color data.

61. The apparatus according to any one of claims 55 to 60, further comprising: A component for converting the real data into a first potential tensor using a first neural encoder; A component for generating a first quantization potential tensor using a first quantizer and a first predefined set of quantization levels, at least based on the first potential tensor, wherein the first quantization potential tensor includes at least one symbol or element; A component for determining a first estimated probability distribution of possible values ​​using a first probability model for a corresponding symbol or element among the at least one symbol or element of the first quantized potential tensor; as well as A component for encoding, using a first entropy encoder, at least based on a first estimated probability distribution of the possible values, the corresponding symbol or element of the at least one symbol or element of the first quantized potential tensor into the first bitstream.

62. The apparatus according to any one of claims 55 to 61, further comprising: A component for decoding the first bitstream into the first quantization potential tensor or the same quantization potential tensor as the first quantization potential tensor using a first entropy decoder and the first probability model or a copy of the first probability model; The component used to generate the first result potential tensor using the first dequantizer; as well as A component for converting the first result potential tensor into the initial reconstructed data using a first neural decoder.

63. The apparatus according to any one of claims 55 to 62, further comprising: Components for converting the residual real data into a second potential tensor using a second neural encoder; A component for generating a second quantization potential tensor using a second quantizer and a second predefined set of quantization levels, at least based on the second potential tensor, wherein the second quantization potential tensor includes at least one symbol or element; Components for converting the auxiliary data into auxiliary features using an auxiliary encoder; A component used to input the auxiliary features into the second probability model; A component for determining a second estimated probability distribution of possible values ​​using a second probability model for a corresponding symbol or element among the at least one symbol or element of the second quantized potential tensor; as well as A component for encoding, using a second entropy encoder at least based on a second estimated probability distribution of the possible values, the corresponding symbol or element of the at least one symbol or element of the second quantized potential tensor into the second bitstream.

64. The apparatus according to any one of claims 55 to 63, further comprising: Components for using a second entropy decoder, the auxiliary encoder or another auxiliary encoder identical to the auxiliary encoder, and the second probability model or another probability model identical to the second probability model to decode the second bitstream into the second quantization potential tensor, or a quantization potential tensor identical to the second quantization potential tensor; The component used to generate the second result potential tensor using the second dequantizer; as well as A component for converting the second result potential tensor into the residual reconstruction data using a second neural decoder.

65. The apparatus according to any one of claims 61 to 64, wherein at least one of the first neural encoder, the first neural decoder, the first probability model, the second neural encoder, the second neural decoder, the second probability model, and the auxiliary encoder comprises: Neural network components.

66. The apparatus according to any one of claims 55 to 65, wherein the first codec and the second codec are trained end-to-end by reducing at least one of distortion loss and rate loss.

67. The apparatus of claim 66, wherein the first codec is trained prior to the second codec, and wherein the second codec is trained at least based on the first codec or at least based on data generated by the first codec.

68. The apparatus of claim 66, wherein the first codec and the second codec are trained simultaneously.

69. The apparatus of claim 66, wherein the first codec and the second codec are trained at alternating intervals.

70. The apparatus according to any one of claims 55 to 69, wherein the auxiliary data includes one or more of the following: the initial reconstruction data, the first latent tensor, and the first result latent tensor.

71. The apparatus according to any one of claims 55 to 70, wherein the residual true data is determined by the first codec.

72. The apparatus according to any one of claims 57 to 71, wherein the combined reconstructed data is determined by the second codec.

73. The apparatus according to any one of claims 57 to 72, further comprising: Components for determining additional residual real data based at least on the combined reconstructed data and the real data; A component for determining additional auxiliary data based at least on the second bit stream, or at least on data derived from the second bit stream; A component for receiving the additional residual real data and the additional auxiliary data by different codecs; Components for generating additional bitstreams based at least on the additional residual real data and the additional auxiliary data; Components for generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data includes a reconstruction of the additional residual true data, and wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and A component for outputting at least one of the additional reconstruction data and the composite reconstruction data by the different codecs.

74. An apparatus comprising: A component used to receive the first bit stream; Components used to generate initial reconstruction data based on the first bit stream; A component for determining auxiliary data based at least on the first bit stream, or at least on data derived from the first bit stream; Components used to output the initial reconstruction data; A component used to receive the second bit stream; Components for generating residual reconstruction data based on the second bit stream and the auxiliary data; as well as Components used to output the residual reconstruction data.

75. The apparatus of claim 74, further comprising: Components used to determine combined reconstruction data based at least on the initial reconstruction data and the residual reconstruction data.

76. The apparatus according to any one of claims 74 to 75, wherein the first bit stream and the second bit stream are received as part of a combined bit stream.

77. The apparatus according to any one of claims 74 to 76, further comprising: A component for decoding the first bitstream into a first quantized potential tensor using a first entropy decoder and a first probability model; A component for generating a first potential tensor using a first dequantizer and based on the first quantized potential tensor; as well as A component for converting the first potential tensor into the initial reconstructed data using a first neural decoder.

78. The apparatus according to any one of claims 74 to 77, further comprising: Components for using a second entropy decoder, an auxiliary encoder, and a second probability model to decode the second bitstream into a second quantization potential tensor; A component for generating a second potential tensor using a second dequantizer and based on the second quantized potential tensor; as well as A component for using a second neural decoder to convert the second potential tensor into the residual reconstruction data.

79. The apparatus according to any one of claims 74 to 78, wherein at least one of the first neural decoder, the first probability model, the second neural decoder, and the second probability model comprises: Neural network components.

80. The apparatus according to any one of claims 74 to 79, further comprising: A component used to receive additional bit streams; Components for generating additional reconstructed data based on the additional bitstream, wherein the additional reconstructed data is operable to be combined with the combined reconstructed data to form composite reconstructed data; and A component for outputting at least one of the additional reconstruction data and the composite reconstruction data.

Images