Apparatus, method, and computer program for video coding and decoding

Abstract

The process involves receiving input data including rumor and chroma components, providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data, obtaining an encoded rumor component from the output of the first encoder, providing the encoded rumor component to a first decoder, obtaining a reconstructed rumor component from the output of the first decoder, and providing a second ground truth data to a second encoder, which includes at least a second portion of the input data and the reconstructed rumor component. A method comprising providing to (1100) the second encoder comprising a first subencoder (1102) for ground truth data, a second subencoder (1104) for reconstructed chroma components, and a probabilistic model (1106); providing the outputs of the first subencoder and the second subencoder to the probabilistic model; obtaining one or more first probabilities from the output of the probabilistic model; and obtaining encoded chroma components from the output of the second encoder based on the outputs of the first subencoder and one or more probabilities.

Description

Technical Field

[0001] The present invention relates to an apparatus, a method, and a computer program for video coding and decoding.

Background Art

[0002] In video coding, video and image samples are usually encoded using a color representation such as YUV or YCbCr, which consists of one luminance (luma) channel, often denoted as Y, and two chrominance (chroma) channels, often denoted as U, V or Cb, Cr. In these cases, the luminance channel, which represents mostly the illumination of the scene, is usually encoded at a certain resolution, while the chrominance channels, which usually represent the differences between certain color components, are often encoded at a second resolution lower than the resolution of the luminance signal. The purpose of this kind of differential representation is to lose the correlation of the color components and to make it possible to compress the data more efficiently.

[0003] Neural networks (NNs) have been used in the context of image and video compression by replacing one or more of the components of conventional codec or by utilizing end-to-end learning-based compression. For intra prediction of luma samples and chroma samples, separate prediction blocks are used for the luma component and the chroma component, and the chroma intra prediction block may perform inter-component prediction from the luma component. It is generally known that at least the luma and chroma components of adjacent samples have some correlation in their values.

[0004] However, NN-based video coding approaches, including solutions based on end-to-end learning-based compression, have been shown to be insufficient to account for this correlation. Therefore, the rate-distortion performance of the codec was not optimal.

Summary of the Invention

[0005] To at least mitigate the above problems, enhanced methods for improving rate distortion performance are presented herein.

[0006] The scope of protection for various embodiments of the present invention is defined by the independent claims. Any embodiments and features described herein that are not covered by the independent claims should be interpreted as useful examples for understanding the various embodiments of the present invention.

[0007] A method according to a first embodiment includes: receiving input data including a rumor component and a chroma component; providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; obtaining an encoded rumor component from the output of the first encoder; providing the encoded rumor component to a first decoder; obtaining a reconstructed rumor component from the output of the first decoder; and providing second ground truth data to a second encoder, wherein the second encoder comprises at least a first subencoder for the ground truth data, a second subencoder for the reconstructed rumor component, and a probabilistic model; providing the outputs of the first subencoder and the second subencoder to the probabilistic model; obtaining one or more first probabilities from the output of the probabilistic model; and obtaining an encoded chroma component from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0008] The apparatus according to the second embodiment includes means for receiving input data including a rumor component and a chroma component; means for providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; means for obtaining an encoded rumor component from the output of the first encoder; means for providing the encoded rumor component to a first decoder; means for obtaining a reconstructed rumor component from the output of the first decoder; and means for obtaining a second ground truth data including at least a second portion of the input data and the reconstructed rumor component. Means for providing ground truth data to a second encoder, the second encoder comprising at least a first subencoder for ground truth data, a second subencoder for reconstructed chroma components, and a probabilistic model; means for providing the outputs of the first subencoder and the second subencoder to the probabilistic model; means for obtaining one or more first probabilities from the output of the probabilistic model; and means for obtaining encoded chroma components from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0009] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

[0010] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder comprises means for converting input data into a plurality of latent tensor elements, the probabilistic model comprises means for estimating the probability of each latent tensor element, and the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element.

[0011] According to one embodiment, the first subencoder of the second encoder is a neural encoder, and the second subencoder of the second encoder is an auxiliary encoder that includes means for generating an auxiliary input to a probabilistic model.

[0012] According to one embodiment, the input to the auxiliary encoder is the reconstruction of the luma component.

[0013] According to one embodiment, the input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0014] According to one embodiment, the input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0015] According to one embodiment, the input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction from the reconstructed luma component.

[0016] A method according to a third aspect includes receiving input data containing encoded chroma components and reconstructed luma components into a decoder, the decoder comprising at least a first sub-decoder, a second sub-decoder, and a probabilistic model, the method comprising receiving, providing the reconstructed luma components to the second sub-decoder, obtaining one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components, obtaining an entropy-decoded chroma component based at least on the encoded chroma components and one or more probabilities, providing the entropy-decoded chroma components to the first sub-decoder, and obtaining a reconstructed chroma component from the output of the first sub-decoder.

[0017] An apparatus according to a fourth aspect is a means for receiving input data including encoded chroma components and reconstructed luma components into a decoder, the decoder comprising at least a first sub-decoder, a second sub-decoder, and a probabilistic model, means for providing the reconstructed luma components to the second sub-decoder, means for obtaining one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components, means for obtaining an entropy-decoded chroma component based at least on the encoded chroma components and one or more probabilities, means for providing the entropy-decoded chroma components to the first sub-decoder, and means for obtaining the reconstructed chroma components from the output of the first sub-decoder.

[0018] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, the decoder's probabilistic model includes means for estimating the probability of each decoded latent tensor element, the entropy decoder includes means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of each latent tensor element, and the neural decoder includes means for converting the plurality of decoded latent tensor elements into reconstructed chroma components.

[0019] According to one embodiment, the first sub-decoder of the decoder is a neural decoder, and the second sub-decoder of the decoder is an auxiliary decoder equipped with means for generating an auxiliary input to a probabilistic model.

[0020] According to one embodiment, the device includes means for concatenating the decoded latent tensor with an auxiliary input decoder along the dimension of the latent tensor channel.

[0021] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0022] A computer-readable storage medium storing the code is arranged to execute one or more of the methods and related embodiments described above.

[0023] For a better understanding of the present invention, reference will now be made, by way of example, to the accompanying drawings.

Brief Description of the Drawings

[0024] [Figure 1] A diagram showing an example of a codec having a neural network (NN) component. [Figure 2] A diagram showing another example of a video coding system having a neural network component. [Figure 3] A diagram showing an example of an end-to-end learning-based codec based on a neural network. [Figure 4] A diagram showing an example of an end-to-end learning-based video coding system based on a neural network. [Figure 5] A diagram showing an example of video coding for a computer. [Figure 6] A diagram showing an example of a pipeline of an end-to-end learning-based system for video coding for a computer. [Figure 7] A diagram showing an example of training of an end-to-end learning-based codec. [Figure 8] A diagram showing an example of a dense split attention (DSA) block. [Figure 9a] A diagram showing a flowchart of a coding method and a decoding method for improving rate-distortion performance according to an embodiment of the present invention. [Figure 9b] A diagram showing a flowchart of a coding method and a decoding method for improving rate-distortion performance according to an embodiment of the present invention. [Figure 10]This figure illustrates an exemplary implementation of a luma component codec according to one embodiment of the present invention. [Figure 11] This figure illustrates an exemplary implementation of a chroma component codec according to one embodiment of the present invention. [Modes for carrying out the invention]

[0025] The following descriptions and drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided to fully understand this disclosure. However, in certain examples, well-known or conventional details are omitted to avoid obscuring the description. References to one embodiment or one representation in this disclosure may refer to the same embodiment, but are not necessarily so, and such references mean at least one of the embodiments.

[0026] Any reference in this specification to “one embodiment” means that certain features, structures, or characteristics described in relation to an embodiment are included in at least one embodiment of this disclosure.

[0027] Before discussing this embodiment in more detail, a brief reference to related technologies is given.

[0028] In the context of machine learning, a neural network (NN) is a computation graph composed of several computational layers, or computational parts. Each layer consists of one or more units, each unit performing basic computations. Units are connected to one or more other units, and these connections may be associated with weights. Weights may be used to scale the signals passing through the associated connections. Weights are learnable parameters, i.e., values ​​that can be learned from training data. There may be other learnable parameters, such as the parameters of the batch normalization layer.

[0029] Two widely used architectures for neural networks are the feedforward architecture and the recurrent architecture. In a feedforward neural network, there are no feedback loops, and each layer receives input from one or more of the preceding layers and provides its output as input to one or more of the following layers. Alternatively, units within a certain layer receive input from one or more of the preceding layers and provide their output to one or more of the following layers.

[0030] The initial layers (the layers closest to the input data) extract semantically low-level features such as edges and textures in the image, while the intermediate and final layers extract higher-level features. Following the feature extraction layers, there may be one or more layers that perform certain tasks such as classification, semantic segmentation, object detection, denoising, style transfer, and super-resolution processing. In a recurrent neural network, there is a feedback loop, so the network can have states, i.e., it can store information or states.

[0031] Neural networks are being used in a growing number of applications for various types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, and device usage data analysis.

[0032] One of the key characteristics of neural networks (and other machine learning tools) is their ability to learn characteristics from input data, either in a supervised or unsupervised manner. Such learning is the result of training algorithms or meta-level neural networks that provide training signals.

[0033] Generally, a training algorithm consists of modifying several properties of a neural network so that its output becomes as close as possible to a desired output. For example, in the case of classifying objects in an image, the output of the neural network may be used to derive a class or category index that indicates the class or category to which the objects in the input image belong. Training usually occurs by minimizing or reducing the error of the output, also called loss. Examples of loss include mean squared error and cross-entropy. In recent deep learning techniques, training is an iterative process, and in each iteration, the algorithm modifies the weights of the neural network to gradually improve the network's output, i.e., gradually reduce the loss.

[0034] In this explanation, the terms "model" and "neural network" are used interchangeably, and the weights of a neural network may be referred to as learnable parameters or simply parameters.

[0035] Training a neural network is an optimization process. The goal of the optimization, or training, process is to allow the model to learn the characteristics of the data distribution from a limited training dataset. In other words, the goal is to use the limited training dataset to learn generalizations to data that has not been seen before, i.e., data that was not used to train the model. This is usually referred to as generalization. In practice, the data may be split into at least two sets: a training set and a validation set. The training set is used to train the network, i.e., to modify its learnable parameters to minimize the loss. The validation set is used to check the network's performance on data that was not used to minimize the loss, as an indication of the model's final performance. In particular, the errors on the training and validation sets are monitored during the training process to understand the following: - Whether the network has learned at all or not - if so, the error in the training set will decrease; otherwise, the model is in an underfitting state. - Whether the network is learning to generalize - In this case, the error on the validation set must also be reduced and not too large compared to the error on the training set. If the error on the training set is small, but the error on the validation set is much larger than the error on the training set, the error on the validation set will not decrease, or may even increase, and the model is overfitted. This means that the model simply memorizes the characteristics of the training set and performs well only on that set, but poorly on sets that are not used to tune its parameters.

[0036] While the above background information on neural networks may be valid at the time of writing this document, the entire field of neural networks and machine learning is rapidly evolving. Therefore, it should be understood that at least some of the embodiments described herein are not limited to the definitions of neural networks, machine learning models, or training algorithms given in the above background information.

[0037] In recent years, neural networks have been used to compress and decompress data such as images, i.e., in image codecs. The most widely used architecture for implementing one component of an image codec is the autoencoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. The neural encoder takes an image as input and generates a code that requires fewer bits than the input image. This code may be obtained by applying binarization or quantization to the encoder's output. The neural decoder receives this code and reconstructs the image that was input to the neural encoder.

[0038] Such neural encoders and decoders may be trained to minimize a combination of bitrate and distortion, the distortion of which may be based on one or more of the following metrics: mean squared error (MSE), peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM). These distortion metrics are correlated to human visual perception quality such that minimizing or maximizing one or more of these distortion metrics results in an improvement in the visual quality of the decoded image as perceived by humans.

[0039] A video codec comprises an encoder that converts the input video into a compressed representation suitable for storage / transmission, and a decoder that decompresses the compressed video representation back into a viewable format. The encoder may discard some information in the original video sequence in order to represent the video in a more compact format (i.e., at a lower bitrate).

[0040] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Expert Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Video Expert Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization bodies and is referred to as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, and is also known as MPEG-4 Part 10 Advanced Video Coding (AVC). Extensions to H.264 / AVC include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0041] The High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Team for Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by both parent standards bodies and is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, and is also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multiview, Fidelity Range, 3D, and Screen Content Coding Extensions, which are sometimes abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0042] VVC (H.266), also known as Multipurpose Video Coding, defined in ITU-T Recommendation H.266 and equivalently specified in ISO / IEC 23090-3 (also known as MPEG-I Part 3), is a video compression standard developed as a successor to HEVC. The reference software for VVC is the VVC Test Model (VTM).

[0043] The AV1 bitstream format and decoding process specifications were developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0044] The basic unit for both the input to a video encoder and the output to a video decoder is, in most cases, a picture. The picture given as input to the encoder is also called the source picture, and the picture decoded by the decoder is called the decoded picture or reconstructed picture.

[0045] The source picture and decode picture are the following set of sample sequences, namely: - Luma (Y) only (monochrome), - Luma and two chromosomes (YCbCr or YcgCo), - Green, blue, red (GBR, also known as RGB), - An array representing other unspecified monochrome or tristimulus color sampling (e.g., YZX, also known as XYZ) Each consists of one or more sample sequences, such as one of the following.

[0046] A component may be defined as an array or a single sample from one of the three sample arrays (luma and two chroma) that make up the picture, or as an array or a single sample of an array that makes up the picture in monochrome format.

[0047] Hybrid video codecs, such as ITU-T H.263 and H.264, may encode video information in two phases. First, pixel values ​​within a certain picture area (or "block") are predicted, for example, by motion compensation means (finding and indicating an area in one of the previously coded video frames that closely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified manner). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size or transfer bitrate).

[0048] Interpretation, sometimes called temporal prediction, motion compensation, or motion-compensated prediction, utilizes temporal redundancy. In interpretation, the source of the prediction is a previously decoded picture.

[0049] Intra prediction utilizes the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial domain or the transformation domain, i.e., either sample values ​​or transformation coefficients can be predicted. Intra prediction may be used in intra coding where no interpretation is applied.

[0050] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantization transformation coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors may be predicted from spatially adjacent motion vectors, and only the difference for the motion vector predictor may be coded. The prediction and intra-prediction of coding parameters are sometimes collectively referred to as in-picture prediction.

[0051] The decoder reconstructs the output video by applying prediction means similar to those of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation), and prediction error decoding (the inverse operation of prediction error coding to reconstruct the quantized prediction error signal in the spatial pixel domain). After applying the prediction means and prediction error decoding means, the decoder sums the prediction signal and the prediction error signal (pixel values) to form an output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing the output video for display and / or before storing the output video as a predictive reference for future frames in the video sequence.

[0052] In a video codec, motion information may be indicated using motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement between the image block in the picture being coded (on the encoder side) or decoded (on the decoder side) and the predicted source block in one of the previously coded or decoded pictures. To efficiently represent motion vectors, they may be coded differentially against block-specific predicted motion vectors. In a video codec, the predicted motion vector may be created in a predetermined manner, for example, by calculating the median of the coded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in a time-reference picture and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, the reference index of a previously coded / decoded picture may be predicted. The reference index may be predicted from adjacent blocks and / or co-located blocks in a time-reference picture. Furthermore, high-efficiency video codecs can employ an additional motion information coding / decoding mechanism, often referred to as merging mode, in which all motion field information, including motion vectors and corresponding reference picture indices for each available reference picture list, is predicted and used without any modification or correction. Similarly, predicting motion field information may be performed using motion field information from adjacent and / or co-located blocks within a time reference picture, with the motion field information used signaled among a list of motion field candidate lists filled with motion field information from available adjacent / co-located blocks.

[0053] In video codecs, the predicted residuals after motion compensation may first be transformed using a transformation kernel (such as DCT) and then coded. This is because some correlation often still exists between the residuals, and the transformation can often help reduce this correlation, resulting in more efficient coding.

[0054] A video encoder may utilize a Lagrangian cost function to find the optimal coding mode, for example, the desired coding mode for a block, block partitioning, and associated motion vectors. This type of cost function uses a weighting coefficient λ to link the (exact or estimated) image distortion resulting from the irreversible coding method with the (exact or estimated) amount of information required to represent the pixel values ​​within the image area. C = D + λR Here, C is the Lagrange cost to be minimized, D is the image distortion (e.g., mean squared error) considering the mode vector and motion vector, and R is the number of bits required to represent the necessary data for reconstructing the image block in the decoder (including the amount of data to represent the candidate motion vector). Rate R may be the actual bitrate or number of bits resulting from the encoding. Alternatively, rate R may be an estimated bitrate or number of bits. One possible way to estimate rate R is to omit the final entropy encoding step and use, for example, a simpler entropy encoding or an entropy encoder in which some of the context states have not been updated according to the previous encoding mode selection.

[0055] Conventional distortion measurement metrics may include, but are not limited to, the peak signal-to-noise ratio (PSNR), mean squared error (MSE), sum of absolute differences (SAD), sum of absolute transform differences (SATD), and structural similarity (SSIM). These are typically measured between a reconstructed video / image signal (which is identical to, or will be identical to, the decoded video / image signal) and the "original" video / image signal provided as input for encoding.

[0056] Partitioning may also be defined as dividing a set into subsets such that each element of the set is in exactly one subset.

[0057] A bitstream may be defined as a sequence of bits, and in some coding formats or standards, may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, forming a representation of coded pictures and associated data that make up one or more coded video sequences.

[0058] The bitstream format may include a sequence of syntactic structures.

[0059] Syntactic elements may be defined as elements of data represented within a bitstream. Syntactic structures may be defined as zero or more syntactic elements coexisting within a bitstream in a specified order.

[0060] A NAL unit may be defined as a syntactic structure containing an indication of the type of subsequent data and a byte that contains that data in the form of an RBSP with start code emulation prevention bytes inserted as needed. A raw byte sequence payload (RBSP) may be defined as a syntactic structure containing integer bytes encapsulated in the NAL unit. An RBSP is either empty or has the form of a string consisting of data bits containing the syntactic elements, followed by an RBSP stop bit and zero or more subsequent bits equal to zero.

[0061] Some coding formats specify a set of parameters that may carry parameter values ​​necessary for decoding or reconstructing a decoded picture. Parameters may be defined as syntactic elements of a parameter set. A parameter set may be defined as a syntactic structure that contains parameters and can be referenced from or activated by another syntactic structure, for example, using an identifier.

[0062] Coding standards or specifications may specify several types of parameter sets. It should be understood that embodiments may apply not only to the types of parameter sets described, but may also apply similarly to any parameter set type.

[0063] A parameter set may be activated, for example, when referenced through its identifier. An adaptive parameter set (APS) may be defined as a syntactic structure that applies to zero or more slices. There can be different types of adaptive parameter sets. An adaptive parameter set may, for example, contain filtering parameters for a particular type of filter. In VVC, three types of APS are specified that carry parameters for an adaptive loop filter (ALF), a lumens mapping with chromascaling (LMCS), and a scaling list. A scaling list may be defined as a list that associates each frequency index with a scaling factor for a scaling process, which multiplies the transformation factor level by the scaling factor to obtain the transformation factor. In VVC, an APS is referenced through its type (e.g., ALF, LMCS, or scaling list) and identifier. In other words, different types of APS have their own range of identifier values.

[0064] The Adaptive Parameter Set (APS) may include parameters for different types of decoding processes, such as adaptive loop filtering or lumamapping with chroma scaling.

[0065] Video coding specifications may allow the use of Supplemental Extensible Information (SEI) messages, etc. Some video coding specifications include SEI Network Abstraction Layer (NAL) units, and some video coding specifications include both prefix SEI NAL units and suffix SEI NAL units, the former type which can initiate picture units, etc., and the latter type which can terminate picture units, etc. An SEI NAL unit contains one or more SEI messages which are not required for decoding the output picture but may support related processes such as picture output timing, post-processing of decoded pictures, rendering, error detection, error hiding, and resource reservation. The H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards specify several SEI messages, and user data SEI messages allow organizations and enterprises to specify SEI messages for their specific use. The standards may include syntax and semantics for specified SEI messages, but may not define the process for handling the messages at the receiving end. As a result, encoders may be required to adhere to standards specifying SEI messages when creating them, while decoders may not be required to process SEI messages for the sake of output order compatibility. One reason for including the syntax and semantics of SEI messages in the standard is to ensure that different system specifications interpret supplementary information in the same way and thus enable interoperability. System specifications may require the use of specific SEI messages at both the end of encoding and the end of decoding, and may specify a process for handling specific SEI messages at the receiving end. SEI messages are generally not extended in future modifications or versions of the standard.

[0066] The phrases "along a bitstream" (e.g., indicating along a bitstream) or "along a bitstream coding unit" (e.g., indicating along a coded tile) may be used to refer to the transmission, signaling, or storage of “out-of-band” data in a manner that is associated with, but not contained within, a bitstream or coding unit, respectively. The phrases "decoding along a bitstream" or "decoding along a bitstream coding unit" may refer to decoding referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with a bitstream or coding unit, respectively. For example, the phrase "along a bitstream" may be used when a bitstream is contained within a container file, such as a file conforming to an ISO-based media file format, and certain file metadata is stored in the file in a manner that is associated with the bitstream, such as boxes in sample entries for a track containing a bitstream, sample groups for a track containing a bitstream, or timed metadata tracks associated with a track containing a bitstream.

[0067] Image and video codecs may use a set of filters to enhance the visual quality of predicted visual content, which may be applied in-loop, out-loop, or both. In the case of in-loop filters, a filter applied to one block in the frame currently being encoded will affect the encoding of another block in the same frame, and / or another block in another frame predicted from the current frame. In-loop filters can affect bitrate and / or visual quality. In fact, enhanced blocks will have smaller residuals (the difference between the original block and the predicted and filtered block), and therefore require fewer bits for encoding. Out-loop filters are applied to the frame after it has been reconstructed, so the filtered visual content may only affect the visual quality of the frame output by the decoder, and not as a source for prediction.

[0068] Recently, neural networks (NNs) have been used in the context of image and video compression, primarily by following two approaches.

[0069] One approach involves using neural networks (NNs) to replace one or more components of traditional codecs such as VVC / H.266. Here, the term "traditional" refers to codecs where components and their parameters may not be learned from the data. Examples of such components include: - For example, an additional in-loop filter by adding a neural network (NN) as an additional in-loop filter to a conventional loop filter. - For example, a single in-loop filter by having a neural network that replaces all conventional in-loop filters. - Intraframe prediction. - Interframe prediction. - Conversion and / or inverse conversion. - A probabilistic model for arithmetic codecs. - others That is the case.

[0070] Figure 1 illustrates an example of the functionality of a neural network (NN) as a component of a conventional codec pipeline in one embodiment. In particular, Figure 1 illustrates an encoder that also includes a decoding loop. Figure 1 is shown to include the components described below.

[0071] A lumane intra-prediction block or circuit 101. This block or circuit performs intra-prediction in the lumane domain, for example, by using already reconstructed data from the same frame. The operation of the lumane intra-prediction block or circuit 101 may be performed by a deep neural network, such as a convolutional autoencoder.

[0072] Chroma intra-prediction block or circuit 102. This block or circuit performs intra-prediction in the chroma domain, for example, by using already reconstructed data from the same frame. The chroma intra-prediction block or circuit 102 may also perform cross-component prediction, for example, predicting chroma from luma. The operation of the chroma intra-prediction block or circuit 102 may be performed by a deep neural network, such as a convolutional autoencoder.

[0073] An intra-prediction block or circuit 103 and an inter-prediction block or circuit 104. These blocks or circuits perform intra-prediction and inter-prediction, respectively. The intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 may perform predictions for all components, e.g., lumers and chromas. The operation of the intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 may be performed by two or more deep neural networks, such as convolutional autoencoders.

[0074] A probability estimation block or circuit 105 for entropy coding. This block or circuit performs probability predictions for the next symbol to encode or decode, and the predicted values ​​are subsequently provided to an entropy coding module 112, such as an arithmetic coding module, for encoding or decoding the next symbol. The operation of the probability estimation block or circuit 105 may be performed by a neural network.

[0075] A transform and quantization (T / Q) block or circuit 106. These are actually two blocks or circuits. The transform and quantization block or circuit 106 transforms the input data into different domains; for example, an FFT transform would transform the data into the frequency domain. The transform and quantization block or circuit 106 may quantize its input value into a smaller set of possible values. The decoding loop may also include an inverse quantization block or circuit and an inverse transform block or circuit 113. One or both of the transform block or circuit and the quantization block or circuit may be replaced by one or more neural networks. One or both of the inverse transform block or circuit and the inverse quantization block or circuit 113 may be replaced by one or more neural networks.

[0076] An in-loop filter block or circuit 107. The operation of the in-loop filter block or circuit 107 is carried out within the decoding loop and filters the output of the inverse transform block or circuit, or in any case, the reconstructed data, to enhance the reconstructed data with respect to one or more predetermined quality metrics. This filter may affect both the quality of the decoded data and the bitrate of the bitstream output by the encoder. The operation of the in-loop filter block or circuit 107 may be carried out by a neural network, such as a convolutional autoencoder. For example, the operation of the in-loop filter may be carried out by multiple steps or filters, and one or more steps may be carried out by a neural network.

[0077] A post-processing filter block or circuit 108. Since the post-processing filter block or circuit 108 may not affect the encoding process, it may be implemented only on the decoder side. The post-processing filter block or circuit 108 filters the reconstructed data output by the in-loop filter block or circuit 107 to enhance the quality of the reconstructed data. The post-processing filter block or circuit 108 may be replaced by a neural network, such as a convolutional autoencoder.

[0078] Resolution-adaptive block or circuit 109: This block or circuit may downsample the input video frame before encoding. Then, in the decoding loop, the reconstructed data may be upsampled to the original resolution by the upsampling block or circuit 110. The operation of the resolution-adaptive block or circuit 109 may be carried out by a neural network such as a convolutional autoencoder.

[0079] Encoder control block or circuit 111. This block or circuit optimizes the encoder parameters, such as what transformations to use, what quantization parameters (QP) to use, and which intra-prediction modes (out of N intra-prediction modes) to use. The operation of the encoder control block or circuit 111 may be carried out by a neural network, such as a classifier convolutional network or a regressive convolutional network.

[0080] The ME / MC block or circuit 114 performs motion estimation and / or motion compensation, which are two key operations performed when performing interframe prediction. ME / MC represents motion estimation / motion compensation.

[0081] Another approach, commonly referred to as "end-to-end learning compression," uses neural networks (NNs) as the primary component of the image / video codec. This second approach has two main options.

[0082] Option 1: Reuse the video coding pipeline but replace most or all of its components with neural networks. Referring to Figure 2, an example of a modified video coding pipeline based on a neural network, according to one embodiment, is illustrated. The neural network example includes, but is not limited to, a compressed representation of the neural network. Figure 2 shows that it includes the following components: - Neural transformation block or circuit 202: This block or circuit transforms the output of the addition / subtraction operation 203 into a new representation of the data, which may have lower entropy and therefore be more compressible. - Quantization block or circuit 204: This block or circuit quantizes the input data 201 into a smaller set of possible values. - Inverse transform and inverse quantization block or circuit 206. These blocks or circuits perform the inverse or nearly inverse operations of the transform and quantization, respectively. - Encoder parameter control block or circuit 208. This block or circuit may control and optimize some or all parameters of the encoding process, such as one or more parameters of the encoding block or circuit. - Entropy coding block or circuit 210. This block or circuit may perform reversible coding based on entropy, for example. One common entropy coding technique is arithmetic coding. - Neural intracodec block or circuit 212. This block or circuit may also be an image compression and decompression block or circuit that may be used to code and decode intraframes. Encoder 214 may be an encoder block or circuit, such as the neural encoder section of an autoencoder neural network. Decoder 216 may be a decoder block or circuit, such as the neural decoder section of an autoencoder neural network. Intracoding block or circuit 218 may be a block or circuit that performs several intermediate steps between the encoder and decoder, such as quantization, entropy coding, entropy decoding, and / or inverse quantization. - Deep loop filter block or circuit 220. This block or circuit performs filtering of the reconstructed data to enhance the reconstructed data. - Decode picture buffer block or circuit 222. This block or circuit is a memory buffer that holds the decoded frame, e.g., the reconstructed frame 224, and the enhanced reference frame 226 used for interpretation. - Interframe prediction block or circuit 228. This block or circuit performs interframe prediction, for example, predicting from a frame that is temporally close, for example, frame 232. ME / MC 230 performs motion estimation and / or motion compensation, which are two key operations performed when interframe prediction is performed. ME / MC represents motion estimation / motion compensation.

[0083] Option 2: Redesign the entire pipeline as follows: - The encoder NN is configured to perform nonlinear transformations. - Quantization and lossless encoding of the output of encoder NNs. - Reversible decoding and inverse quantization, - The decoder NN is configured to perform a nonlinear inverse transform.

[0084] An example of Option 2 is illustrated in detail in Figure 3, which shows the encoder NN and decoder NN, which are part of the example neural autoencoder architecture. In Figure 3, the analysis network 301 is the encoder NN and the synthesis network 302 is the decoder NN, which together are sometimes referred to as the spatial correlation tool 303, or neural autoencoder.

[0085] As shown in Figure 3, the input data 304 is analyzed by an encoder NN (analysis network 301), which outputs a new representation of the input data. This new representation may be more compressible. This new representation may then be quantized to discrete numerical values ​​by a quantizer 305. The quantized data is then reversibly coded, for example, by an arithmetic encoder 306, thus obtaining a bitstream 307. The example shown in Figure 3 includes an arithmetic decoder 308 and an arithmetic encoder 306. The arithmetic encoder 306, or the arithmetic decoder 308, or a combination of the arithmetic encoder 306 and the arithmetic decoder 308 may be referred to as an arithmetic codec in some embodiments. On the decoding side, the bitstream is first reversibly decoded, for example, by using an arithmetic codec decoder 308. The reversibly decoded data is dequantized and then input to a decoder NN, a synthesis network 302. The output is the reconstructed or decoded data 309.

[0086] In the case of lossy compression, the lossy step may include an encoder neural network and / or quantization.

[0087] To train this system, a training objective function (also called the "training loss") may be used, which may contain one or more terms, a loss term, or simply a loss. In one example, the training loss includes a reconstruction loss term and a rate loss term. The reconstruction loss prompts the system to decode data similar to the input data according to some similarity metric. An example of a reconstruction loss is: - Mean squared error (MSE), - Multiscale structural similarity (MS-SSIM), - The loss obtained from using a pre-trained neural network. For example, error(f1, f2), where f1 and f2 are features extracted by the pre-trained neural network on the input data and the decoded data, respectively, and error() is an error function or distance function such as the L1 norm or L2 norm. - Loss obtained from using an end-to-end learning codec and a neural network trained concurrently. For example, adversarial loss may be used, which is the loss provided by a discriminator neural network trained adversarially with respect to the codec, according to the proposed configuration in terms of generative adversarial networks (GANs) and their variants. That is the case.

[0088] Rate loss prompts the system to compress the output of the encoding stage, such as the output of an arithmetic encoder. In this context, "compression" means reducing the number of bits output by the encoding stage.

[0089] When entropy-based reversible encoders, such as arithmetic encoders, are used, rate loss typically encourages the encoder NN output to have low entropy. An example of rate loss is shown below, i.e., - Differentiable estimate of entropy, - Sparsification loss, i.e., the loss that causes the output of the encoder NN or the output of the quantization to have many zeros. An example is the L0 norm, L1 norm, and L1 norm divided by the L2 norm. - Cross-entropy loss applied to the output of the probabilistic model. The probabilistic model may be a neural network (NN) used to estimate the probability of the next symbol encoded by an arithmetic encoder. That is the case.

[0090] As a weighted sum, one or more of the reconstruction losses or one or more of the rate losses may be used. Different loss terms may be weighted using different weights, which determine how the final system performs in the rate-distortion loss term. For example, if reconstruction losses are given a greater weight than rate losses, the system may learn to reconstruct with less compression but with greater accuracy (as measured by a metric correlated with the reconstruction losses). These weights may be considered hyperparameters of the training session, which may be set manually by the person designing the training session, or they may be set automatically, for example, by grid search or using an additional neural network.

[0091] As shown in Figure 4, a neural network-based end-to-end learning video coding system may include an encoder 401, a quantizer 402, a probabilistic model 403, an entropy codec 420 (e.g., an arithmetic encoder 405 / arithmetic decoder 406), an inverse quantizer 407, and a decoder 408. The encoder 401 and decoder 408 may be two neural networks or may consist mainly of neural network components. The probabilistic model 403 may also consist mainly of neural network components. The quantizer 402, inverse quantizer 407, and entropy codec 420 do not have to be based on neural network components, but may potentially include them.

[0092] On the encoder side, encoder component 401 receives video x409 as input and converts the video from its original signal space to a latent representation which may contain a more compressible representation of the input. For an input image, the latent representation may be a three-dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represents the "channels" that contain information at specific locations. If the input image is a 128×128×3RGB image (128 horizontal pixels, 128 vertical pixels, and three channels for red, green, and blue color components), and the encoder downsamples the input tensor by a factor of two, expanding the channel dimension to 32 channels, then the latent representation is a 64×64×32-dimensional (or "shape") tensor (i.e., 64 horizontal elements, 64 vertical elements, and 32 channels). It should be noted that the order of different dimensions may vary depending on the rules used, and in some cases, for input images, the channel dimension may be the first dimension; therefore, for the example above, the shape of the input tensor may be represented as 3 × 128 × 128 instead of 128 × 128 × 3. For input video (instead of input images), another dimension of the input tensor may be used to represent temporal information.

[0093] The quantizer component 402 quantizes the latent representation into discrete values ​​given a predefined set of quantization levels. The probabilistic model 403 and the arithmetic codec component 420 work together to perform lossless compression on the quantized latent representation and generate a bitstream to be sent to the decoder. Given a symbol to be encoded in the bitstream, the probabilistic model 403 estimates the probability distribution of all possible values ​​for that symbol based on the context constructed from available information in the current encoding / decoding state, such as already encoded / decoded data. The arithmetic encoder 405 then uses the estimated probability distribution to encode the input symbol into the bitstream.

[0094] On the decoder side, the opposite operation is performed. The arithmetic decoder 406 and the probabilistic model 403 first decode symbols from the bitstream to reconstruct the quantized latent representation. Then, the inverse quantizer 407 reconstructs the latent representation into continuous values ​​and passes it to the decoder 408 to reconstruct the input video / image. Note that in this system, the probabilistic model 403 is shared between the encoding system and the decoding system. In practice, this means that a copy of the probabilistic model 403 is used on the encoder side and another exact copy is used on the decoder side.

[0095] In this system, the encoder 401, the probabilistic model 403, and the decoder 408 may be based on a deep neural network. The system may be trained in an end-to-end manner by minimizing the following rate-distortion loss function. L = D + λR

[0096] Here, D is the distortion loss term, R is the rate loss term, and λ is a weight that controls the balance between the two losses. The distortion loss term may be the mean squared error (MSE), structural similarity (SSIM), or other metric that evaluates the quality of the reconstructed video. Multiple distortion losses may be used and integrated into D, such as a weighted sum of MSE and SSIM. The rate loss term is typically the estimated entropy of the quantized latent representation, which indicates the number of bits required to represent the encoded symbol, e.g., bits per pixel (bpp).

[0097] For lossless video / image compression, the system may consist only of the probabilistic model 403 and the arithmetic encoder / decoder 405, 406. Since distortion loss is always zero (i.e., no loss of information), the system loss function consists only of rate loss.

[0098] Since humans are considered to be the end users, i.e., the ones consuming / viewing the decoded images, reducing distortion in image and video compression is often intended to improve human perceptual quality. In recent years, with the advent of machine learning, particularly deep learning, there has been an increase in the number of computers (i.e., autonomous agents) that can analyze data independently of humans and even make decisions based on the analysis results without human intervention. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, and pedestrian tracking. Exemplary use cases and applications include autonomous vehicles, video surveillance cameras and public safety, smart sensor networks, smart TVs and smart advertising, person re-identification, smart traffic monitoring, and drones. When decoded data is consumed by computers, different quality metrics will be used instead of human perceptual quality. Also, dedicated algorithms for compressing and decompressing data for computer consumption may differ from dedicated algorithms for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for computer consumption is referred to here as Video Coding for Machines (VCM).

[0099] VCM (Video Code Streaming) relates to the encoding of video streams to enable consumption by computers. A computer refers to any device other than a human. Examples of computers may include mobile phones, autonomous vehicles, robots, and intelligent devices that may have some degree of autonomy or that may execute intelligent algorithms to process the decoded stream beyond simply reconstructing the original input stream.

[0100] The computer may perform one or more tasks on the decoded stream. Examples of tasks are as follows: - Classification: Classify an image or video into one or more defined categories. The output of the classification task may be a set of detected categories, also known as classes or labels. The output may also include the probability and confidence level for each defined category. - Object detection: Detects one or more objects in a given image or video. The output of the object detection task may be the bounding boxes of the detected objects and their associated classes. The output may also include the probability and confidence level of each detected object. - Instance segmentation: Identifying one or more objects in an image or video at the pixel level. The output of the instance segmentation task may be a binary mask image of the detected objects, or other representations of the binary mask image, such as closed contours. The output may also include the probability and confidence level of each object for each pixel. - Semantic segmentation: Assigns pixels in an image or video to one or more defined semantic categories. The output of a semantic segmentation task may be a binary mask image of the assigned categories, or other representations of the binary mask image, such as closed contours. The output may also include the probability and confidence level for each semantic category for each pixel. - Object Tracking: Track one or more objects within a video sequence. The output of the object tracking task may also include the frame index, object ID, object bounding box, probability, and confidence level for each tracked object. - Caption generation: Generates one or more short text descriptions for an input image or video. The output of the caption generation task may be one or more short text sequences. - Human pose estimation: Estimates the positions of key points, such as wrists, elbows, and knees, from the bodies of one or more people in a video image. The output of human pose estimation includes a set of positions for each key point of the human body detected in the input image or video. - Human Action Recognition: Recognizes the actions of one or more people in an input image or video, such as walking, talking, or shaking hands. The output of human action recognition may be a predefined set of actions, probabilities, and confidence levels for each identified action. - Anomaly detection: Detects anomalous objects or events from the input image or video. The anomaly detection output may include the location of the detected anomalous object, or the frame segment in the input video in which the anomalous event was detected. Includes.

[0101] The receiving device may have multiple "computers" or task neural networks (Task-NNs). These multiple computers may be used in a certain combination determined, for example, by an orchestrator subsystem. The multiple computers may be used sequentially, for example, based on the output of previously used computers, and / or in parallel. For example, compressed and then decompressed video may be analyzed by one computer (NN) for detecting pedestrians, another computer (another NN) for detecting cars, and yet another computer (another NN) for estimating the depth of all pixels in the frame.

[0102] In this description, “task computer,” “computer,” and “task neural network” are interchangeable references, and such references mean any process or algorithm that analyzes or processes (learns from or does not learn from) data for a particular task. The remainder of this description may specify in further detail other assumptions about computers as they exist in this disclosure. The terms “receiver” or “decoder” are used to refer to one or more computers, encompassing one or more computers, and to a physical or abstract entity or device that performs on a video representation encoded and ultimately decoded by another physical or abstract entity or device, the “encoder-side device.”

[0103] The encoded video data may, for example, be stored as a file on a memory device. The stored file may later be provided to another device. Alternatively, the encoded video data may be streamed from one device to another.

[0104] Figure 5 is a schematic diagram of a video coding pipeline to a computer. The VCM encoder 502 encodes the input video into a bitstream 504. To evaluate the size of the bitstream, the bitrate 506 may be calculated from the bitstream 504 508. The VCM decoder 510 decodes the bitstream output by the VCM encoder 502. In Figure 5, the output of the VCM decoder 510 is referred to as “decoded data to the computer” 512. This data may be thought of as the decoded or reconstructed video. However, in some embodiments of this pipeline, this data may not have the same or similar characteristics as the original video input to the VCM encoder 502. For example, this data may not be easily understandable to humans when rendering the data on a screen. The output of the VCM decoder is then input to one or more task neural networks 514. In the figure, there are three exemplary task-NNs and one unspecified (task-NN X) to illustrate that there may be any number of task-NNs 514. The objective of VCM is to obtain a low-bitrate representation of the input video while ensuring that the task-NN still performs well in terms of the evaluation metric 516 associated with each task.

[0105] One possible approach to achieving video coding for computers is an end-to-end learning approach. In this approach, the VCM encoder and VCM decoder are primarily composed of neural networks. Figure 6 illustrates an example pipeline for the end-to-end learning approach. The video is input to the neural network encoder 601. The output of the neural network encoder 601 is input to a reversible encoder 602, such as an arithmetic encoder, which outputs a bitstream 604. The output of the neural network encoder 601 may also be input to a probabilistic model 603, which provides the reversible encoder 602 with an estimate of the probability of the next symbol to be encoded by the reversible encoder 602. The probabilistic model 603 may be learned by machine learning techniques, for example, it may be a neural network. On the decoder side, the bitstream 604 is input to a reversible decoder 605, such as an arithmetic decoder, and its output is input to the neural network decoder 606. The output of the reversible decoder 605 is input to the probabilistic model 603, which provides the reversible decoder 605 with an estimate of the probability of the next symbol to be decoded by the reversible decoder 605. The output of the neural network decoder 606 is the decoded data for computer 607, which may be input to one or more tasks - NN608.

[0106] Figure 7 illustrates an example of how an end-to-end learning system may be trained for the purpose of video coding for a computer. For simplicity, only one task—NN707—is described. A rate loss 705 may be calculated from the output of a probabilistic model 703. The rate loss 705 may provide an approximation of the bitrate required to encode the input video data. A task loss 710 may be calculated 709 from the output 708 of task-NN707.

[0107] The rate loss 705 and task loss 710 may then be used to train a neural network 711 for use within a system such as a neural network encoder 701, a probabilistic model 703, or a neural network decoder 706. Training may be performed by first calculating the gradient of each loss with respect to the trainable neural network parameters that contribute to or influence the calculation of that loss. The gradients are then used by an optimization method such as Adam to update the trainable parameters of the neural network.

[0108] Computer tasks may be performed on the decoder side (instead of the encoder side) for several reasons, such as the encoder-side device lacking the capability (computation, power, memory) to run the neural network that performs these tasks, or because some aspect or performance of the task neural network has been modified or improved by the time the decoder-side device requires the task results (e.g., different or additional semantic classes, a better neural network architecture). Furthermore, there may be a need for customization to perform these machine learning tasks, as different clients will each run different neural networks.

[0109] A dense split attention (DSA) block is an attention block that estimates one or more attention maps and applies one or more attention maps to one or more data tensors. - The attention map may be a vector, matrix, or tensor. For example, the attention map may have values ​​within the range [0,1]. - One or more data tensors may include one or more input tensors to an attention block, and / or one or more feature maps extracted within the attention block, and / or one or more feature maps extracted outside the attention block. - The application of one or more attention maps to one or more data tensors may include, for example, multiplying one or more attention maps by one or more data tensors by using element-wise multiplication. Other operations may also be considered.

[0110] Figure 8 illustrates an example of a DSA block, where one type of NN layer is a ResBlock containing an NN layer.

[0111] A DSA block may include extracting features from its input based on at least one or more initial NN layers, then splitting the extracted features across the channel axis to obtain two split features, summing the split features, performing a global average operation on the summed features, processing the output of the global average operation based on at least one or more NN layers, inputting the result of this processing into a softmax operation, splitting the result of the softmax operation across the channel axis to obtain two attention tensors, multiplying the two attention tensors by the two previously determined split features to obtain two attention split features, summing the two attention split features, concatenating the summed attention split features with features determined based on at least one or more initial NN layers, processing the result of the concatenation by at least one or more NN layers, and summing the output of this processing with the input of the DSA block to obtain the output of the DSA block. Here, the global averaging operation may be a global pooling (average pooling) operation that calculates the average value for a patch of feature maps. This allows for the aggregation of spatial information from feature maps into a single channel, and enables the utilization of feature relationships between channels.

[0112] In video coding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (lumen) channel, also denoted as Y, and two chrominance (chromen) channels, also denoted as U, V, or Cb, Cr. In these cases, the luminance channel, which largely represents the illumination of the scene, is usually coded at a certain resolution, while the chrominance channel, which represents the difference between certain color components, is often coded at a second resolution lower than the resolution of the luminance signal. The purpose of this type of difference representation is to eliminate the correlation between color components and to allow for more efficient data compression.

[0113] As described above, neural networks (NNs) are used in terms of image and video compression by replacing one or more components of conventional codecs, as shown in Figure 1, or by utilizing end-to-end learning-based compression, as shown in Figures 2 and 3. For intra-prediction of lumar and chroma samples, Figure 1 shows separate prediction blocks for the lumar and chroma components, and the chroma intra-prediction block may perform cross-component prediction from the lumar. It is known that the lumar and chroma components of at least adjacent samples usually have some correlation in their values. However, NN-based video coding approaches, including solutions based on end-to-end learning-based compression, have been shown to be insufficient to account for this correlation. Therefore, the rate-distortion performance of the codecs was not optimal.

[0114] Here, we will introduce methods to improve rate distortion performance.

[0115] Figure 9a shows an encoding method according to an embodiment, the method comprising: receiving input data including a lumen component and a chroma component (900); providing first ground truth data to a first encoder (902), wherein the first ground truth data includes at least a first portion of the input data; obtaining an encoded lumen component from the output of the first encoder (904); providing the encoded lumen component to a first decoder (906); obtaining a reconstructed lumen component from the output of the first decoder (908); and at least a second portion of the input data and the reconstructed lumen component. Providing (910) second ground truth data to a second encoder, wherein the second encoder includes at least a first subencoder for the ground truth data, a second subencoder for the reconstructed chroma component, and a probabilistic model; providing (910) the output of the first subencoder and the output of the second subencoder to the probabilistic model; obtaining one or more first probabilities from the output of the probabilistic model; and obtaining an encoded chroma component from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0116] A decoding method according to an embodiment is shown in Figure 9b, the method comprising receiving input data including encoded chroma components and reconstructed luma components into a decoder (920), wherein the decoder includes at least a first sub-decoder, a second sub-decoder, and a probabilistic model, the method comprising receiving (920), providing the reconstructed luma components to the second sub-decoder (922), obtaining one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components (924), obtaining an entropy-decoded chroma component based at least on the encoded chroma components and one or more probabilities (926), providing the entropy-decoded chroma components to the first sub-decoder (928), and obtaining a reconstructed chroma component from the output of the first sub-decoder (930).

[0117] Therefore, the method reflects the encoding and decoding modes in a codec comprising two separate codecs for encoding the rumor and chroma components of the input data used, the first codec comprising a first encoder and a first decoder, and the second codec comprising a second encoder and a second decoder. These two codecs may be referred to as the rumor codec and the chroma codec, respectively. The (first) encoder of the rumor codec receives the entire ground truth data, including both the rumor and chroma components, as input, and the (first) decoder of the rumor codec reconstructs only the rumor component. The bitrate output by the encoder of the rumor codec represents the encoded rumor component. Subsequently, the (second) encoder of the chroma codec may receive the entire ground truth data, including both the rumor and chroma components, as input. The (second) encoder of the chroma codec also receives the reconstructed rumor component as input. The ground truth data and the reconstructed chroma components are processed using their respective sub-encoders, and some of the outputs of the sub-encoders are fed into the chroma codec's probabilistic model, which estimates the probabilities used to encode the outputs of the sub-encoders processing the ground truth data. The bitstream output by the chroma codec's (second) encoder represents the encoded chroma components.

[0118] The (second) decoder of the chroma codec receives the encoded chroma component and the reconstructed luma component. The reconstructed luma component is input to a second sub-decoder of the decoder. The decoder has a probabilistic model, preferably identical, or at least substantially identical, to the encoder of the chroma codec, for providing one or more probabilities based on at least the output of the second sub-decoder and the previously decoded chroma component. The decoded chroma component is determined based on at least the encoded chroma component and one or more probabilities and is input to the first sub-decoder. The first sub-decoder of the chroma codec then reconstructs only the chroma component.

[0119] By using the reconstructed lumar components as additional input to the chromacodec's probabilistic model (on both the encoder and decoder sides), the chromacodec improves its rate distortion characteristics conditionally on the reconstructed lumar components. Therefore, the rate distortion characteristics of the entire codec (the combination of the lumarcodec and chromacodec) are improved.

[0120] According to one embodiment, the first and second encoders and decoders belong to an end-to-end learning intra-frame codec or an end-to-end learning image codec.

[0121] Therefore, using learning-based video compression, particularly in the context of end-to-end learning-based intra-frame codecs, it is possible to improve rate distortion performance. However, it should be noted that the method and related embodiments are also applicable to end-to-end learning-based inter-frame codecs.

[0122] According to one embodiment, the learning intraframe codec is trained in an end-to-end manner by minimizing D+λR, where D is the distortion loss term, R is the rate loss term, and λ is a weight that controls the balance between the losses.

[0123] The distortion loss term may be calculated based on at least the distortion function, ground truth data, and the output of the decoder included in the trained intraframe codec. The rate loss term may be calculated based on at least an estimate of the bitrate (e.g., the size of the bitstream in bits) of the bitstream output of the encoder included in the trained intraframe codec.

[0124] This optimization process results in a so-called rate-distortion trade-off, where a balance is found between distortion D and rate loss R. Rate loss may indicate the bitrate of the encoded image, and distortion is defined as follows: - Mean squared error (MSE), - Multiscale structural similarity (MS-SSIM), - Multiple strain losses, such as the weighted sum of MSE and MS-SSIM, - Other metrics for evaluating the quality of reconstructed images You may also indicate pixel fidelity distortions such as those mentioned above.

[0125] According to one embodiment, training of a learning-type intraframe codec is performed jointly with respect to distortion loss D and rate loss R.

[0126] According to another embodiment, training of a learning intraframe codec is performed in two alternating phases, where only the distortion loss D is used in the first of the two alternating phases, and only the rate loss R is used in the second of the two alternating phases.

[0127] Various embodiments of the implementation options are described below. Note that the terms “single frame,” “frame,” and “image” may be used interchangeably. These terms may refer to input data to an end-to-end (e2e) learning intraframe codec. For brevity, in at least some embodiments, an image is considered a data type. However, the embodiments described herein and their underlying principles may be extended to other types of data, such as video and audio. Also, in at least some of the embodiments, YUV is considered the input color format. However, the embodiments may be similarly extended to other color formats, such as RGB. In the YUV format, “Y” represents the lightness or “luma” value, and “U” and “V” represent the color or “chroma” value. In one example, the input image may be an image in YUV4:4:4 color format and may be represented as a three-dimensional array (or tensor) having a size of 256 × 256 × 3, with a horizontal size of 256 pixels and a vertical size of 256 pixels, and the three channels corresponding to the Y, U, and V components, respectively. In another example, the input image may be in YUV4:2:0 color format and may be represented by a combination of a 256x256 matrix for the lumens and a 128x128x2 two-dimensional array (or tensor) for the chromens.

[0128] Generally, an end-to-end (e2e) learning intraframe codec is analogous to an end-to-end (e2e) learning image compression system. Typically, it may consist of an image encoder, a quantizer, a probabilistic model, an entropy codec (e.g., an arithmetic encoder / decoder), an inverse quantizer, and an image decoder. The image encoder, probabilistic model, and image decoder may primarily consist of neural network components. The quantizer, entropy codec, and inverse quantizer are typically not based on neural network components, but may still contain them.

[0129] An e2e learning intraframe codec may be used as part of a video codec, and the intraframe codec may code each of one or more first frames of a video independently of any other frames, and another codec called an interframe codec may code each of one or more second frames of a video based on at least one or more other frames, for example, based on at least data obtained from one or more first frames.

[0130] Lumacodec The input to the Rumacodec may include at least ground truth data (e.g., blocks or the entire image), and at least a portion of the ground truth data will be encoded / compressed. The input to the Rumacodec may also include one or more additional data, such as block / image resolution. Examples of input ground truth data are as follows: - Ground truth including both lumern and chroman components. For example, the ground truth may be an image in the YUV4:4:4 color format represented as a 256×256×3 multidimensional array or tensor, with a horizontal size of 256 pixels, a vertical size of 256 pixels, and three channels corresponding to the Y, U, and V components, respectively. Alternatively, the ground truth may be an image in the YUV4:2:0 color format represented by a combination of a 256×256 matrix for the lumern components and a 128×128×2 two-dimensional array (or tensor) for the chroman components. - Ground truth containing only the rumor component. For example, the ground truth may be a 256x256x1 image, with a horizontal size of 256 pixels, a vertical size of 256 pixels, and one channel for the Y component. That is the case.

[0131] Figure 10 illustrates an exemplary implementation of a ruma component codec according to one embodiment. The input data x is ground truth containing both ruma and chroma components, where h × w × 3 represents the size of x having height h, width w, and number of channels 3. The ruma component codec (i.e., the first codec) is used to encode the ruma component of the input data; that is, the bitstream output by the encoder of the ruma codec (i.e., the first encoder) represents the encoded ruma component, and the output of the decoder (i.e., the first decoder) represents the reconstructed or decoded ruma component.

[0132] The encoder 1000 of the Luma component codec may comprise a neural encoder 1002, a quantizer, a probabilistic model 1004, and an entropy encoder 1006. In the figure, the quantizer is not shown for simplification. The neural encoder 1002 may comprise a first convolutional layer ("Conv5×5,48,1", where conv represents convolution, 5×5 is the kernel size, 48 is the number of output channels, and 1 is the stride value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. The neural encoder 1002 outputs a latent tensor which may be quantized. A latent tensor or quantized latent tensor is input to a probabilistic model 1004, the dimensions of which may be h / / 16 × w / / 16 × 128, where h / / 16 indicates the height, w / / 16 indicates the width, and 128 indicates the number of channels in the latent tensor. The probabilistic model outputs an estimate of the probability of each element of the (quantized) latent tensor. The probabilistic model may be learned from the data using machine learning techniques, for example, the probabilistic model may be a neural network. On the encoder side, the output of the probabilistic model 1004 is used as one of the inputs to the entropy encoder 1006. The entropy encoder may be an arithmetic encoder. The entropy encoder receives at least the (quantized) latent tensor and the output of the probabilistic model and outputs a bitstream 1008.

[0133] The decoder 1010 of the Luma component codec may comprise an entropy decoder 1012, a probabilistic model 1014, an inverse quantizer, and a neural decoder 1016. In the figure, the inverse quantizer is not shown for simplification. Probabilistic models 1014 and 1004 are assumed to be identical or substantially identical, for example, they may be copies of the same probabilistic model. The entropy decoder 1012 may be an arithmetic decoder. The entropy decoder receives at least the output of the bitstream 1008 and the probabilistic model 1014 and outputs a (quantized) decoded latent tensor. The decoded latent tensor may then undergo inverse quantization. The decoded latent tensor or the inversely quantized decoded latent tensor is then input to the neural decoder 1016. The neural decoder may include a first transposed convolutional layer ("UpConv5×5,384,2", where UpConv refers to transposed convolution, 5×5 is the kernel size, 384 is the number of output channels, and 2 is the stride value), a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer.

[0134] Output of neural decoder 1010

number

[0135] Chroma codec The input to the chroma component codec may include at least ground truth data (e.g., blocks or the entire image), and at least a portion of the ground truth will be encoded / compressed. The input to the chroma codec may also include one or more additional data, such as block / image resolution. Examples of input ground truth data are as follows: - Ground truth including both lumern and chroman components. For example, the ground truth may be an image in the YUV4:4:4 color format represented as a 256×256×3 multidimensional array or tensor, with a horizontal size of 256 pixels, a vertical size of 256 pixels, and three channels corresponding to the Y, U, and V components, respectively. In another example, the ground truth may be an image in the YUV4:2:0 color format represented by a combination of a 256×256 matrix for the lumern components and a 128×128×2 two-dimensional array (or tensor) for the chroman components. - Ground truth containing only the chroma component. For example, the ground truth may be a 128x128x2 image, with a horizontal size of 128 pixels and a vertical size of 128 pixels, and two channels corresponding to the U and V components, respectively. That is the case.

[0136] Figure 11 illustrates an exemplary implementation of a chroma component codec according to one embodiment. The input data x is ground truth containing both lumana and chroma components, where h × w × 3 represents the size of x having height h, width w, and number of channels 3. For clarity, in this example, the input data to the chroma component codec is the same as the input data to the lumana component codec. The chroma component codec is used to encode the chroma component of the input data; that is, the bitstream output by the encoder of the chroma codec represents the encoded chroma component, and the output of the decoder of the chroma codec represents the reconstructed or decoded chroma component.

[0137] The encoder 1100 of the chroma component codec may comprise a neural encoder (i.e., a first sub-encoder) 1102, an auxiliary encoder (i.e., a second sub-encoder) 1104, a quantizer (not shown), a probabilistic model 1106, and an entropy encoder 1108. Similar to the neural encoder 1002 of the chroma component codec, the neural encoder 1102 of the chroma component codec may also comprise a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. The dimension of the latent tensor is h / / 16 × w / / 16 × 64, the height is h / / 16, the width is w / / 16, and the number of channels is 64.

[0138] The chroma component codec also includes an encoder-side auxiliary encoder 1104 that generates an auxiliary input to the probabilistic model 1106, and a decoder-side auxiliary encoder 1118 that generates an auxiliary input to the probabilistic model 1116. In one exemplary implementation, the encoder-side auxiliary encoder 1104 and the decoder-side auxiliary encoder 1118 may be two copies of the same component, for example, a neural network having the same architecture and weights. In another exemplary implementation, the encoder-side auxiliary encoder 1104 and the decoder-side auxiliary encoder 1118 may be different neural network components, for example, a neural network having a different architecture, or a neural network with the same architecture but different weights.

[0139] According to one embodiment, the input to the auxiliary encoder

number

number

[0140] According to another embodiment, the input to the auxiliary encoder

number

[0141] A masked version of the reconstructed rumor component may be obtained via a masking operation performed on the reconstructed rumor component. The masking operation may mask out some of the elements of the reconstructed rumor component (for example, by setting them to zero or other predetermined values), such as elements whose spatial coordinates do not correspond to the spatial coordinates of elements in the chroma latent tensor being coded or decoded by an entropy encoder or entropy decoder.

[0142] According to another embodiment, the input to the auxiliary encoder

number

[0143] A smoothed version of the reconstructed rumor component may be obtained through a smoothing operation performed on the reconstructed rumor component. The smoothing operation may remove elements of the reconstructed rumor component that represent high-frequency information, such as elements whose information does not correspond to or does not correlate well with the information of the elements in the chroma component.

[0144] According to another embodiment, the input to the auxiliary encoder

number

[0145] Therefore, input

number

[0146] In yet another embodiment, the input to the auxiliary encoder

number

[0147] The auxiliary encoder may have the same architecture as the neural encoder of the chroma component codec, but any suitable architecture for extracting features from an image may be suitable. The output of the auxiliary encoder 1104 is input to the probabilistic model 1106 to act as additional contextual information.

[0148] According to another embodiment, the additional contextual information provided to the probabilistic model includes data obtained based on the reconstructed rumor components.

[0149] According to another embodiment, the additional contextual information provided to the probabilistic model includes a lumern latent tensor or an inversely quantized lumern latent tensor. Since the reconstructed lumern components consist of sufficient high-frequency information from the input data, the auxiliary input to the probabilistic model 1106 can assist in the estimation of the chromern latent probability density function. This probabilistic model can result in significant performance improvements in coding the chromern components.

[0150] The latent tensor (e.g., chroma latent tensor) output by the chroma codec encoder 1102 may be input to a probabilistic model. The probabilistic model 1106 may be a neural network. Using auxiliary inputs as additional contextual information, the probabilistic model 1106 outputs estimates of the probabilities of one or more elements of the chroma latent tensor. On the encoder side, the output of the probabilistic model 1106 is used as one of the inputs to the entropy encoder 1108. The entropy encoder may be an arithmetic encoder. The entropy encoder 1108 receives at least the (quantized) latent tensor and the output of the probabilistic model 1108 and outputs a bitstream 1110 representing the encoded chroma components.

[0151] The chroma component codec decoder 1112 may comprise an entropy decoder 1114, a probabilistic model 1116, an inverse quantizer (not shown), an auxiliary encoder 1118, and a neural decoder 1120. The entropy decoder 1114 may be an arithmetic decoder. The entropy decoder 1114 receives at least the bitstream 1110 and the output of the probabilistic model 1116 and outputs a decoded (quantized) latent tensor. The probabilistic model 1116 may need to be the same or substantially the same probabilistic model 1108 available on the encoder side. In one exemplary embodiment, the auxiliary encoder 1118 is the same or substantially the same auxiliary encoder 1104 available on the encoder side. The decoded (quantized) latent tensor may undergo inverse quantization. After inverse quantization, the inversely quantized decoded latent tensor may be concatenated with the auxiliary input along the channel dimension. After concatenation, the dimension of the de-quantized decoded latent tensor may be h / / 16 × w / / 16 × 128. The new decoded latent tensor is then input to the neural decoder 1120. The neural decoder 1120 may comprise a first transposed convolutional layer, a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a nonlinear activation function ReLU, and a convolutional layer.

[0152] The output of the neural decoder 1120 is a reconstruction of the chroma component, which may have a size of h / / 2 × w / / 2 × 2. In this exemplary implementation, since the input data is in YUV4:2:0 format, the chroma component has a spatial size that is one-quarter smaller than the lumana component. Thus, the chroma component has half the height of the lumana component, and the chroma component has half the width of the lumana component. Finally, one channel of the output is

[0153]

number

number

[0154]

number

number

[0155] In one embodiment, a rumor component codec may be first trained on a certain number of training images or blocks, and then a chroma component codec may be trained at least on reconstructed rumor components obtained based on the trained rumor component codec. In one embodiment, the rumor component codec and the chroma component codec are trained in two consecutive steps, in which the rumor component codec is trained only in the first step and the chroma component codec is trained only in the second step. In another embodiment, the two steps of the above embodiment may be performed multiple times, such as in which the rumor component codec is trained only in the first step, the chroma component codec is trained only in the second step, the rumor component codec is trained only in the third step, the chroma component codec is trained only in the fourth step, and so on.

[0156] In another embodiment, the rumor component codec and the chroma component codec may be trained together, i.e., the chroma component codec is conditional on the rumor component codec, and both are trained simultaneously or substantially simultaneously. Alternatively, the rumor component codec and the chroma component codec may be trained in alternating phases, for example, the rumor component codec being trained for a first number of iterations, then the chroma component codec being trained for a second number of iterations, then the rumor component codec being trained for a third number of iterations, then the chroma component codec being trained for a fourth number of iterations, and so on.

[0157] According to one embodiment, the intraframe codec includes a first codec used to code the chroma component of an input image, and a second codec used to code the lumen component of an input image, at least based on the reconstructed chroma component.

[0158] The methods presented above regarding coding and decoding modes may be implemented in each coding and decoding device. One embodiment of the device includes means for receiving input data including rumor and chroma components, means for providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data, means for obtaining an encoded rumor component from the output of the first encoder, means for providing the encoded rumor component to a first decoder, means for obtaining a reconstructed rumor component from the output of the first decoder, and a second ground truth data including at least a second portion of the input data and the reconstructed rumor component. Means for providing truth data to a second encoder, the second encoder comprising at least a first subencoder for ground truth data, a second subencoder for reconstructed chroma components, and a probabilistic model; means for providing the outputs of the first subencoder and the second subencoder to the probabilistic model; means for obtaining one or more first probabilities from the output of the probabilistic model; and means for obtaining encoded chroma components from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0159] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

[0160] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder comprises means for converting input data into a plurality of latent tensor elements, the probabilistic model comprises means for estimating the probability of each latent tensor element, and the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element.

[0161] According to one embodiment, the first subencoder of the second encoder is a neural encoder, and the second subencoder of the second encoder is an auxiliary encoder that includes means for generating an auxiliary input to a probabilistic model.

[0162] According to one embodiment, the input to the auxiliary encoder is the reconstruction of the luma component.

[0163] According to one embodiment, the input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0164] According to one embodiment, the input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0165] According to one embodiment, the input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction from the reconstructed luma component.

[0166] An apparatus in another embodiment is a means for receiving input data including encoded chroma components and reconstructed luma components into a decoder, the decoder comprising at least a first sub-decoder, a second sub-decoder, and a probabilistic model, means for providing the reconstructed luma components to the second sub-decoder, means for obtaining one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components, means for obtaining an entropy-decoded chroma component based at least on the encoded chroma components and one or more probabilities, means for providing the entropy-decoded chroma components to the first sub-decoder, and means for obtaining the reconstructed chroma components from the output of the first sub-decoder.

[0167] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, the decoder's probabilistic model comprises means for estimating the probability of each decoded latent tensor element, the entropy decoder comprises means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probability of each latent tensor element, and the neural decoder comprises means for converting the plurality of decoded latent tensor elements into reconstructed chroma components.

[0168] According to one embodiment, the first sub-decoder of the decoder is a neural decoder, and the second sub-decoder of the decoder is an auxiliary decoder equipped with means for generating an auxiliary input to a probabilistic model.

[0169] According to one embodiment, the device includes means for concatenating the decoded latent tensor with an auxiliary input decoder along the dimension of the latent tensor channel.

[0170] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0171] In a further embodiment, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory has code thereon, and the code, when executed by the at least one processor, means for receiving input data including rumor and chroma components, and for providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data, and for obtaining an encoded rumor component from the output of the first encoder, and for providing the encoded rumor component to a first decoder, and for obtaining a reconstructed rumor component from the output of the first decoder. The device provides a second encoder with second ground truth data, which includes at least a second portion of the input data and a reconstructed rumor component, wherein the second encoder comprises at least a first subencoder for the ground truth data, a second subencoder for the reconstructed rumor component, and a probabilistic model; provides the output of the first subencoder and the output of the second subencoder to the probabilistic model; obtains one or more first probabilities from the output of the probabilistic model; and obtains an encoded chroma component from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0172] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

[0173] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder comprises code configured to cause the device to convert input data into a plurality of latent tensor elements, the probabilistic model comprises code configured to cause the device to estimate the probability of each latent tensor element, and the entropy encoder comprises code configured to cause the device to output a bitstream encoded at least partially on the plurality of latent tensor elements and the probability of each latent tensor element.

[0174] According to one embodiment, the first subencoder of the second encoder is a neural encoder, and the second subencoder of the second encoder is an auxiliary encoder comprising code configured to cause the device to generate an auxiliary input to a probabilistic model.

[0175] According to one embodiment, the input to the auxiliary encoder is the reconstruction of the luma component.

[0176] According to one embodiment, the input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0177] According to one embodiment, the input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0178] According to one embodiment, the input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction from the reconstructed luma component.

[0179] In a further embodiment, an apparatus is provided comprising at least one processor and at least one memory, wherein the at least one memory has code thereon, and the code, when executed by the at least one processor, receives input data including encoded chroma components and reconstructed luma components into a decoder, the decoder comprising at least a first sub-decoder, a second sub-decoder, and a probabilistic model, wherein the apparatus is made to perform at least the following: receive, provide the reconstructed luma components to the second sub-decoder, obtain one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components, obtain an entropy-decoded chroma component based at least on the encoded chroma components and one or more probabilities, provide the entropy-decoded chroma components to the first sub-decoder, and obtain a reconstructed chroma component from the output of the first sub-decoder.

[0180] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, wherein the decoder's probabilistic model includes code configured to cause the device to estimate the probability of each decoded latent tensor element, the entropy decoder includes code configured to cause the device to output a plurality of decoded latent tensor elements based at least partially on the input data and the probability of each latent tensor element, and the neural decoder includes code configured to cause the device to convert the plurality of decoded latent tensor elements into reconstructed chroma components.

[0181] According to one embodiment, the first sub-decoder of the decoder is a neural decoder, and the second sub-decoder of the decoder is an auxiliary decoder comprising code configured to cause the device to generate an auxiliary input to a probabilistic model.

[0182] According to one embodiment, the device includes a code configured to cause the device to concatenate the decoded latent tensor with an auxiliary input decoder along the dimension of the latent tensor channel.

[0183] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0184] Such a device may include, for example, a functional unit disclosed in any of Figures 1 to 7 for carrying out an embodiment.

[0185] Such a device further comprises code stored in the at least one non-temporary memory, which, when executed by the at least one processor, causes the device to implement one or more of the embodiments disclosed herein.

[0186] While exemplary embodiments are described above with reference to the encoder, it should be understood that the resulting bitstream and decoder may have corresponding elements in them. Similarly, while exemplary embodiments are described with reference to the decoder, it should be understood that the encoder may have a structure and / or computer program for generating the bitstream to be decoded by the decoder. For example, some embodiments are described in relation to generating prediction blocks as part of encoding. Embodiments can be similarly realized by generating prediction blocks as part of decoding, but differ in that coding parameters such as horizontal and vertical offsets are decoded from the bitstream rather than being determined by the encoder.

[0187] The embodiments of the present invention described above describe the codec in the context of separate encoder and decoder devices to aid in understanding the associated processes. However, it will be apparent that the device, structure, and operation may be implemented as a single encoder-decoder device / structure / operation. Furthermore, it is possible that the encoder and decoder may share some or all common elements.

[0188] While the above examples illustrate embodiments of the present invention operating within a codec in an electronic device, it will be apparent that the present invention, as defined in the claims, may be implemented as part of any video codec. Therefore, for example, embodiments of the present invention may be implemented in a video codec that performs video coding over a fixed or wired communication path.

[0189] Therefore, the user device may include a video codec such as the video codec described in the embodiments of the present invention above. It will be clear that the term "user device" is intended to cover any suitable type of wireless user device, such as a mobile phone, portable data processing device, or portable web browser.

[0190] Furthermore, elements of the Public Land Mobile Network (PLMN) may also be equipped with video codecs as described above.

[0191] In general, various embodiments of the present invention may be implemented in hardware, dedicated circuitry, software, logic, or any combination thereof. For example, some embodiments may be implemented in hardware, while others may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited to these. Various embodiments of the present invention may be illustrated and described using block diagrams, flowcharts, or some other illustrations, but it should be understood that these blocks, apparatus, systems, techniques, or methods described herein may, as non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controller or other computing device, or any combination thereof.

[0192] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device such as a processor entity, by hardware, or by a combination of software and hardware. Furthermore, it should be noted that any block of the logic flow shown in the figure may represent a program step, or an interconnected logic circuit, block, and function, or a combination of a program step and a logic circuit, block, and function. The software may be stored on a physical medium such as a memory chip or memory block implemented within the processor, a magnetic medium such as a hard disk or floppy disk, or an optical medium such as a DVD and its data variants, or a CD.

[0193] Memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, in non-limiting examples, one or more of general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures.

[0194] Embodiments of the present invention may be put into practice in various components, such as integrated circuit modules. Designing integrated circuits is generally a highly automated process. Complex and powerful software tools are available to translate logic-level designs into semiconductor circuit designs prepared for etching and forming on semiconductor substrates.

[0195] Programs such as those offered by Synopsys, Inc. in Mountain View, California, and Cadence Design, Inc. in San Jose, California, use well-established design rules, along with a library of pre-stored design modules, to automatically route conductors and place components on a semiconductor chip. Once the design for the semiconductor circuit is complete, the resulting design may be sent to a semiconductor manufacturing facility or "fab" in a standardized electronic format (e.g., Opus or GDSII).

[0196] The foregoing description, using illustrative and non-limiting examples, has provided a sufficient and useful description of exemplary embodiments of the invention. However, various modifications and adaptations will become apparent to those skilled in the art who have read the foregoing description in conjunction with the accompanying drawings and claims. Nevertheless, all such modifications and similar modifications of the teachings of the invention will still fall within the scope of the invention.

Claims

1. Means for receiving input data including rumor and chroma components, Means for providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; A means for obtaining the encoded luma component from the output of the first encoder, Means for providing the encoded luma component to the first decoder, A means for obtaining the reconstructed luma component from the output of the first decoder, Means for providing a second ground truth data to a second encoder, comprising at least a second portion of the input data and the reconstructed rumor component, wherein the second encoder comprises at least a first sub-encoder for the ground truth data, a second sub-encoder for the reconstructed rumor component, and a probabilistic model. Means for providing the output of the first sub-encoder and the output of the second sub-encoder to the probabilistic model, Means for obtaining one or more first probabilities from the output of the aforementioned probability model, A means for obtaining an encoded chroma component from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities. A device equipped with the following features.

2. The apparatus according to claim 1, wherein the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

3. The first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder. The neural encoder comprises means for converting the input data into a plurality of latent tensor elements, The aforementioned probability model includes means for estimating the probability of each latent tensor element, The apparatus according to claim 1 or 2, wherein the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probabilities of each latent tensor element.

4. The apparatus according to any one of claims 1 to 3, wherein the first sub-encoder of the second encoder is a neural encoder, and the second sub-encoder of the second encoder is an auxiliary encoder comprising means for generating an auxiliary input to the probabilistic model.

5. The apparatus according to claim 4, wherein the input to the auxiliary encoder is the reconstruction of the luma component.

6. The apparatus according to claim 4, wherein the input to the auxiliary encoder is a masked version of the reconstructed luma component.

7. The apparatus according to claim 4, wherein the input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

8. The apparatus according to claim 4, wherein the input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction from the reconstructed rumor component.

9. Receiving input data including rumor and chroma components, To provide first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data. From the output of the first encoder, the encoded luma component is obtained, The encoded luma component is provided to the first decoder, From the output of the first decoder, the reconstructed luma component is obtained, The present invention provides a second encoder with second ground truth data comprising at least a second portion of the input data and the reconstructed rumor component, wherein the second encoder comprises at least a first subencoder for the ground truth data, a second subencoder for the reconstructed rumor component, and a probabilistic model. The output of the first sub-encoder and the output of the second sub-encoder are provided to the probabilistic model, Obtaining one or more first probabilities from the output of the aforementioned probability model, Based on the output of the first sub-encoder and the one or more probabilities, the encoded chroma component is obtained from the output of the second encoder. Methods that include...

10. A means for receiving input data including encoded chroma components and reconstructed luma components into a decoder, wherein the decoder comprises at least a first sub-decoder, a second sub-decoder, and a probabilistic model, Means for providing the reconstructed luma component to the second sub-decoder, Means for obtaining one or more probabilities from the output of the probabilistic model, based at least on the output of the second sub-decoder and the previously decoded chroma components, Means for obtaining an entropy-decoded chroma component based on the encoded chroma component and at least one or more probabilities, Means for providing the entropy-decoded chroma component to the first sub-decoder, A means for obtaining the reconstructed chroma component from the output of the first sub-decoder, A device equipped with the following features.

11. The decoder comprises an entropy decoder and a neural decoder, The probability model of the decoder comprises means for estimating the probability of each decoded latent tensor element. The entropy decoder comprises means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of each latent tensor element, The apparatus according to claim 10, wherein the neural decoder comprises means for converting the plurality of decoded latent tensor elements into the reconstructed chroma components.

12. The apparatus according to claim 10 or 11, wherein the first sub-decoder of the decoder is a neural decoder, and the second sub-decoder of the decoder is an auxiliary decoder comprising means for generating an auxiliary input to the probabilistic model.

13. The apparatus according to claim 12, further comprising means for concatenating the decoded latent tensor with the auxiliary input decoder along the dimension of the latent tensor channel.

14. The apparatus according to any one of claims 10 to 13, wherein the decoder belongs to an end-to-end learning type intraframe codec.

15. The process involves receiving input data containing encoded chroma components and reconstructed luma components into a decoder, wherein the decoder comprises at least a first sub-decoder, a second sub-decoder, and a probabilistic model. The reconstructed luma component is provided to the second sub-decoder, Obtaining one or more probabilities from the output of the probabilistic model based at least on the output of the second sub-decoder and the previously decoded chroma components, Obtaining an entropy-decoded chroma component based on the encoded chroma component and at least one of the probabilities, The entropy-decoded chroma component is provided to the first sub-decoder, The reconstructed chroma component is obtained from the output of the first sub-decoder. Methods that include...

Images