A device, a method, and a computer program for encoding and decoding video.

Description

1 / 59 A device, a method, and a computer program for encoding and decoding video. TECHNICAL FIELD

[0001] The present invention relates to an apparatus, a method and a computer program for encoding and decoding video. BACKGROUND

[0002] In video encoding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (luma) channel, also denoted as Y, and two chrominance (chroma) channels, also denoted as U, V or as Cb, Cr. In these cases, the luminance channel, primarily representing the scene's illumination, is typically encoded at a certain resolution, while the chrominance channels, typically representing differences between certain color components, are often encoded at a second, lower resolution than the luminance signal. The intention of this type of differential representation is to decorrelate the color components and be able to compress the data more efficiently.

[0003] Neural networks (NNs) have been used in the context of image and video compression, replacing one or more of the components of a traditional codec, or using end-to-end learned compression. For intra-sample prediction of luma and chroma, separate prediction blocks were used for luma and chroma components, where the intra-chroma prediction block can perform cross-component prediction of the luma component. It is known that the luma component and the chroma components of at least neighboring samples typically have some correlation in their values.

[0004] However, NN-based video coding approaches, including end-to-end learned compression-based solutions, have proven inadequate considering this correlation. Thus, the Petition 870250090159, dated 03 / 10 / 2025, page 11 / 226 2 / 59 rate distortion performance of the codecs has been suboptimal. SUMMARY

[0005] To at least mitigate the above problems, improved methods for enhancing rate distortion performance are introduced in this document.

[0006] The scope of protection sought for various embodiments of the invention is established by the independent claims. The embodiments and features, if any, described in this descriptive report that are not covered by the scope of the independent claims should be interpreted as useful examples for understanding various embodiments of the invention.

[0007] A method according to a first aspect comprises receiving input data comprising a luma component and a chroma component; providing first fundamental truth data to a first encoder, said fundamental truth data comprising at least a first part of the input data; obtaining an encoded luma component from an output of the first encoder; providing the encoded luma component to a first decoder; obtaining a reconstructed luma component from an output of the first decoder; providing second fundamental truth data comprising at least a second part of the input data and the reconstructed luma component to a second encoder, said second encoder comprising at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model;Provide an output from the first sub-encoder and an output from the second sub-encoder to the probability model; obtain one or more first probabilities from an output of the probability model; and obtain, based on an output from the first sub-encoder and one or more probabilities, a chroma-encoded component from an output of the second encoder.

[0008] An apparatus according to a second aspect comprises means for receiving input data comprising a luma component and a Petition 870250090159, dated 03 / 10 / 2025, page 12 / 226 3 / 59 chroma component; means for providing first fundamental truth data for a first encoder, wherein said fundamental truth data comprise at least a first part of the input data; means for obtaining an encoded luma component from an output of the first encoder; means for providing the encoded luma component for a first decoder; means for obtaining a reconstructed luma component from an output of the first decoder; means for providing second fundamental truth data comprising at least a second part of the input data and the reconstructed luma component for a second encoder, wherein said second encoder comprises at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model;means for providing an output from the first subencoder and an output from the second subencoder to the probability model; means for obtaining one or more first probabilities from an output of the probability model; and means for obtaining, based on an output from the first subencoder and one or more probabilities, a chroma-encoded component from an output of the second encoder.

[0009] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learned intraframe codec.

[0010] According to one embodiment, said first encoder comprises a neural encoder, a probability model, and an entropy encoder; wherein the neural encoder comprises means for converting the input data into a plurality of latent tensor elements; the probability model comprises means for estimating a probability of each latent tensor element; and the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element. Petition 870250090159, dated 03 / 10 / 2025, page 13 / 226 4 / 59

[0011] According to one embodiment, said first sub-encoder of the second encoder is a neural encoder and said second sub-encoder of the second encoder is an auxiliary encoder comprising means for generating an auxiliary input to the probability model.

[0012] According to one embodiment, an input to the auxiliary encoder is a reconstruction of the luma component.

[0013] According to one embodiment, an input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0014] According to one embodiment, an input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0015] According to one embodiment, an input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction of the reconstructed luma component.

[0016] A method according to a third aspect comprises receiving input data comprising an encoded chroma component and a reconstructed luma component in a decoder, said decoder comprising at least a first subdecoder, a second subdecoder and a probability model; providing the reconstructed luma component to the second subdecoder; obtaining one or more probabilities from an output of the probability model, based on at least one output of the second subdecoder and a previously decoded chroma component; obtaining an entropy-decoded chroma component based on at least the encoded chroma component and one or more probabilities; providing the entropy-decoded chroma component to the first subdecoder; and obtaining a reconstructed chroma component from an output of the first subdecoder.

[0017] An apparatus according to a fourth aspect comprises means for receiving input data comprising an encoded chroma component and a luma component reconstructed in a decoder, said decoder comprising at least one first subdecoder, a Petition 870250090159, dated 03 / 10 / 2025, page 14 / 226 5 / 59 second subdecoder and a probability model; means for providing the reconstructed luma component for the second subdecoder; means for obtaining one or more probabilities from an output of the probability model, based on at least one output of the second subdecoder and a previously decoded chroma component; means for obtaining an entropy-decoded chroma component based on at least the encoded chroma component and one or more probabilities; means for providing the entropy-decoded chroma component for the first subdecoder; and means for obtaining a reconstructed chroma component from an output of the first subdecoder.

[0018] According to one embodiment, said decoder comprises an entropy decoder and a neural decoder; wherein the probability model of the decoder comprises means for estimating a probability of each decoded latent tensor element; the entropy decoder comprises means for outputting a plurality of at least partially decoded latent tensor elements based on the input data and the probability of each latent tensor element; and the neural decoder comprises means for converting the plurality of decoded latent tensor elements into the reconstructed chroma component.

[0019] According to one embodiment, said first subdecoder of the decoder is a neural decoder and said second subdecoder of the decoder is an auxiliary decoder comprising means for generating an auxiliary input to the probability model.

[0020] According to one embodiment, the apparatus comprises means for concatenating the decoded latent tensors with the auxiliary input decoder along dimensions of latent tensor channels.

[0021] According to one embodiment, the decoder belongs to an end-to-end learned intraframe codec.

[0022] Computer-readable storage media stored with code on them are arranged to perform the above methods and a Petition 870250090159, dated 03 / 10 / 2025, page 15 / 226 6 / 59 or more of the modalities related to them. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] To better understand the present invention, reference will now be made by way of example to the accompanying drawings, in which:

[0024] Figure 1 shows an example of a codec with neural network (NN) components;

[0025] Figure 2 shows another example of a video encoding system with neural network components;

[0026] Figure 3 shows an example of an end-to-end learned codec based on a neural network;

[0027] Figure 4 shows an example of an end-to-end learned video coding system based on a neural network;

[0028] Figure 5 shows an example of a video encoding for machines;

[0029] Figure 6 shows an example of an end-to-end learned system pipeline for machine video encoding;

[0030] Figure 7 shows an example of training a codec learned end-to-end;

[0031] Figure 8 shows an example of a Dense Divided Attention (DSA) block;

[0032] Figures 9a and 9b show a flowchart of an encoding method and a decoding method for improving rate distortion performance according to an embodiment of the invention;

[0033] Figure 10 illustrates an exemplary implementation of a luma component codec according to an embodiment of the invention; and

[0034] Figure 11 illustrates an exemplary implementation of a chroma component codec according to an embodiment of the invention. DETAILED DESCRIPTION OF SOME EXEMPLARY MODALITIES

[0035] The following description and drawings are illustrative and should not be used Petition 870250090159, dated 03 / 10 / 2025, page 16 / 226 7 / 59 interpreted as unnecessarily limiting. Specific details are provided for a full understanding of the revelation. However, in certain cases, well-known or conventional details are not described in order to avoid obscuring the description. References to one or a (1) modality in the present revelation may be, but do not necessarily are, references to the same modality and such references mean at least one of the modalities.

[0036] The reference in this descriptive report to “a (1) modality” or to “a modality” means that a particular feature, structure or characteristic described in conjunction with the modality is included in at least one modality of the disclosure.

[0037] Before discussing the present embodiments in more detail, a brief reference to the related technology is provided.

[0038] In the context of machine learning, a neural network (NN) is a computation graph consisting of multiple computation layers, that is, multiple computation portions. Each layer consists of one or more units, where each unit performs an elementary computation. A unit is connected to one or more other units, and the connection may be associated with a weight. The weight can be used to scale the signal passing through the associated connection. Weights are learnable parameters, that is, values ​​that can be learned from training data. There may be other learnable parameters, such as those from batch normalization layers.

[0039] Two widely used architectures for neural networks are feed-forward and recurrent architectures. Feed-forward neural networks are those in which there is no feedback loop: each layer receives input from one or more of the preceding layers and provides its output as input to one or more of the subsequent layers. In addition, units within a certain layer receive input from units in one or more of the preceding layers and provide output to one or more of the following layers.

[0040] Initial layers (those closest to the input data) extract Petition 870250090159, dated 03 / 10 / 2025, page 17 / 226 8 / 59 semantically extract low-level features, such as edges and textures in images, and intermediate and final layers extract higher-level features. After the feature extraction layers, there may be one or more layers performing a certain task, such as classification, semantic segmentation, object detection, noise suppression, style transfer, super-resolution, etc. In recurrent neural networks, there is a feedback loop, so the network becomes stateful, that is, it is able to memorize information or a state.

[0041] Neural networks are being used in an increasing number of applications for many different types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, device usage data analysis, etc.

[0042] One of the important properties of neural networks (and other machine learning tools) is that they are able to learn properties of input data, either in a supervised or unsupervised manner. Such learning is the result of a training algorithm or a meta-level neural network that provides the training signal.

[0043] In general, the training algorithm consists of altering some properties of the neural network so that its output is as close as possible to a desired output. For example, in the case of object classification in images, the output of the neural network can be used to derive a class or category index that indicates the class or category to which the object in the input image belongs. Training usually happens by minimizing or reducing the output error, also referred to as loss. Examples of losses are mean squared error, cross-entropy, etc. In recent deep learning techniques, training is an iterative process, in which, in each iteration, the algorithm modifies the weights of the neural network to achieve a gradual improvement in the network's output, that is, to gradually reduce the loss.

[0044] In this description, the terms “model” and “neural network” are used interchangeably, and neural network weights are also sometimes referred to as learnable parameters or simply as parameters. Petition 870250090159, dated 03 / 10 / 2025, page 18 / 226 9 / 59

[0045] Training a neural network is an optimization process. The goal of the optimization or training process is to make the model learn the properties of the data distribution from a limited training dataset. In other words, the goal is to learn to use a limited training dataset to learn to generalize to data never seen before, that is, data that was not used to train the model. This is generally referred to as generalization. In practice, data can be divided into at least two sets, the training set and the validation set. The training set is used to train the network, that is, to modify its learnable parameters to minimize loss. The validation set is used to check the network's performance on data that was not used to minimize loss, as an indication of the model's final performance.Specifically, errors in the training set and validation set are monitored during the training process to understand the following: If the network is learning, the training set error should decrease; otherwise, the model is in underfitting mode. - If the network is learning to generalize - in that case, the validation set error also needs to decrease and not be much larger than the training set error. If the training set error is low, but the validation set error is much larger than the training set error, or it doesn't decrease, or even increases, the model is in the overfitting regime. This means that the model has memorized the properties of the training set and performs well only on that set, but performs poorly on a set not used to adjust its parameters.

[0046] Although the above background information on neural networks may be valid at the time this document was written, the field of neural networks and machine learning in general is still developing. Petition 870250090159, dated 03 / 10 / 2025, page 19 / 226 10 / 59 quickly. Thus, it should be understood that at least some of the modalities described in this document are not limited to the definition of a neural network, or a machine learning model, or a training algorithm that was provided in the background information above.

[0047] Recently, neural networks have been used to compress and decompress data such as images, that is, in an image codec. The most widely used architecture to implement a component of an image codec is the autoencoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. The neural encoder receives an image as input and produces a code that requires fewer bits than the input image. This code can be obtained by applying a binarization or quantization process to the encoder output. The neural decoder takes this code and reconstructs the image that was fed into the neural encoder.

[0048] Such a neural encoder and neural decoder can be trained to minimize a combination of bit rate and distortion, wherein the distortion may be based on one or more of the following metrics: Mean Squared Error (MSE), Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), or similar. These distortion metrics should be correlated to the quality of human visual perception, such that minimizing or maximizing one or more of these distortion metrics results in improved visual quality of the decoded image as perceived by humans.

[0049] A video codec comprises an encoder that transforms the input video into a compressed representation suitable for storage / transmission and a decoder that can decompress the compressed video representation back into a viewable form. An encoder may discard some information in the original video sequence in order to represent the video in a more compact form (i.e., at a lower bitrate). Petition 870250090159, dated 03 / 10 / 2025, page 20 / 226 11 / 59

[0050] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Expert Group (VCEG) of the Telecommunications Standardization Sector of the International Telecommunication Union (ITU-T) and the Moving Picture (MPEG) Expert Group of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization organizations and is referred to as ITU-T Recommendation H.264 and International Standard ISO / IEC 14496-10, also known as MPEG-4 Advanced Video Coding (AVC) Part 10. Extensions of H.264 / AVC include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0051] High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) was developed by the Joint Collaborative Video Coding Team (JCT-VC) of VCEG and MPEG. The standard was published by both parent standardization organizations and is referred to as ITU-T Recommendation H.265 and International Standard ISO / IEC 23008-2, also known as MPEG-H High Efficiency Video Coding Part 2 (HEVC). Later versions of H.265 / HEVC included scalable, multiview, fidelity range, three-dimensional, and screen content coding extensions, which can be abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0052] Versatile Video Coding (H.266 also known as VVC), defined in ITU-T Recommendation H.266 and equivalently in ISO / IEC 230903, (also referred to as MPEG-I Part 3) is a video compression standard developed as the successor to HEVC. A reference software for VVC is the VVC Test Model (VTM).

[0053] An AV1 bitstream format specification and decoding process was developed by the Alliance for Open Media (ACM). The AV1 specification was published in 2018. The ACM is reportedly working on the AV2 specification. Petition 870250090159, dated 03 / 10 / 2025, page 21 / 226 12 / 59

[0054] An elementary unit for input to a video encoder and output to a video decoder, respectively, is in most cases an image. An image given as input to an encoder may also be referred to as a source image, and an image decoded by a decoder may be referred to as a decoded image or a reconstructed image.

[0055] The source and decoded images are each comprised of one or more sample arrangements, as one of the following sets of sample arrangements: - Only luma (Y) (monochrome), - Luma and two chromas (YCbCr or YCgCo), Green, Blue, and Red (GBR, also known as RGB), - Arrangements that represent other unspecified monochromatic color samplings or tristimuli (e.g., YZX, also known as XYZ).

[0056] A component can be defined as a single array or sample of one of the three sample arrays (luma and two chroma) that make up an image, or the array or a single sample of the array that makes up an image in monochrome format.

[0057] Hybrid video codecs, for example, ITU-T, H.263, and H.264, can encode video information in two phases. First, the pixel values ​​in a given image area (or “block”) are predicted, for example, by means of motion compensation (finding and indicating an area in one of the previously encoded video frames that closely corresponds to the block being encoded) or by spatial means (using the pixel values ​​around the block to be encoded in a specified manner). Second, the prediction error, that is, the difference between the predicted pixel block and the original pixel block, is encoded. This can be done by transforming the difference into pixel values ​​using a specified transform (e.g., Discrete Cosine Transform). Petition 870250090159, dated 03 / 10 / 2025, page 22 / 226 13 / 59 (DCT) or a variant thereof), quantizing the coefficients and entropy-encoding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting encoded video representation (file size or transmission bitrate).

[0058] Inter-prediction, which may also be called temporal prediction, motion compensation, or motion-compensated prediction, exploits temporal redundancy. In inter-prediction, the prediction sources are previously decoded images.

[0059] Intra prediction utilizes the fact that adjacent pixels within the same image are likely to be correlated. Intra prediction can be performed in the transform or spatial domain, i.e., sample values ​​or transform coefficients can be predicted. Intra prediction can be exploited in intra coding, where no inter prediction is applied.

[0060] One result of the encoding procedure is a set of encoding parameters, such as motion vectors and quantized transform coefficients. Many parameters can be entropy-encoded more efficiently if they are predicted first from spatially or temporally neighboring parameters. For example, a motion vector can be predicted from spatially adjacent motion vectors, and only the difference relative to the motion vector predictor can be encoded. Encoding parameter prediction and intra-prediction can be collectively referred to as image prediction.

[0061] The decoder reconstructs the output video by applying a prediction medium similar to the encoder to form a predicted representation of the pixel blocks (using the spatial or motion information created by the encoder and stored in the compressed representation) and prediction error decoding (the inverse operation of prediction error encoding, which recovers the quantized prediction error signal in the spatial pixel domain). After applying the prediction medium and Petition 870250090159, dated 03 / 10 / 2025, page 23 / 226 14 / 59 Prediction Error Decoding: The decoder sums the prediction and prediction error signals (pixel values) to form the output video frame. The decoder (and encoder) may also apply additional filtering methods to improve the quality of the output video before displaying it and / or storing it as a prediction reference for subsequent frames in the video sequence.

[0062] In video codecs, motion information can be indicated with motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement of the image block in the image to be encoded (on the encoder side) or decoded (on the decoder side) and the prediction source block in one of the previously encoded or decoded images. In order to efficiently represent motion vectors, these can be differentially encoded with respect to specific block-predicted motion vectors. In video codecs, predicted motion vectors can be created in a predefined way, for example, by calculating the median of the encoded or decoded motion vectors of adjacent blocks.Another way to create motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or colocalized blocks in temporal reference images and flag the chosen candidate as the motion vector predictor. In addition to predicting motion vector values, the reference index of previously encoded / decoded images can be predicted. The reference index can be predicted from adjacent blocks and / or colocalized blocks in temporal reference images. Furthermore, high-efficiency video codecs may employ an additional motion information encoding / decoding mechanism, often called blending / blend mode, where all motion field information, including motion vector and corresponding reference image index for each available reference image list, is predicted and used without any further modifications. Petition 870250090159, dated 03 / 10 / 2025, page 24 / 226 15 / 59 modification / correction. Similarly, motion field information prediction can be performed using motion field information from adjacent blocks and / or colocalized blocks in temporal reference images, and the motion field information used is flagged from a list of motion field candidates populated with available adjacent / colocated block motion field information.

[0063] In video codecs, the prediction residue after motion compensation can first be transformed with a transform kernel (such as DCT) and then encoded. The reason for this is that there is often still some correlation between the residues, and the transform can, in many cases, help to reduce this correlation and provide more efficient encoding.

[0064] Video encoders can use Lagrangian cost functions to find ideal encoding modes, for example, the desired encoding mode for a block, block partition, and associated motion vectors. This type of cost function uses a weighting factor λ to combine the image distortion (exact or estimated) due to lossy encoding methods and the amount (exact or estimated) of information required to represent pixel values ​​in an image area: C = D + √R where C is the Lagrangian cost to be minimized, D is the image distortion (e.g., Mean Squared Error) with the considered mode and motion vectors, and R is the number of bits needed to represent the data required to reconstruct the image block in the decoder (including the amount of data to represent the candidate motion vectors). The rate R can be the actual bit rate or bit count resulting from the encoding. Alternatively, the rate R can be an estimated bit rate or bit count. One possible way to estimate the rate R is to omit the final entropy encoding step and use, for example, an entropy encoding. Petition 870250090159, dated 03 / 10 / 2025, page 25 / 226 16 / 59 simpler or an entropy encoder where some of the context states were not updated according to previous encoding mode selections.

[0065] Conventionally used distortion metrics may include, but are not limited to, peak signal-to-noise ratio (PSNR), mean squared error (MSE), sum of absolute differences (SAD), sum of absolute transformed differences (SATD), and structural similarity (SSIM), typically measured between the reconstructed video / image signal (which is or would be identical to the decoded video / image signal) and the “original” video / image signal provided as input for encoding.

[0066] A partition can be defined as a division of a set into subsets such that each element of the set is in exactly one of the subsets.

[0067] A bitstream can be defined as a sequence of bits that, in some formats or encoding standards, can be in the form of a Network Abstraction Layer (NAL) unit stream or a byte stream, which forms the representation of encoded images and associated data that form one or more encoded video sequences.

[0068] A bitstream format can comprise a sequence of syntax structures.

[0069] A syntax element can be defined as a data element represented in the bitstream. A syntax structure can be defined as zero or more syntax elements present together in the bitstream in a specified order.

[0070] A NAL unit can be defined as a syntax structure containing an indication of the data type to follow and bytes containing that data in the form of an RBSP interleaved as needed with initial code emulation prevention bytes. A raw byte sequence payload (RBSP) can be defined as a syntax structure containing an integer number of bytes that is encapsulated in a NAL unit. An RBSP Petition 870250090159, dated 03 / 10 / 2025, p. 26 / 226 17 / 59 is either empty or in the form of a sequence of data bits containing syntax elements followed by an RBSP stop bit and followed by zero or more subsequent bits equal to 0.

[0071] Some encoding formats specify parameter sets that may carry parameter values ​​needed for decoding or reconstructing decoded images. A parameter can be defined as a syntax element of a parameter set. A parameter set can be defined as a syntax structure that contains parameters and that can be referenced or activated by another syntax structure, for example, using an identifier.

[0072] A coding standard or specification may specify several types of parameter sets. It is necessary to understand that modalities may be applied, but without limitation, to the described types of parameter sets and modalities could, in the same way, be applied to any type of parameter set.

[0073] A parameter set can be activated when referenced, for example, through its identifier. An adaptation parameter set (APS) can be defined as a syntax structure that applies to zero or more slices. There can be different types of adaptation parameter sets. An adaptation parameter set can, for example, contain filtering parameters for a particular type of filter. In VVC, three types of APSs are specified carrying parameters for one of: adaptive loop filter (ALF), luma mapping with chroma scaling (LMCS), and scaling lists. A scaling list can be defined as a list that associates each frequency index with a scaling factor for the scaling process, which multiplies transform coefficient levels by a scaling factor, resulting in transform coefficients.In VVC, an APS is referenced by its type (e.g., ALF, LMCS, or scheduling list) and an identifier. In other words, different types of APSs have their own value ranges. Petition 870250090159, dated 03 / 10 / 2025, page 27 / 226 18 / 59 identifier.

[0074] An Adaptive Parameter Set (APS) may comprise parameters for decoding processes of different types, such as adaptive loop filtering or luma mapping with chroma scaling.

[0075] Video coding specifications may enable the use of Supplemental Enhancement Information (SEI) messages or similar. Some video coding specifications include Network Abstraction Layer (NAL) SEI units, and some video coding specifications contain both prefix NAL SEI units and suffix NAL SEI units, where the former type may start an image unit or similar and the latter type may terminate an image unit or similar. A NAL SEI unit contains one or more SEI messages, which are not required for decoding output images but may assist in related processes such as image output timing, post-processing of decoded images, rendering, error detection, error hiding, and resource reservation. Several SEI messages are specified in H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.267 standards.Standard 274A / SEI, and the user data SEI messages enable organizations and enterprises to specify SEI messages for their own use. Standards may contain the syntax and semantics for the specified SEI messages, but a process for handling the messages at the recipient may not be defined. Consequently, encoders may be required to follow the standard that specifies an SEI message when they create SEI message(s), and decoders may not be required to process SEI messages for output order conformity. One of the reasons for including the syntax and semantics of SEI messages in standards is to allow different system specifications to interpret the supplementary information identically and thus interoperate. It is intended that system specifications may require the use of particular SEI messages in both encoding and decoding, and additionally the process for. Petition 870250090159, dated 03 / 10 / 2025, page 28 / 226 19 / 59 Handling private SEI messages at the recipient can be specified. SEI messages are generally not extended in future versions or amendments to the standard.

[0076] The expression along the bitstream (e.g., indicating along the bitstream) or along an encoded unit of a bitstream (e.g., indicating along an encoded block) may be used in the claims and embodiments described to refer to transmission, signaling, or storage in such a way that “out-of-band” data is associated with, but not included in, the bitstream or encoded unit, respectively. The expression decoding along the bitstream or along an encoded unit of a bitstream or similar may refer to the decoding of said out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) that is associated with the bitstream or encoded unit, respectively.For example, the expression "along the bitstream" can be used when the bitstream is contained in a container file, such as a file that conforms to the ISO Base Media File Format, and certain file metadata is stored in a way that associates the metadata with the bitstream, such as boxes in the sample entry for a track containing the bitstream, a sample group for the track containing the bitstream, or a timed metadata track associated with the track containing the bitstream.

[0077] Image and video codecs can use a set of filters to enhance the visual quality of predicted visual content and can be applied in a loop or out of a loop, or both. In the case of loop filters, the filter applied to a block in the currently encoded frame will affect the encoding of another block in the same frame and / or in another frame that is predicted from the current frame. A loop filter can affect the bitrate and / or visual quality. In fact, an enhanced block will cause a smaller residue (difference between the original block and the predicted and filtered block), thus requiring fewer bits to be encoded. A filter Petition 870250090159, dated 03 / 10 / 2025, page 29 / 226 20 / 59 out of loop will be applied to a frame after it has been reconstructed; the filtered visual content will not be used as a source for prediction and thus can only impact the visual quality of the frames emitted by the decoder.

[0078] Recently, neural networks (NNs) have been used in the context of image and video compression, mainly following two approaches.

[0079] In one approach, NNs are used to replace one or more of the components of a traditional codec, such as VVC / H.266. Here, the term "traditional" refers to those codecs whose components and their parameters cannot be learned from the data. Examples of such components are: - An additional loop filter, for example, using NN as an additional loop filter in addition to traditional loop filters. - A single loop filter, for example, where the NN replaces all traditional loop filters. - Intraframe prediction. Inter-frame prediction. - Transform and / or inverse transform. - Probability model for the arithmetic codec. Etc.

[0080] Figure 1 illustrates examples of NNs functioning as components of a traditional codec pipeline, according to one embodiment. In particular, Figure 1 illustrates an encoder, which also includes a decoding loop. Figure 1 is shown as including the components described below:

[0081] A luma 101 intra-prediction block or circuit. This block or circuit performs intra-domain prediction in luma, for example, using already reconstructed data from the same frame. The luma 101 intra-prediction block or circuit operation can be performed by a deep neural network as a convolutional autoencoder.

[0082] A 102 intra-domain prediction block or circuit. This block or circuit performs intra-domain prediction in the chroma domain, for example, using already existing data. Petition 870250090159, dated 03 / 10 / 2025, page 30 / 226 21 / 59 reconstructed from the same frame. The 102 chroma intra-prediction block or circuit can perform cross-component prediction, for example, predicting chroma from luma. The operation of the 102 chroma intra-prediction block or circuit can be performed by a deep neural network as a convolutional autoencoder.

[0083] An intra-prediction block or circuit 103 and an inter-prediction block or circuit 104. These blocks or circuits perform intra-prediction and inter-prediction, respectively. The intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 can perform prediction on all components, for example, luma and chroma. The operations of the intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 can be performed by two or more deep neural networks as convolutional autoencoders.

[0084] A probability estimation block or circuit 105 for entropy encoding. This block or circuit performs probability prediction for the next symbol to be encoded or decoded, which is then provided to the entropy encoding module 112, like the arithmetic encoding module, to encode or decode the next symbol. The operation of the probability estimation block or circuit 105 can be performed by a neural network.

[0085] A transform and quantization (T / Q) block or circuit 106. These are actually two blocks or circuits. The transform and quantization block or circuit 106 can perform a transform of input data to a different domain, for example, the FFT transform would transform the data to the frequency domain. The transform and quantization block or circuit 106 can quantize its input values ​​to a smaller set of possible values. In the decoding loop, there may be an inverse quantization block or circuit and an inverse transform block or circuit 113. One or both, the transform block or circuit and the quantization block or circuit, may be replaced by one or two or more neural networks. One or both, the inverse transform block or circuit and the inverse quantization block or circuit 113, Petition 870250090159, dated 03 / 10 / 2025, page 31 / 226 22 / 59 can be replaced by one, two, or more neural networks.

[0086] A 107-loop filter block or circuit. 107-loop filter block or circuit operations are performed in a decoding loop, and it performs filtering on the output of the inverse transform block or circuit, or in any case on the reconstructed data, to improve the reconstructed data with respect to one or more predetermined quality metrics. This filter can affect the quality of the decoded data and the bit rate of the bitstream emitted by the encoder. The 107-loop filter block or circuit operation can be performed by a neural network, such as a convolutional autoencoder. In examples, the loop filter operation can be performed by multiple stages or filters, where one or more stages can be performed by neural networks.

[0087] A post-processing filter block or circuit 108. The post-processing filter block or circuit 108 can be implemented only on the decoder side, as it may not affect the encoding process. The post-processing filter block or circuit 108 filters the reconstructed data emitted by the loop filter block or circuit 107, to refine the reconstructed data. The post-processing filter block or circuit 108 can be replaced by a neural network, such as a convolutional autoencoder.

[0088] A resolution adaptation block or circuit 109: this block or circuit can subsample the input video frames before encoding. Then, in the decoding loop, the reconstructed data can be oversampled, by the oversampling block or circuit 110, to the original resolution. The operation of the resolution adaptation block or circuit 109 can be performed by a neural network as a convolutional autoencoder.

[0089] A 111 encoder control block or circuit. This block or circuit performs encoder parameter optimization, such as which transform to use, which quantization parameters (QP) to use, which intra prediction mode (of N intra prediction modes) to use, and the like. The operation of the block or circuit Petition 870250090159, dated 03 / 10 / 2025, page 32 / 226 23 / 59 of encoder control 111 can be performed by a neural network, such as a convolutional classifier network, or as a convolutional regression network.

[0090] A ME / MC 114 block or circuit performs motion estimation and / or motion compensation, which are two key operations to be performed when performing inter-frame prediction. ME / MC stands for motion estimation / motion compensation.

[0091] In another approach, commonly referred to as “end-to-end learned compression”, NNs are used as the main components of image / video codecs. In this second approach, there are two main options:

[0092] Option 1: Reuse the video encoding pipeline, but replace most or all of the components with NNs. Referring to Figure 2, it illustrates an example of a modified video encoding pipeline based on a neural network, according to one modality. An example of a neural network may include, but is not limited to, a compressed representation of a neural network. Figure 2 is shown as including the following components: - A neural transform block or circuit 202: this block or circuit transforms the output of an addition / subtraction operation 203 into a new representation of that data, which may have lower entropy and thus be more compactable. - A quantization block or circuit 204: this block or circuit quantizes input data 201 into a smaller set of possible values. - Inverse transform and inverse quantization blocks or circuits 206. These blocks or circuits perform the inverse or approximately inverse operation of the transform and quantization, respectively. - A 208 encoder parameter control block or circuit. This block or circuit can control and optimize some or all of the encoding process parameters, such as parameters from one or more of the blocks. Petition 870250090159, dated 03 / 10 / 2025, page 33 / 226 24 / 59 or encoding circuits. - An entropy coding block or circuit 210. This block or circuit can perform lossless coding, for example, based on entropy. A popular entropy coding technique is arithmetic coding. - An intracodec neural block or circuit 212. This block or circuit can be an image compression and decompression block or circuit, which can be used to encode and decode an intra frame. An encoder 214 can be an encoder block or circuit, such as the neural encoder part of a self-encoding neural network. A decoder 216 can be a decoder block or circuit, such as the neural decoder part of a self-encoding neural network. An intra-encoding block or circuit 218 can be a block or circuit that performs some intermediate steps between encoder and decoder, such as quantization, entropy encoding, entropy decoding, and / or inverse quantization. - A 220 deep loop filter block or circuit. This block or circuit performs filtering of reconstructed data to improve its quality. - A decoding image buffer block or circuit 222. This block or circuit is a memory buffer that holds the decoded frame, for example, reconstructed frames 224 and enhanced reference frames 226 to be used for inter-prediction. - An inter-frame prediction block or circuit 228. This block or circuit performs inter-frame prediction, for example, it predicts from frames, for example, frames 232, that are temporarily close. A ME / MC 230 performs motion estimation and / or motion compensation, which are two key operations to be performed when performing inter-frame prediction. ME / MC means motion estimation / motion compensation.

[0093] Option 2: redesign the entire pipeline, as follows. - The NN encoder is configured to perform a nonlinear transform; Petition 870250090159, dated 03 / 10 / 2025, page 34 / 226 25 / 59 - Lossless quantization and encoding of the NN encoder output; - Lossless decoding and dequantization; The NN encoder is configured to perform a nonlinear inverse transform.

[0094] An example of option 2 is described in detail in Figure 3, which shows an encoding NN and a decoding NN as parts of a neural autoencoder architecture, according to an example. In Figure 3, the Analysis Network 301 is an Encoding NN, and the Synthesis Network 302 is the Decoding NN, which together can be referred to as spatial correlation tools 303, or as a neural autoencoder.

[0095] As shown in Figure 3, the input data 304 is analyzed by the Encoding NN (Analysis Network 301), which outputs a new representation of this input data. The new representation may be more compact. This new representation may then be quantized, by a quantizer 305, to a discrete number of values. The quantized data is then losslessly encoded, for example, by an arithmetic encoder 306, thus obtaining a bit stream 307. The example shown in Figure 3 includes an arithmetic decoder 308 and an arithmetic encoder 306. The arithmetic encoder 306, or the arithmetic decoder 308, or the combination of the arithmetic encoder 306 and the arithmetic decoder 308 may be referred to as an arithmetic codec in some embodiments. On the decoding side, the bitstream is first decoded without loss, for example, using the 308 arithmetic codec decoder.The lossless decoded data is dequantized and then fed into the Decoding NN, Synthesis Network 302. The output is the reconstructed or decoded data 309.

[0096] In the case of lossy compaction, the lossy steps may include NN Encoding and / or quantization.

[0097] To train this system, a training objective function (also called a “training loss”) can be used, which may comprise one or more terms, or loss terms, or simply losses. In a Petition 870250090159, dated 03 / 10 / 2025, page 35 / 226 26 / 59 For example, training loss comprises a reconstruction loss term and a rate loss term. Reconstruction loss prompts the system to decode data that is similar to the input data, according to some similarity metric. Examples of reconstruction losses are: Mean squared error (MSE); - Multiscale structural similarity (MS-SSIM); - Losses resulting from the use of a pre-trained neural network. For example, error(f1, f2), where f1 and f2 are the features extracted by a pre-trained neural network for the input data and the decoded data, respectively, and error() is a distance or error function, such as L1 norm or L2 norm; - Losses resulting from the use of a neural network that is trained simultaneously with the codec learned end-to-end. For example, adversarial loss can be used, which is the loss provided by a discriminating neural network that is trained adversarially with respect to the codec, following the configurations proposed in the context of Generative Adversarial Networks (GANs) and their variants.

[0098] Rate loss prompts the system to compress the output of the encoding stage, such as the output of the arithmetic encoder. “Compressing” in this context means reducing the number of bits emitted by the encoding stage.

[0099] When an entropy-based lossless encoder is used, such as an arithmetic encoder, the rate loss typically encourages the NN encoder output to have low entropy. Examples of rate losses are as follows: - A differentiable estimate of entropy; - A sparsification loss, that is, a loss that causes the output of the encoding NN or the output of the quantization to have many zeros. Examples are L0 norm, L1 norm, L1 norm divided by L2 norm; - A cross-entropy loss applied to the output of a probability model, where the probability model can be a non-digital structure (NN) used to estimate the probability of the next symbol being encoded by a... Petition 870250090159, dated 03 / 10 / 2025, page 36 / 226 27 / 59 arithmetic encoder.

[0100] One or more reconstruction losses can be used, and one or more rate losses can be used, as a weighted sum. The different loss terms can be weighted using different weights, and these weights determine how the final system performs in terms of rate distortion loss. For example, if more weight is given to reconstruction losses relative to rate losses, the system can learn to compact less but reconstruct more accurately (as measured by a metric that correlates with reconstruction losses). These weights can be considered as hyperparameters of the training session and can be set manually by the person designing the training session, or automatically, for example, grid search or using additional neural networks.

[0101] As shown in Figure 4, an end-to-end learned neural network-based video coding system may contain an encoder 401, a quantizer 402, a probability model 403, an entropy codec 420 (e.g., arithmetic encoder 405 / arithmetic decoder 406), a dequantizer 407, and a decoder 408. Encoder 401 and decoder 408 may be two neural networks, or comprise primarily neural network components. Probability model 403 may also comprise primarily neural network components. Quantizer 402, dequantizer 407, and entropy codec 420 may not be based on neural network components, but they may also potentially comprise neural network components.

[0102] On the encoder side, the 401 encoder component receives an x409 video as input and converts the video from its original signal space into a latent representation that can comprise a more compactable representation of the input. In the case of an input image, the latent representation can be a three-dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represents Petition 870250090159, dated 03 / 10 / 2025, page 37 / 226 28 / 59 the “channels” that contain information at that specific location. If the input image is a 128x128x3 RGB image (with a horizontal side of 128 pixels, a vertical side of 128 pixels, and 3 channels for the Red, Green, and Blue color components), and if the encoder subsamples the input tensor by 2 and expands the channel dimension to 32 channels, then the latent representation is a tensor with dimensions (or “format”) of 64x64x32 (that is, with a horizontal side of 64 elements, a vertical side of 64 elements, and 32 channels). It should be noted that the order of the different dimensions may differ depending on the convention used; in some cases, for the input image, the channel dimension may be the first dimension, so for the example above, the format of the input tensor may be represented as 3x128x128, instead of 128x128x3.In the case of an input video (instead of just an input image), another dimension in the input tensor can be used to represent temporal information.

[0103] The quantizer component 402 quantizes the latent representation into discrete values ​​given a predefined set of quantization levels. Probability model 403 and arithmetic codec component 420 work together to perform lossless compression for the quantized latent representation and generate bitstreams to be sent to the decoder side. Given a symbol to be encoded into the bitstream, probability model 403 estimates the probability distribution of all possible values ​​for that symbol based on a context that is constructed from information available in the current encoding / decoding state, such as data that has already been encoded / decoded. Then, arithmetic encoder 405 encodes the input symbols into bitstreams using the estimated probability distributions.

[0104] On the decoder side, opposite operations are performed. The arithmetic decoder 406 and the probability model 403 first decode symbols from the bit stream to recover the quantized latent representation. Then the dequantizer 407 reconstructs the latent representation in Petition 870250090159, dated 03 / 10 / 2025, page 38 / 226 29 / 59 continuous values ​​are passed to the 408 decoder to recover the input video / image. Note that the 403 probability model in this system is shared between the encoding and decoding systems. In practice, this means that one copy of the 403 probability model is used on the encoder side, and another exact copy is used on the decoder side.

[0105] In this system, encoder 401, probability model 403, and decoder 408 can be based on deep neural networks. The system can be trained in an end-to-end manner minimizing the following rate-loss distortion function: L=D + ÀR,

[0106] where D is the distortion loss term, R is the rate loss term, and λ is the weight that controls the balance between the two losses. The distortion loss term can be the mean squared error (MSE), structure similarity (SSIM), or other metrics that assess the quality of the reconstructed video. Multiple distortion losses can be used and integrated into D, as a weighted sum of MSE and SSIM. The rate loss term is typically the estimated entropy of the quantized latent representation, which indicates the number of bits needed to represent the encoded symbols, for example, bits per pixel (bpp).

[0107] For lossless video / image compression, the system may contain only the 403 probability model and 405, 406 arithmetic encoder / decoder. The system loss function contains only rate loss, since distortion loss is always zero (i.e., no information loss).

[0108] Reducing distortion in image and video compression often aims to increase human perceptual quality, since humans are considered to be the end users, i.e., consuming / watching the decoded image. Recently, with the advent of machine learning, especially deep learning, there is a growing number of machines (i.e., autonomous agents) that analyze data independently of humans and can even make decisions with Petition 870250090159, dated 03 / 10 / 2025, page 39 / 226 30 / 59 based on analysis results without human intervention. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, pedestrian tracking, etc. Illustrative use cases and applications include autonomous cars, video security cameras and public safety, smart sensor networks, smart TV and smart advertising, people re-identification, intelligent traffic monitoring, drones, etc. When decoded data is consumed by machines, a different quality metric should be used instead of human perceptual quality. Furthermore, algorithms dedicated to compressing and decompressing data for machine consumption are likely to differ from those for compressing and decompressing data for human consumption.The set of tools and concepts for compressing and decompressing data for machine consumption is referred to here as Video Machine Coding (VCM).

[0109] VCM refers to encoding continuous video streams to enable consumption by machines. Machine is used to indicate any device except a human being. Examples of machines might be a mobile phone, an autonomous vehicle, a robot, and such intelligent devices that may have a degree of autonomy or run an intelligent algorithm to process the decoded continuous stream in addition to reconstructing the original input continuous stream.

[0110] A machine can perform one or multiple tasks in the decoded continuous stream. Examples of tasks may include the following: - Classification: classify an image or video into one or more predefined categories. The output of a classification task can be a set of detected categories, also known as classes or labels. The output can also include the probability and confidence of each predefined category. - Object detection: detecting one or more objects in a given image or video. The output of an object detection task can be the boxes Petition 870250090159, dated 03 / 10 / 2025, page 40 / 226 31 / 59 boundary values ​​and the detected classes of the detected objects. The output may also include the probability and confidence of each detected object. - Instance segmentation: identifying one or more objects in an image or video at the pixel level. The output of an instance segmentation task can be binary mask images or other representations of binary mask images, for example, closed outlines, of the detected objects. The output can also include the probability and confidence of each object for each pixel. Semantic segmentation: assigning pixels in an image or video to one or more predefined semantic categories. The output of a semantic segmentation task can be binary mask images or other representations of binary mask images, for example, closed outlines, of the assigned categories. The output can also include the probability and confidence of each semantic category for each pixel. - Object tracking: tracking one or more objects in a video sequence. The output of an object tracking task may include frame index, object ID, object boundary boxes, probability, and confidence for each tracked object. - Captioning: Generate one or more short text descriptions for an input image or video. The output of the captioning task can be one or more short text sequences. - Human pose estimation: estimating the position of key points, e.g., wrists, elbows, knees, etc., of one or more human bodies in a video image. The output of a human pose estimate includes sets of locations for each key point of a human body detected in the input image or video. - Recognition of human action: recognizing the actions, for example, walking, talking, shaking hands, of one or more people in an image. Petition 870250090159, dated 03 / 10 / 2025, page 41 / 226 32 / 59 or video input. The output of human action recognition can be a set of predefined actions, probability, and confidence level for each identified action. - Anomaly detection: detect abnormal objects or events in an input image or video. The output of an anomaly detection may include the locations of abnormal objects or segments detected from frames where abnormal events are detected in the video input.

[0111] The receiver-side device likely has multiple “machines” or task neural networks (Task NNs). These multiple machines can be used in a certain combination that is, for example, determined by an orchestrating subsystem. The multiple machines can be used, for example, in succession, based on the output of the previously used machine and / or in parallel. For example, a video that has been compressed and then decompressed can be analyzed by one machine (NN) to detect pedestrians, by another machine (another NN) to detect cars, and by yet another machine (another NN) to estimate the depth of all pixels in the frames.

[0112] In this description, “task machine” and “machine” and “task neural network” are referred to interchangeably, and by such reference is meant any process or algorithm (learned or not from data) that analyzes or processes data for a certain task. In the remainder of the description, other assumptions made regarding the machines considered in this disclosure may be specified in more detail. Furthermore, the term “receiver side” or “decoder side” is used to refer to the physical or abstract entity or device that contains one or more machines and executes that one or more machines on an encoded and eventually decoded video representation that is encoded by another physical or abstract entity or device, the “encoder side device”.

[0113] Encoded video data can be stored on a memory device, for example, as a file. The stored file Petition 870250090159, dated 03 / 10 / 2025, page 42 / 226 33 / 59 can subsequently be provided to another device. Alternatively, the encoded video data can be streamed from one device to another.

[0114] Figure 5 is a general illustration of the Video Encoding for Machines pipeline. A VCM 502 encoder encodes the video input into a bitstream 504. A bitrate 506 can be computed 508 from the bitstream 504 to evaluate the bitstream size. A VCM 510 decoder decodes the bitstream emitted by the VCM 502 encoder. In Figure 5, the output of the VCM 510 decoder is referred to as “Machine-Decoded Data” 512. This data can be considered as the decoded or reconstructed video. However, in some implementations of this pipeline, this data may not have the same or similar characteristics to the original video that was emitted to the VCM 502 encoder. For example, this data may not be easily understandable by a human when rendering the data on a screen. The VCM decoder output is then fed into one or more 514 task neural networks.In the figure, to illustrate that there can be any number of task NNs 514, there are three exemplary task NNs and one unspecified one (Task X NN). The goal of VCM is to obtain a low data rate representation of the input video, while ensuring that the task NNs still perform well in terms of the evaluation metric 516 associated with each task.

[0115] One of the specific approaches to performing video encoding for machines is an end-to-end learned approach. In this approach, the VCM encoder and the VCM decoder consist mainly of neural networks. Figure 6 illustrates an example of a pipeline for the end-to-end learned approach. The video is fed into a neural network encoder 601. The output of the neural network encoder 601 is fed into a lossless encoder 602, such as an arithmetic encoder, which outputs a bitstream 604. The output of the neural network encoder 601 can also be fed into a probability model 603 that provides for the Petition 870250090159, dated 03 / 10 / 2025, page 43 / 226 34 / 59 lossless encoder 602 an estimate of the probability of the next symbol being encoded by lossless encoder 602. The probability model 603 can be learned through machine learning techniques, for example, it could be a neural network. On the decoder side, the bitstream 604 is fed into a lossless decoder 605, such as an arithmetic decoder, whose output is fed into a neural network decoder 606. The output of the lossless decoder 605 can be fed into a probability model 603, which provides the lossless decoder 605 with an estimate of the probability of the next symbol being decoded by lossless encoder 605. The output of the neural network decoder 606 is the decoded data for machines 607, which can be fed into one or more task NNs 608.

[0116] Figure 7 illustrates an example of how the end-to-end learned system can be trained for the purpose of video encoding for machines. For simplification, only a task NN 707 is illustrated. A rate loss 705 can be computed from the output of the probability model 703. The rate loss 705 provides an approximation of the bit rate required to encode the input video data. A rate loss 710 can be computed 709 from the output 708 of the task NN 707.

[0117] Rate loss 705 and task loss 710 can then be used to train 711 the neural networks used in the system, such as the neural network encoder 701, the probability model 703, the neural network decoder 706. Training can be performed by first computing gradients of each loss with respect to the trainable neural network parameters that are contributing to or affecting the computation of the loss. The gradients are then used by an optimization method, such as Adam, to update the trainable parameters of the neural networks.

[0118] Machine tasks may be performed on the decoder side (instead of the encoder side) for multiple reasons, for example, because the encoder-side device does not have the capabilities (computational, Petition 870250090159, dated 03 / 10 / 2025, page 44 / 226 35 / 59 power, memory) to run the neural networks that perform these tasks, or because some aspects or the performance of the task neural networks may have changed or improved by the time the decoder-side device needs the task results (e.g., different or additional semantic classes, better neural network architecture). Additionally, there could be a need for customization, where different clients would run different neural networks to perform these machine learning tasks.

[0119] Dense Divided Attention Block (DSA) is an attention block that estimates one or more attention maps, and applies the one or more attention maps to one or more data tensors. - An attention map can be a vector, a matrix, or a tensor. For example, an attention map might have values ​​in the range [0, 1]. - Where one or more data tensors may comprise one or more input tensors for the attention block, and / or one or more feature maps extracted within the attention block, and / or one or more feature maps extracted outside the attention block. - Applying one or more attention maps to one or more data tensors might involve multiplying the one or more attention maps by the one or more data tensors, for example, using element-by-element multiplication. Other operations may also be considered.

[0120] Figure 8 illustrates an example of a DSA block, in which one type of NN layer is a ResBlock, which comprises NN layers.

[0121] The DSA block may comprise extracting features from its input based on at least one or more initial NN layers, then splitting the extracted features along the channel geometric axis to obtain two split features, summing the split features, performing a global averaging operation on the summed features, processing the output of the global averaging operation based on at least one or more NN layers, inserting Petition 870250090159, dated 03 / 10 / 2025, page 45 / 226 36 / 59 The result of this processing in a Softmax operation, divide the result of the Softmax operation on the geometric channel axis to obtain two attention tensors, multiply the two attention tensors with the two previously determined split features to obtain two attended split features, sum the two attended split features, concatenate the summed attended split features with features determined based on at least one or more initial NN layers, process the result of the concatenation through at least one or more NN layers, sum the output of this processing with the input of the DSA block, to obtain the output of the DSA block. Here, the global averaging operation can be a global pooling operation (an average pooling) that calculates the average value for fragments of a feature map. This can aggregate spatial information from a feature map to a single channel, to help explore the interchannel relationship of features.

[0122] In video encoding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (luma) channel, also denoted as Y, and two chrominance (chroma) channels, also denoted as U, V or as Cb, Cr. In these cases, the luminance channel, primarily representing the scene's illumination, is typically encoded at a certain resolution, while the chrominance channels, typically representing differences between certain color components, are frequently encoded at a second, lower resolution than the luminance signal. The intention of this type of differential representation is to decorrelate the color components and be able to compress the data more efficiently.

[0123] As mentioned above, neural networks (NNs) have been used in the context of image and video compression, replacing one or more of the components of a traditional codec, as shown in Figure 1, or using end-to-end learned compression, as shown in Figures 2 and 3. For intra-sample prediction of luma and chroma, Figure 1 shows separate prediction blocks for luma and chroma components, where Petition 870250090159, dated 03 / 10 / 2025, page 46 / 226 37 / 59 The intra-chroma prediction block can perform cross-component luma prediction. It is known that the luma component and the chroma components of at least neighboring samples typically have some correlation in their values. However, NN-based video coding approaches, including end-to-end learned compression-based solutions, have proven inadequate considering this correlation. Thus, the rate distortion performance of codecs has been suboptimal.

[0124] Now, methods for improving rate distortion performance are introduced.

[0125] An aspect-based encoding method is shown in Figure 9a, wherein the method comprises receiving (900) input data comprising a luma component and a chroma component; providing (902) first fundamental truth data to a first encoder, said fundamental truth data comprising at least a first part of the input data; obtaining (904) an encoded luma component from an output of the first encoder; providing (906) the encoded luma component to a first decoder; obtaining (908) a reconstructed luma component from an output of the first decoder;provide (910) second fundamental truth data comprising at least a second part of the input data and the reconstructed luma component for a second encoder, said second encoder comprising at least a first subencoder for the fundamental truth data and a second subencoder for the reconstructed luma component and a probability model; provide (912) an output from the first subencoder and an output from the second subencoder for the probability model; obtain (914) one or more first probabilities from an output of the probability model; and obtain (916), based on an output from the first subencoder and one or more probabilities, an encoded chroma component from an output of the second encoder.

[0126] A method of decoding according to an aspect is shown Petition 870250090159, dated 03 / 10 / 2025, page 47 / 226 38 / 59 in Figure 9b, wherein the method comprises receiving (920) input data comprising an encoded chroma component and a reconstructed luma component in a decoder, said decoder comprising at least a first subdecoder, a second subdecoder and a probability model; providing (922) the reconstructed luma component to the second subdecoder; obtaining (924) one or more probabilities from an output of the probability model, based on at least one output of the second subdecoder and a previously decoded chroma component; obtaining (926) an entropy-decoded chroma component based on at least the encoded chroma component and one or more probabilities; providing (928) the entropy-decoded chroma component to the first subdecoder; and obtaining (930) a reconstructed chroma component from an output of the first subdecoder.

[0127] Thus, the methods reflect the encoding and decoding aspects in a codec that comprises two separate codecs to encode the luma component and the chroma component of the input data is used, a first codec comprising a first encoder and a first decoder and a second codec comprising a second encoder and a second decoder. These two codecs can be referred to as luma codec and chroma codec, respectively. The (first) luma codec encoder can receive the entirety of the fundamental truth data including both the luma component and the chroma component as input, and the (first) luma codec decoder reconstructs the luma component only. The bitrate that is emitted by the luma codec encoder represents the encoded luma component.So, the (second) chroma codec encoder can receive the entirety of the fundamental truth data, including both the luma component and the chroma component, as input. The (second) chroma codec encoder also receives the reconstructed luma component as input. The fundamental truth data and the reconstructed luma component are processed together. Petition 870250090159, dated 03 / 10 / 2025, page 48 / 226 39 / 59 respective subcoders and some of the subcoder outputs are fed into a chroma codec probability model, which estimates probabilities that are used to encode an output of the subcoder that processes the fundamental truth data. The bitstream that is emitted by the (second) chroma codec encoder represents the encoded chroma component.

[0128] The (second) chroma codec decoder receives the encoded chroma component and the reconstructed luma component. The reconstructed luma component is fed into a second subdecoder of the decoder. The decoder comprises a probability model, preferably the same or at least substantially the same as in the chroma codec encoder, to provide one or more probabilities based on at least one output from the second subdecoder and a previously decoded chroma component. A decoded chroma component is determined based on at least the encoded chroma component and one or more probabilities and fed into the first subdecoder. The first chroma codec subdecoder then reconstructs the chroma component(s) only.

[0129] With the additional input of the reconstructed luma component into the chroma codec probability model (both on the encoder and decoder sides), the chroma codec is conditioned to the reconstructed luma component, which improves the rate distortion performance of the chroma codec. Therefore, the rate distortion performance of the entire codec (the combination of the luma codec and the chroma codec) is improved.

[0130] According to one embodiment, the first and second encoders and decoders belong to an end-to-end learned intraframe codec, or to an end-to-end learned picture codec.

[0131] Thus, rate distortion performance improvements are achievable with learned video compression, especially in the context of end-to-end learned intraframe codec. However, it is noted that the Petition 870250090159, dated 03 / 10 / 2025, p. 49 / 226 The 40 / 59 method and related modalities are also applicable to end-to-end learned interframe codecs.

[0132] According to one embodiment, the learned interframe codec is trained in an end-to-end manner minimising D + AR, where D is a distortion loss term, R is a rate loss term, and λ is a weight that controls a balance between said losses.

[0133] The distortion loss term can be computed based on at least one distortion function, fundamental truth data, and a decoder output comprised in the learned interframe codec. The rate loss term can be computed based on at least an estimate of the bit rate of a bitstream (e.g., size of that bitstream in bits) output from the encoder comprised in the learned interframe codec.

[0134] This optimization process results in a so-called rate-distortion trade-off, where a balance is found between distortion D and rate loss R. Rate loss can indicate a bit rate of the encoded image, and distortion can indicate a pixel fidelity distortion as follows: Mean squared error (MSE); - Multiscale structural similarity (MS-SSIM); Multiple distortion losses, such as a weighted sum of MSE and MSSSIM; Other metrics that assess the quality of the reconstructed image.

[0135] According to one embodiment, training of the learned interframe codec is performed jointly with respect to D distortion loss and R rate loss.

[0136] According to another method, training of the learned interframe codec is carried out in two alternating phases, wherein in the first phase of the two alternating phases only the D distortion loss is used, and in the second phase of the two alternating phases only the R rate loss is used. Petition 870250090159, dated 03 / 10 / 2025, page 50 / 226 41 / 59

[0137] Several implementation options related to the following are described. Note that the terms “single frame”, “frame”, and “image” can be used interchangeably. These terms can refer to the input data for an end-to-end learned intraframe codec (e2e). For the sake of simplification, in at least some of the implementations, image is considered as the data type. However, the implementations and their underlying principles described herein can be extended to other data types, such as video, audio, etc. Additionally, in at least some of the implementations, YUV is considered as the input color format. However, the implementations can equally be extended to other color formats, such as RGB. In YUV format, “Y” represents brightness, or “luma” value; and “U” and “V” represent color, or “chroma” values.In one example, the input image could be a 4:4:4 YUV color format image, represented as a three-dimensional array (or tensor) with a size of 256x256x3, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for the Y, U, and V components, respectively. In another example, the input image could be a 4:2:0 YUV color format image, represented by the combination of a 256x256 matrix for the luma component and a two-dimensional array (or tensor) of 128x128x2 for the chroma component.

[0138] In general, an e2e learned intraframe codec resembles an e2e learned image compression system. Typically, it may consist of an image encoder, quantizer, probability model, entropy codec (e.g., arithmetic encoder / decoder), dequantizer, and image decoder. The image encoder, probability model, and image decoder may primarily comprise neural network components. The quantizer, entropy codec, and dequantizer are typically not based on neural network components, but they may nevertheless comprise neural network components.

[0139] An intraframe codec learned from e2e can be used as part of Petition 870250090159, dated 03 / 10 / 2025, page 51 / 226 42 / 59 a video codec, in which the intra-frame codec can encode each of one or more first frames of a video independently of any other frame, and in which another codec called the inter-frame codec can encode each of one or more second frames of a video based on at least one or more other frames, for example, based at least on data derived from the one or more first frames.

[0140] Luma codec

[0141] The input to the Luma codec may comprise at least fundamental truth data (e.g., a block or entire image), where at least part of the fundamental truth data must be encoded / compressed. In addition, the input to the Luma codec may comprise one or more extra data points, such as block / image resolution. Examples of input fundamental truth data are as follows: - A fundamental truth including both luma and chroma components. For example, the fundamental truth could be a YUV 4:4:4 color format image, represented as a 256x256x3 multidimensional array or tensor, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for Y, U, and V components, respectively. In another example, the fundamental truth could be a YUV 4:2:0 color format image, represented by the combination of a 256x256 matrix for the luma component and a 128x128x2 two-dimensional array (or tensor) for the chroma component. - A fundamental truth that includes luma component only. For example, the fundamental truth could be a 256x256x1 image, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 1 channel is for the Y component.

[0142] Figure 10 illustrates an exemplary implementation of a luma component codec according to a modality. The input data x is a fundamental truth including both luma component and component Petition 870250090159, dated 03 / 10 / 2025, p. 52 / 226 43 / 59 chroma, ehxwx 3 denotes the size of x with height h, width we, and number of channels 3. The luma component codec (i.e., the first codec) is used to encode the luma component of the input data; that is, the bitstream that is emitted by the luma codec encoder (i.e., the first encoder) represents the encoded luma component, and the output of the decoder (i.e., the first decoder) represents the reconstructed or decoded luma component.

[0143] The luma component codec 1000 may comprise a neural encoder 1002, a quantizer, a probability model 1004, and an entropy encoder 1006. In the figure, the quantizer is not illustrated for simplification purposes. The neural encoder 1002 may comprise a first convolutional layer (“Conv5x5, 48, 1”, where conv means convolution, 5x5 is the kernel size, 48 is the number of output channels, and 1 is the step value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. The neural encoder 1002 emits a latent tensor, which can be quantized.The latent tensor or a quantized latent tensor can be fed into a 1004 probability model, and the dimension of the latent tensor can be h / / 16 x w / / 16 x 128, where h / / 16 indicates the height, w / / 16 indicates the width, and 128 indicates the number of channels of the latent tensor. The probability model outputs an estimate of the probability of each element of the latent (quantized) tensor. The probability model can be learned from data using machine learning techniques; for example, the probability model can be a neural network. On the encoder side, the output of the 1004 probability model is used as one of the inputs to an entropy encoder 1006. The entropy encoder can be an arithmetic encoder. The entropy encoder receives at least the... Petition 870250090159, dated 03 / 10 / 2025, page 53 / 226 44 / 59 latent tensor (quantized) and the output of the probability model and emits a bit stream 1008.

[0144] The luma component codec decoder 1010 may comprise an entropy decoder 1012, a probability model 1014, a dequantizer, and a neural decoder 1016. In the figure, the dequantizer is not illustrated for simplification purposes. It is assumed that the probability model 1014 and the probability model 1004 are the same or substantially the same; for example, they may be copies of the same probability model. The entropy decoder 1012 may be an arithmetic decoder. The entropy decoder receives at least the bit stream 1008 and the output of the probability model 1014, and outputs a decoded (quantized) latent tensor. The decoded latent tensor may be subjected to dequantization. The decoded latent tensor or the dequantized decoded latent tensor is then fed into the neural decoder 1016.The neural decoder may comprise a first convolutional transposition layer (“UpConv5x5, 384, 2”, where UpConv refers to convolutional transposition, 5x5 is the kernel size, 384 is the number of output channels, and 2 is the step value), a first DSA block, a second convolutional transposition layer, a second DSA block, a third convolutional transposition layer, a third DSA block, a fourth convolutional transposition layer, a fourth DSA block, a nonlinear ReLU activation function, and a convolutional layer.

[0145] The output Ϋ of the 1010 neural decoder is a reconstruction of the luma component, where the size can be hx wx 1.

[0146] Chroma codec

[0147] The input for the chroma component codec may comprise at least fundamental truth data (e.g., a block or entire image), where at least part of the fundamental truth must be encoded / compressed. Furthermore, the input for the chroma codec may Petition 870250090159, dated 03 / 10 / 2025, page 54 / 226 45 / 59 include one or more extra data points, such as block / image resolution. Examples of fundamental truth input data are as follows: - A fundamental truth including both luma and chroma components. For example, the fundamental truth could be a YUV 4:4:4 color format image, represented as a 256x256x3 multidimensional array or tensor, where the horizontal side is 256 pixels, the vertical side is 256 pixels, and 3 channels are for Y, U, and V components, respectively. In another example, the fundamental truth could be a YUV 4:2:0 color format image, represented by the combination of a 256x256 matrix for the luma component and a 128x128x2 two-dimensional array (or tensor) for the chroma component. - A fundamental truth that includes only the chroma component. For example, the fundamental truth could be a 128x128x2 image, where the horizontal side is 128 pixels, the vertical side is 128 pixels, and the two channels are for the U component and the V component, respectively.

[0148] Figure 11 illustrates an exemplary implementation of a chroma component codec according to one embodiment. The input data x is a fundamental truth including both luma and chroma components, ehxwx 3 denotes the size of x, with height h, width w, and number of channels 3. To clarify, in this example, the input data for the chroma component codec is the same as the input data for the luma component codec. The chroma component codec is used to encode the chroma component of the input data; that is, the bitstream that is emitted by the chroma codec encoder represents the encoded chroma component, and the output of the chroma codec decoder represents the reconstructed or decoded chroma component.

[0149] The chroma component codec encoder 1100 may comprise a neural encoder (i.e., the first sub-encoder) 1102, an auxiliary encoder (i.e., the second sub-encoder) 1104, a quantizer (not shown in the figure), a probability model 1106 and an entropy encoder 1108. Petition 870250090159, dated 03 / 10 / 2025, page 55 / 226 46 / 59 Similar to the luma component codec neural encoder 1002, the chroma component codec neural encoder 1102 may also comprise a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. The latent tensor dimension would be h / / 16 x w / / 16 x 64, where the height is h / / 16, the width is w / / 16, and the number of channels is 64.

[0150] The chroma component codec also includes an encoder-side auxiliary encoder 1104, which generates auxiliary input for the probability model 1106, and a decoder-side auxiliary encoder 1118, which generates auxiliary input for the probability model 1116. In one example implementation, the encoder-side auxiliary encoder 1104 and the decoder-side auxiliary encoder 1118 may be two copies of the same component, for example, neural networks with the same architecture and weights. In another example implementation, the encoder-side auxiliary encoder 1104 and the decoder-side auxiliary encoder 1118 may be different neural network components, for example, neural networks with different architectures or neural networks with the same architecture but different weights.

[0151] According to one embodiment, an Ϋ input to the auxiliary encoder is the luma component reconstruction. In other words, the Ϋ input to the auxiliary encoder 1104 is the luma component codec output that has a size of hxwx 1.

[0152] According to another embodiment, an Ϋ input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0153] The masked version of the rebuilt luma component can be obtained via a masking operation performed on the rebuilt luma component. The masking operation can mask (for example, set Petition 870250090159, dated 03 / 10 / 2025, page 56 / 226 47 / 59 as zero or another predetermined value) some of the elements of the reconstructed luma component, such as the elements whose spatial coordinates do not correspond to the spatial coordinates of the element in the latent chroma tensor that is being encoded or decoded by the entropy encoder or entropy decoder.

[0154] According to another embodiment, an Ϋ input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0155] The smoothed version of the reconstructed luma component can be obtained via a smoothing operation performed on the reconstructed luma component. The smoothing operation can remove those elements from the reconstructed luma component that represent high-frequency information, such as those elements whose information does not correspond or is not well correlated with the information of the elements in the chroma component.

[0156] According to another embodiment, an Ϋ input to the auxiliary encoder is a predicted version of the chroma component obtained based on at least one prediction process, such as a neural network predictor, and the reconstructed luma component.

[0157] Thus, the input Ϋ can first be fed into a predictive neural network to predict the chroma component and the predicted chroma component can be used as an input to the auxiliary encoder 1104.

[0158] In yet another embodiment, an Ϋ input to the auxiliary encoder is the luma latent tensor that is emitted by the entropy decoder of the luma codec, or the dequantized luma latent tensor that is emitted by the dequantizer of the luma codec.

[0159] The auxiliary encoder may have the same architecture as the chroma component codec neural encoder, however any architecture suitable for extracting features from an image may be suitable. The auxiliary encoder output 1104 is fed into the probability model 1106 to operate as extra context information. Petition 870250090159, dated 03 / 10 / 2025, page 57 / 226 48 / 59

[0160] According to another embodiment, the extra context information that is provided to the probability model comprises data that are derived based on the reconstructed luma component.

[0161] According to another embodiment, the extra context information that is provided to the probability model comprises the latent luma tensor or the dequantized latent luma tensor. Since the reconstructed luma component consists of sufficient high-frequency information from the input data, the auxiliary input to probability model 1106 can help estimate the latent chroma probability density function. This probability model can bring significant performance gains to the encoding of the chroma component.

[0162] The latent tensor that is output by the chroma codec encoder 1102 (e.g., the latent chroma tensor) can be fed into the probability model. Probability model 1106 can be a neural network. With the auxiliary input as extra context information, probability model 1106 outputs an estimate of the probability of one or more elements of the latent chroma tensor.On the encoder side, the output of the 1106 probability model is used as one of the inputs to an 1108 entropy encoder. The entropy encoder can be an arithmetic encoder. The 1108 entropy encoder receives at least the latent (quantized) tensor and the output of the 1108 probability model and outputs a 1110 bit stream representing the encoded chroma component.

[0163] The 1112 decoder of the chroma component codec may comprise an entropy decoder 1114, a probability model 1116, a dequantizer (not shown in the figure), an auxiliary encoder 1118, and a neural decoder 1120. The entropy decoder 1114 may be an arithmetic decoder. The entropy decoder 1114 receives at least the bit stream 1110 and the output of the probability model 1116, and outputs a decoded latent (quantized) tensor. The probability model 1116 may need to be equal to or substantially equal to the probability model 1108. Petition 870250090159, dated 03 / 10 / 2025, page 58 / 226 49 / 59 which is available on the encoder side. In an exemplary implementation, auxiliary encoder 1118 is equal to or substantially equal to auxiliary encoder 1104 which is available on the encoder side. The decoded (quantized) latent tensor can be dequantized. After dequantization, the dequantized decoded latent tensor can be concatenated with the auxiliary input across the channel dimension. After concatenation, the dimension of the dequantized decoded latent tensor can be h / / 16 xw / / 16 x 728. Then, the new decoded latent tensor is inserted into the 1120 neural decoder. The 1120 neural decoder can comprise a first convolutional transposition layer, a first DSA block, a second convolutional transposition layer, a second DSA block, a third convolutional transposition layer, a third DSA block, a nonlinear ReLU activation function, and a convolutional layer.

[0164] The output of the 1120 neural decoder is a chroma component reconstruction, and the size can be h / / 2 x wll2 x 2. In this example implementation, because the input data is in YUV 4:2:0 format, the chroma component is 4 times smaller than the luma component in spatial size. Consequently, the chroma component is half the luma component in height, and the chroma component is half the luma component in width. Finally, one output channel can be used for U reconstruction, and the other output channel can be used for V reconstruction, where the size of each of U and V is equal to h / / 2 x wll2 x 7.

[0165] According to one embodiment, the luma component codec can be trained first, and then the chroma component codec can be trained based at least on the reconstructed luma component that is obtained based on the trained luma component codec, for a certain number of images or training blocks. According to one embodiment, the luma component codec and the chroma component codec are trained in two sequential steps, where in a first step only the component codec Petition 870250090159, dated 03 / 10 / 2025, page 59 / 226 In a first step, only the luma component codec is trained, and in a second step, only the chroma component codec is trained. In another mode, the two steps of the previous mode can be performed multiple times, so that in a first step only the luma component codec is trained, in a second step only the chroma component codec is trained, in a third step only the luma component codec is trained, in a fourth step only the chroma component codec is trained, and so on.

[0166] In another embodiment, the luma component codec and the chroma component codec can be trained together, that is, the chroma component codec is conditioned on the luma component codec and both are trained at the same time or substantially at the same time. Alternatively, the luma codec and the chroma codec can be trained in alternating phases, for example, the luma codec is trained for a first number of iterations, then the chroma codec is trained for a second number of iterations, then the luma codec is trained for a third number of iterations, then the chroma codec is trained for a fourth number of iterations, and so on.

[0167] According to one embodiment, the intraframe codec includes a first codec that is used to encode the chroma component of the input image, and a second codec that is used to encode the luma component of the input image based at least on the reconstructed chroma component.

[0168] The methods presented above relating to encoding and decoding aspects can be implemented in a respective encoding and decoding apparatus. An apparatus according to one aspect comprises means for receiving input data comprising a luma component and a chroma component; means for providing first fundamental truth data to a first encoder, said fundamental truth data comprising at least a first part of the input data; means for obtaining an encoded luma component from an output of the first encoder; means for providing the luma component Petition 870250090159, dated 03 / 10 / 2025, page 60 / 226 51 / 59 encoded for a first decoder; means for obtaining a reconstructed luma component from an output of the first decoder; means for providing second fundamental truth data comprising at least a second part of the input data and the reconstructed luma component for a second encoder, said second encoder comprising at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model; means for providing an output of the first sub-encoder and an output of the second sub-encoder for the probability model; means for obtaining one or more first probabilities from an output of the probability model; and means for obtaining, based on an output of the first sub-encoder and one or more probabilities, an encoded chroma component from an output of the second encoder.

[0169] According to one embodiment, the first encoder, the first decoder and the second encoder belong to an end-to-end learned intraframe codec.

[0170] According to one embodiment, said first encoder comprises a neural encoder, a probability model, and an entropy encoder; wherein the neural encoder comprises means for converting the input data into a plurality of latent tensor elements; the probability model comprises means for estimating a probability of each latent tensor element; and the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element.

[0171] According to one embodiment, said first sub-encoder of the second encoder is a neural encoder and said second sub-encoder of the second encoder is an auxiliary encoder comprising means for generating an auxiliary input to the probability model.

[0172] According to one modality, an input for the encoder Petition 870250090159, dated 03 / 10 / 2025, page 61 / 226 52 / 59 auxiliary is the reconstruction of the luma component.

[0173] According to one embodiment, an input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0174] According to one embodiment, an input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0175] According to one embodiment, an input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction of the reconstructed luma component.

[0176] An apparatus according to another aspect comprises means for receiving input data comprising an encoded chroma component and a reconstructed luma component in a decoder, said decoder comprising at least a first subdecoder, a second subdecoder and a probability model; means for providing the reconstructed luma component to the second subdecoder; means for obtaining one or more probabilities from an output of the probability model, based at least on an output of the second subdecoder and on a previously decoded chroma component; means for obtaining an entropy-decoded chroma component based at least on the encoded chroma component and on one or more probabilities; means for providing the entropy-decoded chroma component to the first subdecoder; and means for obtaining a reconstructed chroma component from an output of the first subdecoder.

[0177] According to one embodiment, said decoder comprises an entropy decoder and a neural decoder; wherein the probability model of the decoder comprises means for estimating a probability of each decoded latent tensor element; the entropy decoder comprises means for outputting a plurality of at least partially decoded latent tensor elements based on the input data and the probability of each latent tensor element; and the neural decoder comprises means for converting the plurality of latent tensor elements Petition 870250090159, dated 03 / 10 / 2025, page 62 / 226 53 / 59 decoded in the reconstructed chroma component.

[0178] According to one embodiment, said first subdecoder of the decoder is a neural decoder and said second subdecoder of the decoder is an auxiliary decoder comprising means for generating an auxiliary input to the probability model.

[0179] According to one embodiment, the apparatus comprises means for concatenating the decoded latent tensors with the auxiliary input decoder along dimensions of latent tensor channels.

[0180] According to one embodiment, the decoder belongs to an end-to-end learned intraframe codec.

[0181] As a further aspect, an apparatus is provided comprising: at least one processor and at least one memory, wherein said at least one memory is stored with code therein, which, when executed by said at least one processor, causes the apparatus to perform at least: receiving input data comprising a luma component and a chroma component; means for providing first fundamental truth data to a first encoder, wherein said fundamental truth data comprise at least a first part of the input data; obtaining an encoded luma component from an output of the first encoder; providing the encoded luma component to a first decoder; obtaining a reconstructed luma component from an output of the first decoder;provide second fundamental truth data comprising at least a second portion of the input data and the reconstructed luma component for a second encoder, said second encoder comprising at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model; provide an output from the first sub-encoder and an output from the second sub-encoder for the probability model; obtain one or more first probabilities from an output of the probability model; and obtain, based on an output from the first; Petition 870250090159, dated 03 / 10 / 2025, page 63 / 226 54 / 59 sub-encoder and in one or more probabilities, a chroma-encoded component of an output from the second encoder.

[0182] According to one embodiment, the first encoder, the first decoder and the second encoder belong to an end-to-end learned intraframe codec.

[0183] According to one embodiment, said first encoder comprises a neural encoder, a probability model, and an entropy encoder; wherein the neural encoder comprises code configured to make the apparatus convert the input data into a plurality of latent tensor elements; the probability model comprises code configured to make the apparatus estimate a probability of each latent tensor element; and the entropy encoder comprises code configured to make the apparatus output a bit stream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element.

[0184] According to one embodiment, said first sub-encoder of the second encoder is a neural encoder and said second sub-encoder of the second encoder is an auxiliary encoder comprising code configured to make the apparatus generate an auxiliary input for the probability model.

[0185] According to one embodiment, an input to the auxiliary encoder is the reconstruction of the luma component.

[0186] According to one embodiment, an input to the auxiliary encoder is a masked version of the reconstructed luma component.

[0187] According to one embodiment, an input to the auxiliary encoder is a smoothed version of the reconstructed luma component.

[0188] According to one embodiment, an input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction of the reconstructed luma component.

[0189] As another additional aspect, a device is provided that Petition 870250090159, dated 03 / 10 / 2025, p. 64 / 226 55 / 59 comprises: at least one processor and at least one memory, wherein said at least one memory is stored with code which, when executed by said at least one processor, causes the apparatus to perform at least: receiving input data comprising an encoded chroma component and a luma component reconstructed in a decoder, wherein said decoder comprises at least a first subdecoder, a second subdecoder and a probability model; providing the reconstructed luma component to the second subdecoder; obtaining one or more probabilities from an output of the probability model, based on at least one output of the second subdecoder and a previously decoded chroma component; obtaining an entropy-decoded chroma component based on at least the encoded chroma component and one or more probabilities; providing the entropy-decoded chroma component to the first subdecoder;and obtain a reconstructed chroma component from an output of the first subdecoder.

[0190] According to one embodiment, said decoder comprises an entropy decoder and a neural decoder; wherein the probability model of the decoder comprises code configured to make the apparatus estimate a probability of each decoded latent tensor element; the entropy decoder comprises code configured to make the apparatus output a plurality of at least partially decoded latent tensor elements based on the input data and the probability of each latent tensor element; and the neural decoder comprises code configured to make the apparatus convert the plurality of decoded latent tensor elements into the reconstructed chroma component.

[0191] According to one embodiment, said first subdecoder of the decoder is a neural decoder and said second subdecoder of the decoder is an auxiliary decoder comprising code configured to make the apparatus generate an auxiliary input for the probability model.

[0192] According to one embodiment, the apparatus comprises code. Petition 870250090159, dated 03 / 10 / 2025, page 65 / 226 56 / 59 configured to make the device concatenate the decoded latent tensors with the auxiliary input decoder along latent tensor channel dimensions.

[0193] According to one embodiment, the decoder belongs to an end-to-end learned intraframe codec.

[0194] Such devices may comprise, for example, the functional units revealed in Figures 1-7 to implement the modalities.

[0195] Such apparatus further comprises code, stored in said at least one non-transient memory which, when executed by said at least one processor, causes the apparatus to perform one or more of the modalities disclosed herein.

[0196] Above, when the exemplary modalities were described with reference to an encoder, it must be understood that the resulting bitstream and the decoder have corresponding elements within them. Similarly, when the exemplary modalities were described with reference to a decoder, it is necessary to understand that the encoder may have a computer structure and / or program to generate the bitstream to be decoded by the decoder. For example, some modalities were described related to the generation of a prediction block as part of the encoding. The modalities can be performed similarly by generating a prediction block as part of the decoding, with the difference that the encoding parameters, such as horizontal and vertical deviation, are decoded from the bitstream and not determined by the encoder.

[0197] The embodiments of the invention described above describe the codec in terms of separate encoder and decoder apparatus in order to aid in understanding the processes involved. However, it would be appreciated if the apparatus, structures and operations could be implemented as a single encoder-decoder apparatus / structure / operation. Additionally, it is possible that the encoder and decoder share some or all of the common elements. Petition 870250090159, dated 03 / 10 / 2025, page 66 / 226 57 / 59

[0198] Although the above examples describe embodiments of the invention operating within a codec inside an electronic device, it would be appreciated that the invention, as defined in the claims, could be implemented as part of any video codec. Thus, for example, embodiments of the invention may be implemented in a video codec that can implement video encoding via fixed or wired communication paths.

[0199] Thus, user equipment may comprise a video codec such as those described in the embodiments of the invention above. It should be noted that the term user equipment is intended to encompass any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, or portable web browsers.

[0200] Additionally, elements of a public terrestrial mobile network (PLMN) may also comprise video codecs, as described above.

[0201] In general, the various embodiments of the invention can be implemented in special-purpose hardware or circuits, software, logic, or any combination thereof. For example, some aspects may be implemented in hardware, while other aspects may be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although the invention is not limited thereto. Although various aspects of the invention may be illustrated and described as block diagrams, flowcharts, or using some other pictorial representation, it is well understood that these blocks, apparatus, systems, techniques, or methods described herein may be implemented, as non-limiting examples, in hardware, software, firmware, special-purpose circuits or logic, general-purpose hardware or controller, or other computing devices, or some combination thereof.

[0202] The embodiments of this invention can be implemented by Petition 870250090159, dated 03 / 10 / 2025, page 67 / 226 58 / 59 computer software executable by a mobile device's data processor, as in the processor entity, or by hardware, or by a combination of software and hardware. Additionally, in this respect, it should be noted that any blocks of the logic flow, as in the Figures, may represent program steps, or logic circuits, interconnected blocks and functions, or a combination of program steps and logic circuits, blocks and functions. The software may be stored on such physical media as memory chips or memory blocks implemented in the processor, magnetic media such as hard disks or floppy disks, and optical media such as, for example, DVDs and data variants thereof, CDs.

[0203] Memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. Data processors may be of any type suitable to the local technical environment and may include one or more general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), and multi-core processor architecture-based processors, as non-limiting examples.

[0204] Modalities of the inventions can be practiced in various components, such as integrated circuit modules. The design of integrated circuits is, to a large extent, a highly automated process. Complex and powerful software tools are available to convert a logic-level design into a semiconductor circuit design ready to be etched and formed onto a semiconductor substrate.

[0205] Programs, such as those provided by Synopsys, Inc. of Mountain View, California, and Cadence Design of San Jose, California, automatically route conductors and locate components on a semiconductor chip using well-established design rules, as well as Petition 870250090159, dated 03 / 10 / 2025, page 68 / 226 59 / 59 libraries of pre-stored design modules. Once a semiconductor circuit design has been completed, the resulting design, in a standardized electronic format (e.g., Opus, GDSII, or similar), can be transmitted to a semiconductor fabrication facility or fab for manufacturing.

[0206] The preceding description has provided, by means of non-limiting illustrative examples, a complete and informative description of the illustrative embodiment of this invention. However, various modifications and adaptations may become apparent to those skilled in the relevant art, in view of the preceding description, when read in conjunction with the accompanying drawings and the appended claims. Nevertheless, all such modifications and similar modifications of the teachings of this invention will still fall within the scope of this invention. Petition 870250090159, dated 03 / 10 / 2025, page 69 / 226

Claims

1 / 5 CLAIMS 1. Apparatus characterized by comprising means for receiving input data comprising a luma component and a chroma component; means for providing first fundamental truth data to a first encoder, said fundamental truth data comprising at least a first part of the input data; means for obtaining an encoded luma component from an output of the first encoder; means for providing the encoded luma component to a first decoder; means for obtaining a reconstructed luma component from an output of the first decoder;means for providing second fundamental truth data comprising at least a second portion of the input data and the reconstructed luma component for a second encoder, said second encoder comprising at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model; means for providing an output from the first sub-encoder and an output from the second sub-encoder for the probability model; means for obtaining one or more first probabilities from an output of the probability model; and means for obtaining, based on the output of the first sub-encoder and one or more probabilities, an encoded chroma component from an output of the second encoder.

2. Device according to claim 1, characterized in that the first encoder, the first decoder and the second encoder belong to an end-to-end learned intra-frame codec. Petition 870250090159, dated 03 / 10 / 2025, p. 70 / 226 2 / 5 3. Apparatus, according to claim 1 or 2, characterized in that said first encoder comprises a neural encoder, a probability model and an entropy encoder; the neural encoder comprises means for converting the input data into a plurality of latent tensor elements; the probability model comprises means for estimating a probability of each latent tensor element; and the entropy encoder comprises means for outputting a bit stream encoded at least partially based on the plurality of latent tensor elements and the probability of each latent tensor element.

4. Apparatus, according to any of the preceding claims, characterized in that said first sub-encoder of the second encoder is a neural encoder and said second sub-encoder of the second encoder is an auxiliary encoder comprising means for generating an auxiliary input to the probability model.

5. Apparatus, according to claim 4, characterized in that an input for the auxiliary encoder is a reconstruction of the luma component.

6. Apparatus, according to claim 4, characterized in that an input for the auxiliary encoder is a masked version of the reconstructed luma component.

7. Apparatus, according to claim 4, characterized in that an input for the auxiliary encoder is a smoothed version of the reconstructed luma component.

8. Apparatus, according to claim 4, characterized by the fact that an input to the auxiliary encoder is a predicted version of the chroma component obtained as a prediction of the reconstructed luma component.

9. A method characterized by comprising receiving input data comprising a luma component and a chroma component; providing first fundamental truth data to a first encoder, said fundamental truth data comprising at least a first part of the input data; obtaining an encoded luma component from an output of the first encoder; providing the encoded luma component to a first decoder; obtaining a reconstructed luma component from an output of the first decoder; providing second fundamental truth data comprising at least a second part of the input data and the reconstructed luma component to a second encoder, said second encoder comprising at least a first sub-encoder for the fundamental truth data and a second sub-encoder for the reconstructed luma component and a probability model;Provide an output from the first sub-encoder and an output from the second sub-encoder to the probability model; obtain one or more first probabilities from an output of the probability model; and obtain, based on the output of the first sub-encoder and one or more probabilities, a chroma-encoded component from an output of the second encoder.

10. Apparatus characterized by comprising Petition 870250090159, of 03 / 10 / 2025, page 72 / 226 4 / 5 means for receiving input data comprising an encoded chroma component and a reconstructed luma component in a decoder, said decoder comprising at least a first subdecoder, a second subdecoder and a probability model; means for providing the reconstructed luma component to the second subdecoder; means for obtaining one or more probabilities from an output of the probability model, based at least on an output of the second subdecoder and on a previously decoded chroma component; means for obtaining an entropy-decoded chroma component based at least on the encoded chroma component and on one or more probabilities; means for providing the entropy-decoded chroma component to the first subdecoder; and means for obtaining a reconstructed chroma component from an output of the first subdecoder.

11. Apparatus, according to claim 10, characterized in that said decoder comprises an entropy decoder and a neural decoder; wherein the probability model of the decoder comprises means for estimating a probability of each decoded latent tensor element; the entropy decoder comprises means for outputting a plurality of at least partially decoded latent tensor elements based on the input data and the probability of each latent tensor element; and the neural decoder comprises means for converting the plurality of decoded latent tensor elements into the reconstructed chroma component. Petition 870250090159, dated 10 / 03 / 2025, p. 73 / 226 5 / 5 12. Apparatus, according to claim 10 or 11, characterized in that said first subdecoder of the decoder is a neural decoder and said second subdecoder of the decoder is an auxiliary decoder comprising means for generating an auxiliary input to the probability model.

13. Apparatus, according to claim 12, characterized by comprising means for concatenating the decoded latent tensors with the auxiliary input decoder along dimensions of latent tensor channels.

14. Device according to any one of claims 10-13, characterized in that the decoder belongs to an end-to-end learned intraframe codec.

15. A method characterized by comprising receiving input data comprising an encoded chroma component and a reconstructed luma component in a decoder, said decoder comprising at least a first subdecoder, a second subdecoder, and a probability model; providing the reconstructed luma component to the second subdecoder; obtaining one or more probabilities from an output of the probability model, based on at least one output of the second subdecoder and a previously decoded chroma component; obtaining an entropy-decoded chroma component based on at least the encoded chroma component and one or more probabilities; providing the entropy-decoded chroma component to the first subdecoder; and obtaining a reconstructed chroma component from an output of the first subdecoder. Petition 870250090159, dated 03 / 10 / 2025, p. 74 / 226

Images