Apparatus, method, and computer program for video coding and decoding

Abstract

A method comprising: receiving an initial reconstruction of the input data of an encoder; receiving the input data including encoded residuals, the encoded residuals being determined based on at least the ground truth data of the encoder, the initial reconstruction of the input data, and an adaptive signal to a decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit, and a probabilistic model; obtaining an entropy-decoded residual component based on at least the encoded residual component and one or more probabilities of the probabilistic model, the entropy-decoded residual component being represented as a plurality of decoded latent tensor elements; providing the entropy-decoded residual component to the first sub-decoder; providing the adaptive signal to the syntax generation unit to generate one or more syntaxes based on the adaptive signal; adapting the decoding of the residual component in the first sub-decoder based on the one or more syntaxes; obtaining a reconstructed residual component from the output of the first sub-decoder; and obtaining a final reconstruction of the input data by combining the reconstructed residual component with the initial reconstruction of the input data.

Description

Technical Field

[0001] The present invention relates to an apparatus, a method, and a computer program for video coding and decoding.

Background Art

[0002] In video coding, video and image samples are usually encoded using a color representation such as YUV or YCbCr, which consists of one luminance (luma) channel, often denoted as Y, and two chrominance (chroma) channels, often denoted as U, V or Cb, Cr. In these cases, the luminance channel, which represents most of the illumination of the scene, is usually encoded at a certain resolution, while the chrominance channels, which usually represent the difference between certain color components, are often encoded at a second resolution lower than the resolution of the luminance signal. The purpose of this type of differential representation is to lose the correlation of the color components and to enable more efficient compression of the data.

[0003] Neural networks (NNs) have been used in the context of image and video compression by replacing one or more of the components of conventional codec or by utilizing end-to-end learning-based compression.

[0004] However, NN-based video coding approaches, including solutions based on end-to-end learning-based compression, have been shown to be insufficient in terms of optimizing the rate-distortion performance of the codec.

Summary of the Invention

[0005] To at least mitigate the above problems, an enhanced method for improving rate-distortion performance is presented herein.

[0006] The scope of protection for various embodiments of the present invention is defined by the independent claims. Any embodiments and features described herein that are not covered by the independent claims should be interpreted as useful examples for understanding the various embodiments of the present invention.

[0007] The method according to the first aspect involves first ground truth data. The method includes receiving input data including data, providing first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data, obtaining initial encoded input data from the output of the first encoder, providing the initial encoded input data to a first decoder, obtaining an initial reconstruction of the input data from the output of the first decoder, determining residuals based at least on the first ground truth data and the initial reconstruction of the input data, and providing second ground truth data including at least the residuals to a second encoder, wherein the second encoder comprises at least a first subencoder and a probabilistic model, providing second ground truth data to the first subencoder, providing the output of the first subencoder to the probabilistic model, obtaining one or more first probabilities from the output of the probabilistic model, and obtaining encoded residual information from the output of the second encoder based on the output of the first subencoder and one or more probabilities.

[0008] A method according to a second aspect includes receiving the input data including an initial reconstruction of the encoder's input data, the encoded residuals being determined based on at least the encoder's ground truth data, the initial reconstruction of the input data, and an adaptive signal to a decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit, and a probabilistic model; obtaining an entropy-decoded residual based on at least the encoded residual components and one or more probabilities of the probabilistic model, the entropy-decoded residual being represented as a plurality of decoded latent tensor elements; providing the entropy-decoded residual to the first sub-decoder; providing the adaptive signal to the syntax generation unit to generate one or more syntaxes based on the adaptive signal; adapting the decoding of the residual in the first sub-decoder based on the one or more syntaxes; obtaining a reconstructed residual from the output of the first sub-decoder; and obtaining a final reconstruction of the input data by combining the reconstructed residual with the initial reconstruction of the input data.

[0009] An apparatus according to a third aspect includes means for receiving input data including first ground truth data; means for providing the first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; means for obtaining initial encoded input data from the output of the first encoder; means for providing the initial encoded input data to a first decoder; means for obtaining an initial reconstruction of the input data from the output of the first decoder; and means for determining residuals based at least on the first ground truth data and the initial reconstruction of the input data. The means for providing a second ground truth data including at least residuals to a second encoder, wherein the second encoder comprises at least a first sub-encoder for the second ground truth data and a probabilistic model, the means for providing the second ground truth data to the first sub-encoder, the means for providing the output of the first sub-encoder to the probabilistic model, the means for obtaining one or more first probabilities from the output of the probabilistic model, and the means for obtaining encoded residual information from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities.

[0010] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intra-frame codec.

[0011] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder comprises means for converting input data into a plurality of latent tensor elements, the probabilistic model comprises means for estimating the probability of one or more of the plurality of latent tensor elements, and the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of one or more of the plurality of latent tensor elements.

[0012] The apparatus according to the fourth aspect includes means for receiving an encoder's input data including an initial reconstruction of the input data and an encoded residual, the encoded residual being determined based on at least the encoder's ground truth data, the initial reconstruction of the input data and an adaptive signal to a decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit and a probabilistic model; means for obtaining an entropy-decoded residual based on at least the encoded residual and one or more probabilities of the probabilistic model, the entropy-decoded residual being represented as a plurality of decoded latent tensor elements; means for providing the entropy-decoded residual to the first sub-decoder; means for providing the adaptive signal to the syntax generation unit to generate one or more syntaxes based on the adaptive signal; means for adapting the decoding of the residual in the first sub-decoder based on the one or more syntaxes; means for obtaining a reconstructed residual from the output of the first sub-decoder; and means for obtaining a final reconstruction of the input data by combining the reconstructed residual with the initial reconstruction of the input data.

[0013] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, wherein the decoder's probabilistic model includes means for estimating the probability of one or more elements within a plurality of decoded latent tensor elements, the entropy decoder includes means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of one or more elements within the plurality of latent tensor elements, and the neural decoder includes means for converting the plurality of decoded latent tensor elements into reconstructed chroma components.

[0014] According to one embodiment, the first sub-decoder of the decoder is a neural decoder comprising one or more spatial attention units for receiving the one or more syntax and adapting the decoding of the residual components.

[0015] According to one embodiment, the adaptive signal input to the syntax generation network is an inversely quantized latent tensor output by another codec.

[0016] According to one embodiment, the adaptive signal input to the syntax generation network is an initial reconstruction of the input data.

[0017] According to one embodiment, the adaptive signal input to the syntax generation network is the output of an auxiliary encoder.

[0018] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0019] In a fifth aspect, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory has code thereon, and when executed by the at least one processor, the code receives input data including first ground truth data, provides the first ground truth data to a first encoder, the ground truth data including at least a first portion of the input data, obtains initial encoded input data from the output of the first encoder, provides the initial encoded input data to a first decoder, obtains initial reconstruction of the input data from the output of the first decoder, and first ground The device is made to at least perform the following: determine residuals based on truth data and an initial reconstruction of input data; provide second ground truth data including at least residuals to a second encoder, wherein the second encoder comprises at least a first sub-encoder and a probabilistic model for the second ground truth data; provide the second ground truth data to the first sub-encoder; provide the output of the first sub-encoder to the probabilistic model; obtain one or more first probabilities from the output of the probabilistic model; and obtain encoded residual information from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities.

[0020] An apparatus according to the sixth aspect comprises at least one processor and at least one memory, the at least one memory having code thereon, which, when executed by the at least one processor, receives input data including an initial reconstruction of the encoder's input data and an encoded residual, the encoded residual being determined based at least on the encoder's ground truth data, the initial reconstruction of the input data and an adaptive signal to the decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit and a probabilistic model, receiving and at least on one or more probabilities of the encoded residual and the probabilistic model The device is made to at least perform the following: obtain an entropy-decoded residual, the entropy-decoded residual being represented as a plurality of decoded latent tensor elements; provide the entropy-decoded residual to a first sub-decoder; provide the adaptive signal to a syntax generation unit to generate one or more syntaxes based on the adaptive signal; adapt the decoding of the residual in the first sub-decoder based on the one or more syntaxes; obtain a reconstructed residual from the output of the first sub-decoder; and obtain a final reconstruction of the input data by combining the reconstructed residual with an initial reconstruction of the input data.

[0021] A computer-readable storage medium containing the code is arranged to perform one or more of the methods and related embodiments described above.

[0022] To better understand the present invention, references to the accompanying drawings will be made here as an example. [Brief explanation of the drawing]

[0023] [Figure 1] This figure shows an example of a codec that has a neural network (NN) component. [Figure 2]A diagram showing another example of a video coding system having a neural network component. [Figure 3] An example of an end-to-end learning-based codec using a neural network is shown. [Figure 4] A diagram showing an example of an end-to-end learning-based video coding system using a neural network. [Figure 5] A diagram showing an example of video coding for a computer. [Figure 6] A diagram showing an example of a pipeline of an end-to-end learning-based system for video coding for a computer. [Figure 7] A diagram showing an example of training an end-to-end learning-based codec. [Figure 8] A diagram showing an example of a dense split attention (DSA) block. [Figure 9a] A diagram showing a flowchart of a coding method and a decoding method for improving rate-distortion performance according to an embodiment of the present invention. [Figure 9b] A diagram showing a flowchart of a coding method and a decoding method for improving rate-distortion performance according to an embodiment of the present invention. [Figure 10] A diagram for explaining an exemplary implementation of a codec in a first step according to an embodiment of the present invention. [Figure 11] A diagram for explaining an exemplary implementation of a codec in a second step according to an embodiment of the present invention. [Figure 12] A diagram showing an exemplary implementation of a spatial attention block according to an embodiment of the present invention. [Figure 13] A diagram showing an exemplary implementation of a first codec and a second codec, where the output of the first codec is input to the decoder of the second codec. [Figure 14]This figure shows an exemplary implementation of a first codec and a second codec, where the output of the first codec is input to the encoder of the second codec. [Figure 15] This figure shows exemplary implementations of the first and second codecs, where the output of the first codec is input into the probabilistic model of the second codec. [Figure 16] This figure shows an exemplary implementation of the first and second codecs, in which the output of the first codec is input to the decoder of the second codec, the encoder of the second codec, and the probabilistic model of the second codec. [Figure 17] This figure shows exemplary implementations of the first and second codecs, where the output of the second codec is the final output representing the reconstruction of the input to the first codec. [Modes for carrying out the invention]

[0024] The following descriptions and drawings are illustrative and should not be construed as unnecessarily limiting. Specific details are provided to fully understand this disclosure. However, in certain examples, well-known or conventional details are omitted to avoid obscuring the description. References to one embodiment or one representation in this disclosure may refer to the same embodiment, but are not necessarily so, and such references mean at least one of the embodiments.

[0025] Any reference in this specification to “one embodiment” means that certain features, structures, or characteristics described in relation to an embodiment are included in at least one embodiment of this disclosure.

[0026] Before discussing this embodiment in more detail, a brief reference to related technologies is given.

[0027] In the context of machine learning, a neural network (NN) is a computation graph composed of several computational layers, or computational parts. Each layer consists of one or more units, each unit performing basic computations. Units are connected to one or more other units, and these connections may be associated with weights. Weights may be used to scale the signals passing through the associated connections. Weights are learnable parameters, i.e., values ​​that can be learned from training data. There may be other learnable parameters, such as the parameters of the batch normalization layer.

[0028] Two widely used architectures for neural networks are the feedforward architecture and the recurrent architecture. In a feedforward neural network, there are no feedback loops, and each layer receives input from one or more of the preceding layers and provides its output as input to one or more of the following layers. Alternatively, units within a certain layer receive input from one or more of the preceding layers and provide their output to one or more of the following layers.

[0029] The initial layers (layers closest to the input data) may extract semantically low-level features such as edges and textures in the image, while the intermediate and final layers may extract higher-level features. Following the feature extraction layers, there may be one or more layers that perform certain tasks such as classification, semantic segmentation, object detection, denoising, style transfer, and super-resolution processing. In a recurrent neural network, there is a feedback loop, so that the network can have states, i.e., it can store information or states.

[0030] Neural networks are being used in a growing number of applications for various types of devices, such as mobile phones. Examples include image and video analysis and processing, social media data analysis, and device usage data analysis.

[0031] A key characteristic of at least some neural networks (and other machine learning tools) is their ability to learn properties about input data, either in a supervised or unsupervised manner, or both. Such learning is a result of training algorithms, or of meta-level neural networks that provide training signals.

[0032] Generally, a training algorithm consists of modifying several properties of a neural network so that its output becomes as close as possible to a desired output. For example, in the case of classifying objects in an image, the output of the neural network may be used to derive a class or category index that indicates the class or category to which the objects in the input image belong. Training usually occurs by minimizing or reducing the error of the output, also called loss. Examples of loss include mean squared error and cross-entropy. In recent deep learning techniques, training is an iterative process, and in each iteration, the algorithm modifies the weights of the neural network to gradually improve the network's output, i.e., gradually reduce the loss.

[0033] In this explanation, the terms "model" and "neural network" are used interchangeably, and the weights of a neural network may be referred to as learnable parameters or simply parameters.

[0034] Training a neural network is an optimization process. The goal of the optimization, or training, process is to allow the model to learn the characteristics of the data distribution from a limited training dataset. In other words, the goal is to use the limited training dataset to learn generalizations to data that has not been seen before, i.e., data that was not used to train the model. This is usually referred to as generalization. In practice, the data may be split into at least two sets: a training set and a validation set. The training set is used to train the network, i.e., to modify its learnable parameters to minimize the loss. The validation set is used to check the network's performance on data that was not used to minimize the loss, as an indication of the model's final performance. In particular, the errors on the training and validation sets are monitored during the training process to understand the following: - Whether the network has learned at all or not - if so, the error in the training set will decrease; otherwise, the model is in an underfitted state. - Whether the network is learning to generalize - In this case, the error on the validation set must also be reduced and not too large compared to the error on the training set. If the error on the training set is small, but the error on the validation set is much larger than the error on the training set, the error on the validation set will not decrease, or may even increase, and the model is overfitted. This means that the model simply memorizes the characteristics of the training set and performs well only on that set, but poorly on sets that are not used to tune its parameters.

[0035] While the above background information on neural networks may be valid at the time of writing this document, the entire field of neural networks and machine learning is rapidly evolving. Therefore, it should be understood that at least some of the embodiments described herein are not limited to the definitions of neural networks, machine learning models, or training algorithms given in the above background information.

[0036] In recent years, neural networks have been used to compress and decompress data such as images, i.e., in image codecs. The most widely used architecture for implementing one component of an image codec is the autoencoder, which is a neural network consisting of two parts: a neural encoder and a neural decoder. The neural encoder takes an image as input and generates a latent tensor, which may require fewer bits than the input image. The latent tensor may be quantized, losslessly compressed, and obtain a bitstream representing the encoded image. The neural decoder takes the bitstream and reconstructs the image that was input to the neural encoder.

[0037] Such neural encoders and decoders may be trained to minimize a combination of bitrate and distortion, the distortion of which may be based on one or more of the following metrics: mean squared error (MSE), peak signal-to-noise ratio (PSNR), or structural similarity index (SSIM). These distortion metrics are correlated to human visual perception quality such that minimizing or maximizing one or more of these distortion metrics results in an improvement in the visual quality of the decoded image as perceived by humans.

[0038] A video codec comprises an encoder that converts the input video into a compressed representation suitable for storage / transmission, and a decoder that decompresses the compressed video representation back into a viewable format. The encoder may discard some information in the original video sequence in order to represent the video in a more compact format (i.e., at a lower bitrate).

[0039] The H.264 / AVC standard was developed by the Joint Video Team (JVT) of the Video Coding Expert Group (VCEG) of the Telecommunication Standardization Sector of the International Telecommunication Union (ITU-T) and the Video Expert Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC). The H.264 / AVC standard is published by both parent standardization bodies and is referred to as ITU-T Recommendation H.264 and ISO / IEC International Standard 14496-10, and is also known as MPEG-4 Part 10 Advanced Video Coding (AVC). Extensions to H.264 / AVC include Scalable Video Coding (SVC) and Multiview Video Coding (MVC).

[0040] The High Efficiency Video Coding (H.265 / HEVC, also known as HEVC) standard was developed by the Joint Team for Video Coding (JCT-VC) of VCEG and MPEG. The standard was published by both parent standards bodies and is referred to as ITU-T Recommendation H.265 and ISO / IEC International Standard 23008-2, and is also known as MPEG-H Part 2 High Efficiency Video Coding (HEVC). Subsequent versions of H.265 / HEVC include Scalable, Multiview, Fidelity Range, 3D, and Screen Content Coding Extensions, which are sometimes abbreviated as SHVC, MV-HEVC, REXT, 3D-HEVC, and SCC, respectively.

[0041] VVC (H.266), also known as Multipurpose Video Coding, defined in ITU-T Recommendation H.266 and equivalently specified in ISO / IEC 23090-3 (also known as MPEG-I Part 3), is a video compression standard developed as a successor to HEVC. The reference software for VVC is the VVC Test Model (VTM).

[0042] The AV1 bitstream format and decoding process specifications were developed by the Alliance of Open Media (AOM). The AV1 specification was published in 2018. AOM is reportedly working on the AV2 specification.

[0043] The basic unit for both the input to a video encoder and the output to a video decoder is, in most cases, a picture. The picture given as input to the encoder is also called the source picture, and the picture decoded by the decoder is called the decoded picture or reconstructed picture.

[0044] The source picture and decode picture are the following set of sample sequences, namely: - Luma (Y) only (monochrome), - Luma and two chromosomes (YCbCr or YcgCo), - Green, blue, red (GBR, also known as RGB), - An array representing other unspecified monochrome or tristimulus color sampling (e.g., YZX, also known as XYZ) Each consists of one or more sample sequences, such as one of the following.

[0045] A component may be defined as an array or a single sample from one of the three sample arrays (luma and two chroma) that make up the picture, or as an array or a single sample of an array that makes up the picture in monochrome format.

[0046] Hybrid video codecs, such as ITU-T H.263 and H.264, may encode video information in two phases. First, pixel values ​​within a certain picture area (or "block") are predicted, for example, by motion compensation means (finding and indicating an area in one of the previously coded video frames that closely corresponds to the block being coded) or by spatial means (using pixel values ​​around the block being coded in a specified manner). Next, the prediction error, i.e., the difference between the predicted block of pixels and the original block of pixels, is coded. This may be done by transforming the difference in pixel values ​​using a specified transformation (e.g., the discrete cosine transform (DCT) or a variation thereof), quantizing the coefficients, and entropy coding the quantized coefficients. By varying the fidelity of the quantization process, the encoder can control the balance between the precision of the pixel representation (image quality) and the size of the resulting coded video representation (file size or transfer bitrate).

[0047] Interpretation, sometimes called temporal prediction, motion compensation, or motion-compensated prediction, utilizes temporal redundancy. In interpretation, the source of the prediction is a previously decoded picture.

[0048] Intra prediction utilizes the fact that adjacent pixels within the same picture may be correlated. Intra prediction can be performed in the spatial domain or the transformation domain, i.e., either sample values ​​or transformation coefficients can be predicted. Intra prediction may be used in intra coding where no interpretation is applied.

[0049] One outcome of the coding procedure is a set of coding parameters, such as motion vectors and quantization transformation coefficients. Many parameters can be entropy coded more efficiently if they are first predicted from spatially or temporally adjacent parameters. For example, motion vectors may be predicted from spatially adjacent motion vectors, and only the difference for the motion vector predictor may be coded. The prediction and intra-prediction of coding parameters are sometimes collectively referred to as in-picture prediction.

[0050] The decoder reconstructs the output video by applying prediction means similar to those of the encoder to form a predictive representation of pixel blocks (using motion or spatial information created by the encoder and stored in a compressed representation), and prediction error decoding (the inverse operation of prediction error coding to reconstruct the quantized prediction error signal in the spatial pixel domain). After applying the prediction means and prediction error decoding means, the decoder sums the prediction signal and the prediction error signal (pixel values) to form an output video frame. The decoder (and encoder) may also apply additional filtering means to improve the quality of the output video before passing the output video for display and / or before storing the output video as a predictive reference for future frames in the video sequence.

[0051] In a video codec, motion information may be indicated using motion vectors associated with each motion-compensated image block. Each of these motion vectors represents the displacement between the image block in the picture being coded (on the encoder side) or decoded (on the decoder side) and the predicted source block in one of the previously coded or decoded pictures. To efficiently represent motion vectors, they may be coded differentially against block-specific predicted motion vectors. In a video codec, the predicted motion vector may be created in a predetermined manner, for example, by calculating the median of the coded or decoded motion vectors of adjacent blocks. Another method for creating motion vector predictions is to generate a list of candidate predictions from adjacent blocks and / or co-located blocks in a time-reference picture and signal the selected candidates as motion vector predictors. In addition to predicting motion vector values, the reference index of a previously coded / decoded picture may be predicted. The reference index may be predicted from adjacent blocks and / or co-located blocks in a time-reference picture. Furthermore, high-efficiency video codecs can employ an additional motion information coding / decoding mechanism, often referred to as merging mode, in which all motion field information, including motion vectors and corresponding reference picture indices for each available reference picture list, is predicted and used without any modification or correction. Similarly, predicting motion field information may be performed using motion field information from adjacent and / or co-located blocks within a time reference picture, with the motion field information used signaled among a list of motion field candidate lists filled with motion field information from available adjacent / co-located blocks.

[0052] In video codecs, the predicted residuals after motion compensation may first be transformed using a transformation kernel (such as DCT) and then coded. This is because some correlation often still exists between the residuals, and the transformation can often help reduce this correlation, resulting in more efficient coding.

[0053] A video encoder may utilize a Lagrangian cost function to find the optimal coding mode, for example, the desired coding mode for a block, block partitioning, and associated motion vectors. This type of cost function uses a weighting coefficient λ to link the (exact or estimated) image distortion resulting from the irreversible coding method with the (exact or estimated) amount of information required to represent the pixel values ​​within the image area. C = D + λR Here, C is the Lagrange cost to be minimized, D is the image distortion (e.g., mean squared error) considering the mode vector and motion vector, and R is the number of bits required to represent the necessary data for reconstructing the image block in the decoder (including the amount of data to represent the candidate motion vector). Rate R may be the actual bitrate or number of bits resulting from the encoding. Alternatively, rate R may be an estimated bitrate or number of bits. One possible way to estimate rate R is to omit the final entropy encoding step and use, for example, a simpler entropy encoding or an entropy encoder in which some of the context states have not been updated according to the previous encoding mode selection.

[0054] Conventional distortion measurement metrics may include, but are not limited to, the peak signal-to-noise ratio (PSNR), mean squared error (MSE), sum of absolute differences (SAD), sum of absolute transform differences (SATD), and structural similarity (SSIM). These are typically measured between a reconstructed video / image signal (which is identical to, or will be identical to, the decoded video / image signal) and the "original" video / image signal provided as input for encoding.

[0055] Partitioning may also be defined as dividing a set into subsets such that each element of the set is in exactly one subset.

[0056] A bitstream may be defined as a sequence of bits, and in some coding formats or standards, may be in the form of a network abstraction layer (NAL) unit stream or a byte stream, forming a representation of coded pictures and associated data that make up one or more coded video sequences.

[0057] The bitstream format may include a sequence of syntactic structures.

[0058] Syntactic elements may be defined as elements of data represented within a bitstream. Syntactic structures may be defined as zero or more syntactic elements coexisting within a bitstream in a specified order.

[0059] A NAL unit may be defined as a syntactic structure containing an indication of the type of subsequent data and a byte that contains that data in the form of an RBSP with start code emulation prevention bytes inserted as needed. A raw byte sequence payload (RBSP) may be defined as a syntactic structure containing integer bytes encapsulated in the NAL unit. An RBSP is either empty or has the form of a string consisting of data bits containing the syntactic elements, followed by an RBSP stop bit and zero or more subsequent bits equal to zero.

[0060] Some coding formats specify a set of parameters that may carry parameter values ​​necessary for decoding or reconstructing a decoded picture. Parameters may be defined as syntactic elements of a parameter set. A parameter set may be defined as a syntactic structure that contains parameters and can be referenced from or activated by another syntactic structure, for example, using an identifier.

[0061] Coding standards or specifications may specify several types of parameter sets. It should be understood that embodiments may apply not only to the types of parameter sets described, but may also apply similarly to any parameter set type.

[0062] A parameter set may be activated, for example, when referenced through its identifier. An adaptive parameter set (APS) may be defined as a syntactic structure that applies to zero or more slices. There can be different types of adaptive parameter sets. An adaptive parameter set may, for example, contain filtering parameters for a particular type of filter. In VVC, three types of APS are specified that carry parameters for an adaptive loop filter (ALF), a lumens mapping with chromascaling (LMCS), and a scaling list. A scaling list may be defined as a list that associates each frequency index with a scaling factor for a scaling process, which multiplies the transformation factor level by the scaling factor to obtain the transformation factor. In VVC, an APS is referenced through its type (e.g., ALF, LMCS, or scaling list) and identifier. In other words, different types of APS have their own range of identifier values.

[0063] The Adaptive Parameter Set (APS) may include parameters for different types of decoding processes, such as adaptive loop filtering or lumamapping with chroma scaling.

[0064] Video coding specifications may allow the use of Supplemental Extensible Information (SEI) messages, etc. Some video coding specifications include SEI Network Abstraction Layer (NAL) units, and some video coding specifications include both prefix SEI NAL units and suffix SEI NAL units, the former type which can initiate picture units, etc., and the latter type which can terminate picture units, etc. An SEI NAL unit contains one or more SEI messages which are not required for decoding the output picture but may support related processes such as picture output timing, post-processing of decoded pictures, rendering, error detection, error hiding, and resource reservation. The H.264 / AVC, H.265 / HEVC, H.266 / VVC, and H.274 / VSEI standards specify several SEI messages, and user data SEI messages allow organizations and enterprises to specify SEI messages for their specific use. The standards may include syntax and semantics for specified SEI messages, but may not define the process for handling the messages at the receiving end. As a result, encoders may be required to adhere to standards specifying SEI messages when creating them, while decoders may not be required to process SEI messages for the sake of output order compatibility. One reason for including the syntax and semantics of SEI messages in the standard is to ensure that different system specifications interpret supplementary information in the same way and thus enable interoperability. System specifications may require the use of specific SEI messages at both the end of encoding and the end of decoding, and may specify a process for handling specific SEI messages at the receiving end. SEI messages are generally not extended in future modifications or versions of the standard.

[0065] The phrases "along a bitstream" (e.g., indicating along a bitstream) or "along a bitstream coding unit" (e.g., indicating along a coded tile) may be used to refer to the transmission, signaling, or storage of “out-of-band” data in a manner that is associated with, but not contained within, a bitstream or coding unit, respectively. The phrases "decoding along a bitstream" or "decoding along a bitstream coding unit" may refer to decoding referenced out-of-band data (which may be obtained from out-of-band transmission, signaling, or storage) associated with a bitstream or coding unit, respectively. For example, the phrase "along a bitstream" may be used when a bitstream is contained within a container file, such as a file conforming to an ISO-based media file format, and certain file metadata is stored in the file in a manner that is associated with the bitstream, such as boxes in sample entries for a track containing a bitstream, sample groups for a track containing a bitstream, or timed metadata tracks associated with a track containing a bitstream.

[0066] Image and video codecs may use a set of filters to enhance the visual quality of predicted visual content, which may be applied in-loop, out-loop, or both. In the case of in-loop filters, a filter applied to one block in the frame currently being encoded will affect the encoding of another block in the same frame, and / or another block in another frame predicted from the current frame. In-loop filters can affect bitrate and / or visual quality. In fact, enhanced blocks will have smaller residuals (the difference between the original block and the predicted and filtered block), and therefore require fewer bits for encoding. Out-loop filters are applied to the frame after it has been reconstructed, so the filtered visual content may only affect the visual quality of the frame output by the decoder, and not as a source for prediction.

[0067] Recently, neural networks (NNs) have been used in the context of image and video compression, primarily by following two approaches.

[0068] One approach involves using neural networks (NNs) to replace one or more components of traditional codecs such as VVC / H.266. Here, the term "traditional" refers to codecs where components and their parameters may not be learned from the data. Examples of such components include: - For example, an additional in-loop filter by adding a neural network (NN) as an additional in-loop filter to a conventional loop filter. - For example, a single in-loop filter by having a neural network that replaces all conventional in-loop filters. - Intraframe prediction. - Interframe prediction. - Conversion and / or inverse conversion. - A probabilistic model for arithmetic codecs. - others That is the case.

[0069] Figure 1 illustrates an example of the functionality of a neural network (NN) as a component of a conventional codec pipeline in one embodiment. In particular, Figure 1 illustrates an encoder that also includes a decoding loop. Figure 1 is shown to include the components described below.

[0070] A lumane intra-prediction block or circuit 101. This block or circuit performs intra-prediction in the lumane domain, for example, by using already reconstructed data from the same frame. The operation of the lumane intra-prediction block or circuit 101 may be performed by a deep neural network, such as a convolutional autoencoder.

[0071] Chroma intra-prediction block or circuit 102. This block or circuit performs intra-prediction in the chroma domain, for example, by using already reconstructed data from the same frame. The chroma intra-prediction block or circuit 102 may also perform cross-component prediction, for example, predicting chroma from luma. The operation of the chroma intra-prediction block or circuit 102 may be performed by a deep neural network, such as a convolutional autoencoder.

[0072] An intra-prediction block or circuit 103 and an inter-prediction block or circuit 104. These blocks or circuits perform intra-prediction and inter-prediction, respectively. The intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 may perform predictions for all components, e.g., lumers and chromas. The operation of the intra-prediction block or circuit 103 and the inter-prediction block or circuit 104 may be performed by two or more deep neural networks, such as convolutional autoencoders.

[0073] A probability estimation block or circuit 105 for entropy coding. This block or circuit performs probability predictions for the next symbol to encode or decode, and the predicted values ​​are subsequently provided to an entropy coding module 112, such as an arithmetic coding module, for encoding or decoding the next symbol. The operation of the probability estimation block or circuit 105 may be performed by a neural network.

[0074] A transform and quantization (T / Q) block or circuit 106. These are actually two blocks or circuits. The transform and quantization block or circuit 106 transforms the input data into different domains; for example, an FFT transform would transform the data into the frequency domain. The transform and quantization block or circuit 106 may quantize its input value into a smaller set of possible values. The decoding loop may also include an inverse quantization block or circuit and an inverse transform block or circuit 113. One or both of the transform block or circuit and the quantization block or circuit may be replaced by one or more neural networks. One or both of the inverse transform block or circuit and the inverse quantization block or circuit 113 may be replaced by one or more neural networks.

[0075] An in-loop filter block or circuit 107. The operation of the in-loop filter block or circuit 107 is carried out within the decoding loop and filters the output of the inverse transform block or circuit, or in any case, the reconstructed data, to enhance the reconstructed data with respect to one or more predetermined quality metrics. This filter may affect both the quality of the decoded data and the bitrate of the bitstream output by the encoder. The operation of the in-loop filter block or circuit 107 may be carried out by a neural network, such as a convolutional autoencoder. For example, the operation of the in-loop filter may be carried out by multiple steps or filters, and one or more steps may be carried out by a neural network.

[0076] A post-processing filter block or circuit 108. Since the post-processing filter block or circuit 108 may not affect the encoding process, it may be implemented only on the decoder side. The post-processing filter block or circuit 108 filters the reconstructed data output by the in-loop filter block or circuit 107 to enhance the quality of the reconstructed data. The post-processing filter block or circuit 108 may be replaced by a neural network, such as a convolutional autoencoder.

[0077] Resolution-adaptive block or circuit 109: This block or circuit may downsample the input video frame before encoding. Then, in the decoding loop, the reconstructed data may be upsampled to the original resolution by the upsampling block or circuit 110. The operation of the resolution-adaptive block or circuit 109 may be carried out by a neural network such as a convolutional autoencoder.

[0078] Encoder control block or circuit 111. This block or circuit optimizes the encoder parameters, such as what transformations to use, what quantization parameters (QP) to use, and which intra-prediction modes (out of N intra-prediction modes) to use. The operation of the encoder control block or circuit 111 may be carried out by a neural network, such as a classifier convolutional network or a regressive convolutional network.

[0079] The ME / MC block or circuit 114 performs motion estimation and / or motion compensation, which are two key operations performed when performing interframe prediction. ME / MC represents motion estimation / motion compensation.

[0080] Another approach, commonly referred to as "end-to-end learning compression," uses neural networks (NNs) as the primary component of the image / video codec. This second approach has two main options.

[0081] Option 1: Reuse the video coding pipeline but replace most or all of its components with neural networks. Referring to Figure 2, an example of a modified video coding pipeline based on a neural network, according to one embodiment, is illustrated. The neural network example includes, but is not limited to, a compressed representation of the neural network. Figure 2 shows that it includes the following components: - Neural transformation block or circuit 202: This block or circuit transforms the output of the addition / subtraction operation 203 into a new representation of the data, which may have lower entropy and therefore be more compressible. - Quantization block or circuit 204: This block or circuit quantizes the input data 201 into a smaller set of possible values. - Inverse transform and inverse quantization block or circuit 206. These blocks or circuits perform the inverse or nearly inverse operations of the transform and quantization, respectively. - Encoder parameter control block or circuit 208. This block or circuit may control and optimize some or all of the parameters of the encoding process, such as one or more parameters of the encoding block or circuit. - Entropy coding block or circuit 210. This block or circuit may perform reversible coding based on entropy, for example. One common entropy coding technique is arithmetic coding. - Neural intracodec block or circuit 212. This block or circuit may also be an image compression and decompression block or circuit that may be used to code and decode intraframes. Encoder 214 may be an encoder block or circuit, such as the neural encoder section of an autoencoder neural network. Decoder 216 may be a decoder block or circuit, such as the neural decoder section of an autoencoder neural network. Intracoding block or circuit 218 may be a block or circuit that performs several intermediate steps between the encoder and decoder, such as quantization, entropy coding, entropy decoding, and / or inverse quantization. - Deep loop filter block or circuit 220. This block or circuit performs filtering of the reconstructed data to enhance the reconstructed data. - Decode picture buffer block or circuit 222. This block or circuit is a memory buffer that holds the decoded frame, e.g., the reconstructed frame 224, and the enhanced reference frame 226 used for interpretation. - Interframe prediction block or circuit 228. This block or circuit performs interframe prediction, for example, predicting from a frame that is temporally close, for example, frame 232. ME / MC 230 performs motion estimation and / or motion compensation, which are two key operations performed when interframe prediction is performed. ME / MC represents motion estimation / motion compensation.

[0082] Option 2: Redesign the entire pipeline as follows: - The encoder NN is configured to perform nonlinear transformations. - Quantization and lossless encoding of the output of encoder NNs. - Reversible decoding and inverse quantization, - The decoder NN is configured to perform a nonlinear inverse transform.

[0083] An example of Option 2 is illustrated in detail in Figure 3, which shows the encoder NN and decoder NN, which are part of the example neural autoencoder architecture. In Figure 3, the analysis network 301 is the encoder NN and the synthesis network 302 is the decoder NN, which together are sometimes referred to as the spatial correlation tool 303, or neural autoencoder.

[0084] As shown in Figure 3, the input data 304 is analyzed by an encoder NN (analysis network 301), which outputs a new representation of the input data. This new representation may be more compressible. This new representation may then be quantized to discrete numerical values ​​by a quantizer 305. The quantized data is then reversibly coded, for example, by an arithmetic encoder 306, thus obtaining a bitstream 307. The example shown in Figure 3 includes an arithmetic decoder 308 and an arithmetic encoder 306. The arithmetic encoder 306, or the arithmetic decoder 308, or a combination of the arithmetic encoder 306 and the arithmetic decoder 308 may be referred to as an arithmetic codec in some embodiments. On the decoding side, the bitstream is first reversibly decoded, for example, by using an arithmetic codec decoder 308. The reversibly decoded data is dequantized and then input to a decoder NN, a synthesis network 302. The output is the reconstructed or decoded data 309.

[0085] In the case of lossy compression, the lossy step may include an encoder neural network and / or quantization.

[0086] To train this system, a training objective function (also called the "training loss") may be used, which may contain one or more terms, a loss term, or simply a loss. In one example, the training loss includes a reconstruction loss term and a rate loss term. The reconstruction loss prompts the system to decode data similar to the input data according to some similarity metric. An example of a reconstruction loss is: - Mean squared error (MSE), - Multiscale structural similarity (MS-SSIM), - The loss obtained from using a pre-trained neural network. For example, error(f1, f2), where f1 and f2 are features extracted by the pre-trained neural network on the input data and the decoded data, respectively, and error() is an error function or distance function such as the L1 norm or L2 norm. - Loss obtained from using an end-to-end learning codec and a neural network trained concurrently. For example, adversarial loss may be used, which is the loss provided by a discriminator neural network trained adversarially with respect to the codec, according to the proposed configuration in terms of generative adversarial networks (GANs) and their variants. That is the case.

[0087] Rate loss prompts the system to compress the output of the encoding stage, such as the output of an arithmetic encoder. In this context, "compression" means reducing the number of bits output by the encoding stage.

[0088] When entropy-based reversible encoders, such as arithmetic encoders, are used, rate loss typically encourages the encoder NN output to have low entropy. An example of rate loss is shown below, i.e., - Differentiable estimate of entropy, - Sparsification loss, i.e., the loss that causes the output of the encoder NN or the output of the quantization to have many zeros. An example is the L0 norm, L1 norm, and L1 norm divided by the L2 norm. - Cross-entropy loss applied to the output of the probabilistic model. The probabilistic model may be a neural network (NN) used to estimate the probability of the next symbol encoded by an arithmetic encoder. That is the case.

[0089] As a weighted sum, one or more of the reconstruction losses or one or more of the rate losses may be used. Different loss terms may be weighted using different weights, which determine how the final system performs in the rate-distortion loss term. For example, if reconstruction losses are given a greater weight than rate losses, the system may learn to reconstruct with less compression but with greater accuracy (as measured by a metric correlated with the reconstruction losses). These weights may be considered hyperparameters of the training session, which may be set manually by the person designing the training session, or they may be set automatically, for example, by grid search or using an additional neural network.

[0090] As shown in Figure 4, a neural network-based end-to-end learning video coding system may include an encoder 401, a quantizer 402, a probabilistic model 403, an entropy codec 420 (e.g., an arithmetic encoder 405 / arithmetic decoder 406), an inverse quantizer 407, and a decoder 408. The encoder 401 and decoder 408 may be two neural networks or may consist mainly of neural network components. The probabilistic model 403 may also consist mainly of neural network components. The quantizer 402, inverse quantizer 407, and entropy codec 420 do not have to be based on neural network components, but may potentially include them.

[0091] On the encoder side, encoder component 401 receives video x409 as input and converts the video from its original signal space to a latent representation which may contain a more compressible representation of the input. For an input image, the latent representation may be a three-dimensional tensor, where two dimensions represent the vertical and horizontal spatial dimensions, and the third dimension represents the "channels" that contain information at specific locations. If the input image is a 128×128×3RGB image (128 horizontal pixels, 128 vertical pixels, and three channels for red, green, and blue color components), and the encoder downsamples the input tensor by a factor of two, expanding the channel dimension to 32 channels, then the latent representation is a 64×64×32-dimensional (or "shape") tensor (i.e., 64 horizontal elements, 64 vertical elements, and 32 channels). It should be noted that the order of different dimensions may vary depending on the rules used, and in some cases, for input images, the channel dimension may be the first dimension; therefore, for the example above, the shape of the input tensor may be represented as 3 × 128 × 128 instead of 128 × 128 × 3. For input video (instead of input images), another dimension of the input tensor may be used to represent temporal information.

[0092] The quantizer component 402 quantizes the latent representation into discrete values ​​given a predefined set of quantization levels. The probabilistic model 403 and the arithmetic codec component 420 work together to perform lossless compression on the quantized latent representation and generate a bitstream to be sent to the decoder. Given a symbol to be encoded in the bitstream, the probabilistic model 403 estimates the probability distribution of all possible values ​​for that symbol based on the context constructed from available information in the current encoding / decoding state, such as already encoded / decoded data. The arithmetic encoder 405 then uses the estimated probability distribution to encode the input symbol into the bitstream.

[0093] On the decoder side, the opposite operation is performed. The arithmetic decoder 406 and the probabilistic model 403 first decode symbols from the bitstream to reconstruct the quantized latent representation. Then, the inverse quantizer 407 reconstructs the latent representation into continuous values ​​and passes it to the decoder 408 to reconstruct the input video / image. Note that in this system, the probabilistic model 403 is shared between the encoding system and the decoding system. In practice, this means that a copy of the probabilistic model 403 is used on the encoder side and another exact copy is used on the decoder side.

[0094] In this system, the encoder 401, the probabilistic model 403, and the decoder 408 may be based on a deep neural network. The system may be trained in an end-to-end manner by minimizing the following rate-distortion loss function. L = D + λR

[0095] Here, D is the distortion loss term, R is the rate loss term, and λ is a weight that controls the balance between the two losses. The distortion loss term may be the mean squared error (MSE), structural similarity (SSIM), or other metric that evaluates the quality of the reconstructed video. Multiple distortion losses may be used and integrated into D, such as a weighted sum of MSE and SSIM. The rate loss term is typically the estimated entropy of the quantized latent representation, which indicates the number of bits required to represent the encoded symbol, e.g., bits per pixel (bpp).

[0096] For lossless video / image compression, the system may consist only of the probabilistic model 403 and the arithmetic encoder / decoder 405, 406. Since distortion loss is always zero (i.e., no loss of information), the system loss function consists only of rate loss.

[0097] Since humans are considered to be the end users, i.e., the ones consuming / viewing the decoded images, reducing distortion in image and video compression is often intended to improve human perceptual quality. In recent years, with the advent of machine learning, particularly deep learning, there has been an increase in the number of computers (i.e., autonomous agents) that can analyze data independently of humans and even make decisions based on the analysis results without human intervention. Examples of such analysis include object detection, scene classification, semantic segmentation, video event detection, anomaly detection, and pedestrian tracking. Exemplary use cases and applications include autonomous vehicles, video surveillance cameras and public safety, smart sensor networks, smart TVs and smart advertising, person re-identification, smart traffic monitoring, and drones. When decoded data is consumed by computers, different quality metrics will be used instead of human perceptual quality. Also, dedicated algorithms for compressing and decompressing data for computer consumption may differ from dedicated algorithms for compressing and decompressing data for human consumption. The set of tools and concepts for compressing and decompressing data for computer consumption is referred to here as Video Coding for Machines (VCM).

[0098] VCM (Video Code Maker) relates to the encoding of video streams to enable consumption by a computer. A computer refers to an algorithm, system, or other entity that takes video or images or signals derived from video or images as input to perform a task, or a device equipped with such an algorithm, system, or other entity. Examples of computers may include neural network-based classifiers, mobile phones, autonomous vehicles, robots, and intelligent devices that may have some degree of autonomy or may run intelligent algorithms to process a decoded stream beyond reconstructing the original input stream.

[0099] The computer may perform one or more tasks on the decoded stream. Examples of tasks are as follows: - Classification: Classify an image or video into one or more defined categories. The output of the classification task may be a set of detected categories, also known as classes or labels. The output may also include the probability and confidence level for each defined category. - Object detection: Detects one or more objects in a given image or video. The output of the object detection task may be the bounding boxes of the detected objects and their associated classes. The output may also include the probability and confidence level of each detected object. - Instance segmentation: Identifying one or more objects in an image or video at the pixel level. The output of the instance segmentation task may be a binary mask image of the detected objects, or other representations of the binary mask image, such as closed contours. The output may also include the probability and confidence level of each object for each pixel. - Semantic segmentation: Assigns pixels in an image or video to one or more defined semantic categories. The output of a semantic segmentation task may be a binary mask image of the assigned categories, or other representations of the binary mask image, such as closed contours. The output may also include the probability and confidence level for each semantic category for each pixel. - Object Tracking: Track one or more objects within a video sequence. The output of the object tracking task may also include the frame index, object ID, object bounding box, probability, and confidence level for each tracked object. - Caption generation: Generates one or more short text descriptions for an input image or video. The output of the caption generation task may be one or more short text sequences. - Human pose estimation: Estimates the positions of key points, such as wrists, elbows, and knees, from the bodies of one or more people in a video image. The output of human pose estimation includes a set of positions for each key point of the human body detected in the input image or video. - Human Action Recognition: Recognizes the actions of one or more people in an input image or video, such as walking, talking, or shaking hands. The output of human action recognition may be a predefined set of actions, probabilities, and confidence levels for each identified action. - Anomaly detection: Detects anomalous objects or events from the input image or video. The anomaly detection output may include the location of the detected anomalous object, or the frame segment in the input video in which the anomalous event was detected. Includes.

[0100] The receiving device may have multiple "computers" or task neural networks (Task-NNs). These multiple computers may be used in a certain combination determined, for example, by an orchestrator subsystem. The multiple computers may be used sequentially, for example, based on the output of previously used computers, and / or in parallel. For example, compressed and then decompressed video may be analyzed by one computer (NN) for detecting pedestrians, another computer (another NN) for detecting cars, and yet another computer (another NN) for estimating the depth of all pixels in the frame.

[0101] In this description, “task computer,” “computer,” and “task neural network” are interchangeable references, and such references mean any process or algorithm that analyzes or processes (learns from or does not learn from) data for a particular task. The remainder of this description may specify in further detail other assumptions about computers that are possible in this disclosure. The terms “receiver” or “decoder” may also encompass one or more computers, and these one or more computers are used to refer to another physical or abstract entity or device, “encoder-side device,” which may perform on a video representation that is encoded, encoded, and ultimately decoded by the “encoder-side device.”

[0102] The encoded video data may, for example, be stored as a file on a memory device. The stored file may later be provided to another device. Alternatively, the encoded video data may be streamed from one device to another.

[0103] Figure 5 is a schematic diagram of a video coding pipeline to a computer. The VCM encoder 502 encodes the input video into a bitstream 504. To evaluate the size of the bitstream, the bitrate 506 may be calculated from the bitstream 504 508. The VCM decoder 510 decodes the bitstream output by the VCM encoder 502. In Figure 5, the output of the VCM decoder 510 is referred to as “decoded data to the computer” 512. This data may be thought of as the decoded or reconstructed video. However, in some embodiments of this pipeline, this data may not have the same or similar characteristics as the original video input to the VCM encoder 502. For example, this data may not be easily understandable to humans when rendering the data on a screen. The output of the VCM decoder is then input to one or more task neural networks 514. In the figure, there are three exemplary task-NNs and one unspecified (task-NN X) to illustrate that there may be any number of task-NNs 514. The objective of VCM is to obtain a low-bitrate representation of the input video while ensuring that the task-NN still performs well in terms of the evaluation metric 516 associated with each task.

[0104] One possible approach to achieving video coding for computers is an end-to-end learning approach. In this approach, the VCM encoder and VCM decoder are primarily composed of neural networks. Figure 6 illustrates an example pipeline for the end-to-end learning approach. The video is input to the neural network encoder 601. The output of the neural network encoder 601 is input to a reversible encoder 602, such as an arithmetic encoder, which outputs a bitstream 604. The output of the neural network encoder 601 may also be input to a probabilistic model 603, which provides the reversible encoder 602 with an estimate of the probability of the next symbol to be encoded by the reversible encoder 602. The probabilistic model 603 may be learned by machine learning techniques, for example, it may be a neural network. On the decoder side, the bitstream 604 is input to a reversible decoder 605, such as an arithmetic decoder, and its output is input to the neural network decoder 606. The output of the reversible decoder 605 is input to the probabilistic model 603, which provides the reversible decoder 605 with an estimate of the probability of the next symbol to be decoded by the reversible decoder 605. The output of the neural network decoder 606 is the decoded data for computer 607, which may be input to one or more tasks - NN608.

[0105] Figure 7 illustrates an example of how an end-to-end learning system may be trained for the purpose of video coding for a computer. For simplicity, only one task—NN707—is described. A rate loss 705 may be calculated from the output of a probabilistic model 703. The rate loss 705 may provide an approximation of the bitrate required to encode the input video data. A task loss 710 may be calculated 709 from the output 708 of task-NN707.

[0106] The rate loss 705 and task loss 710 may then be used to train a neural network 711 for use within a system such as a neural network encoder 701, a probabilistic model 703, or a neural network decoder 706. Training may be performed by first calculating the gradient of each loss with respect to the trainable neural network parameters that contribute to or influence the calculation of that loss. The gradients are then used by an optimization method such as Adam to update the trainable parameters of the neural network.

[0107] Computer tasks may be performed on the decoder side (instead of the encoder side) for several reasons, such as the encoder-side device lacking the capability (computation, power, memory) to run the neural network that performs these tasks, or because some aspect or performance of the task neural network has been modified or improved by the time the decoder-side device requires the task results (e.g., different or additional semantic classes, a better neural network architecture). Furthermore, there may be a need for customization to perform these machine learning tasks, as different clients will each run different neural networks.

[0108] A dense split attention (DSA) block is an attention block that estimates one or more attention maps and applies one or more attention maps to one or more data tensors. - The attention map may be a vector, matrix, or tensor. For example, the attention map may have values ​​within the range [0,1]. - One or more data tensors may include one or more input tensors to an attention block, and / or one or more feature maps extracted within the attention block, and / or one or more feature maps extracted outside the attention block. - The application of one or more attention maps to one or more data tensors may include, for example, multiplying one or more attention maps by one or more data tensors by using element-wise multiplication. Other operations may also be considered.

[0109] Figure 8 illustrates an example of a DSA block, where one type of NN layer is a ResBlock containing an NN layer.

[0110] A DSA block may include extracting features from its input based on at least one or more initial NN layers, then splitting the extracted features across the channel axis to obtain two split features, summing the split features, performing a global average operation on the summed features, processing the output of the global average operation based on at least one or more NN layers, inputting the result of this processing into a softmax operation, splitting the result of the softmax operation across the channel axis to obtain two attention tensors, multiplying the two attention tensors by the two previously determined split features to obtain two attention split features, summing the two attention split features, concatenating the summed attention split features with features determined based on at least one or more initial NN layers, processing the result of the concatenation by at least one or more NN layers, and summing the output of this processing with the input of the DSA block to obtain the output of the DSA block. Here, the global averaging operation may be a global pooling (average pooling) operation that calculates the average value for a patch of feature maps. This allows for the aggregation of spatial information from feature maps into a single channel, and enables the utilization of feature relationships between channels.

[0111] In video coding, video and image samples are typically encoded using color representations such as YUV or YCbCr, which consist of one luminance (lumen) channel, also denoted as Y, and two chrominance (chromen) channels, also denoted as U, V, or Cb, Cr. In these cases, the luminance channel, which largely represents the illumination of the scene, is usually coded at a certain resolution, while the chrominance channel, which represents the difference between certain color components, is often coded at a second resolution lower than the resolution of the luminance signal. The purpose of this type of difference representation is to eliminate the correlation between color components and to allow for more efficient data compression.

[0112] As described above, neural networks (NNs) are used in image and video compression by replacing one or more components of conventional codecs, as shown in Figure 1, or by utilizing end-to-end learning compression, as shown in Figures 2 and 3. However, NN-based video coding approaches, including solutions based on end-to-end learning compression, are known to be difficult to optimize in terms of rate distortion performance.

[0113] This document presents methods for improving the rate distortion performance of neural network-based video codecs.

[0114] Figure 9a shows an encoding method according to one embodiment, the method comprising: receiving input data including a first ground truth (900); providing the first ground truth data to a first encoder (902), wherein the ground truth data includes at least a first portion of the input data; obtaining initial encoded input data from the output of the first encoder (904); providing the initial encoded input data to a first decoder (906); obtaining initial reconstruction of the input data from the output of the first decoder (908); and providing at least the first ground truth data and the initial reconstruction to a first ground truth data and initial reconstruction. The method includes determining the residuals (910), providing second ground truth data including at least the residuals to a second encoder (912), wherein the second encoder comprises at least a first sub-encoder and a probabilistic model, providing (912), providing the second ground truth data to the first sub-encoder (914), providing the output of the first sub-encoder to the probabilistic model (916), obtaining one or more first probabilities from the output of the probabilistic model (918), and obtaining encoded residual information from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities (920).

[0115] Figure 9b shows a decoding method according to one embodiment, the method comprising: receiving (930) an initial reconstruction of the input data of an encoder, receiving (930) the input data including encoded residuals, wherein the encoded residuals are determined based on at least the ground truth data of the encoder, the initial reconstruction of the input data and an adaptive signal to the decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit and a probabilistic model, and obtaining (932) an entropy-decoded residual based on at least the encoded residuals and one or more probabilities of the probabilistic model, the entropy-decoded The coded residual is represented as a plurality of decoded latent tensor elements, and the process includes obtaining the residual (932), providing the entropy-decoded residual to a first sub-decoder (934), providing the adaptive signal to a syntax generation unit to generate one or more syntaxes based on the adaptive signal (936), adapting the decoding of the residual in the first sub-decoder based on the one or more syntaxes (938), obtaining a reconstructed residual from the output of the first sub-decoder (940), and obtaining a final reconstruction of the input data by combining the reconstructed residual with an initial reconstruction of the input data (942).

[0116] Therefore, the method reflects the encoding and decoding modes in a codec comprising two separate codecs for encoding input data, the first codec comprising a first encoder and a first decoder, and the second codec comprising a second encoder and a second decoder. These two codecs may be referred to as the first-step codec and the second-step codec, respectively. The encoder of the first-step codec may receive ground truth data as input and output a first bitstream, the decoder of the first-step codec decodes the first bitstream and outputs an initial reconstruction of the input data (ground truth data). Internally, the decoder of the first-step codec obtains an adaptive signal, such as a latent tensor or an inversely quantized latent tensor, based at least on the first bitstream. The residual is then calculated based on the ground truth data and the initial reconstruction of the ground truth data. The encoder of the second-step codec receives the residual and outputs a second bitstream representing the encoded residual information.

[0117] The decoder of the second-step codec receives a second bitstream (i.e., encoded residual information) and an adaptive signal, such as an inversely quantized latent tensor, from the first-step codec as input. The adaptive signal, such as an inversely quantized latent tensor, from the first-step codec may be input to the decoder of the second-step codec to adapt the decoding of the residual information. Thus, the latent tensor or inversely quantized latent tensor may also be referred to as the adaptive signal or embedding syntax. Finally, the reconstructed residual from the second-step codec is combined with the initial reconstruction of the ground truth data from the first-step codec to obtain the final reconstruction of the ground truth. The sum of the bitstream output by the encoder of the first-step codec (i.e., the first bitstream) and the bitstream output by the encoder of the second-step codec (i.e., the second bitstream) represents the encoded ground truth.

[0118] Using the adaptive signal, also known as the embedding syntax (e.g., a latent tensor (inversely quantized)), determined by the codec in the first step, the decoder of the codec in the second step may be adapted based on one or more aspects or features of the input data, which may improve the rate distortion performance of the codec in the second step. Thus, the rate distortion performance of the entire codec (the combination of the codec in the first step and the codec in the second step) may be improved.

[0119] According to one embodiment, the first and second encoders and decoders belong to an end-to-end learning intra-frame codec or an end-to-end learning image codec.

[0120] Therefore, using learning-based video compression, particularly in the context of end-to-end learning-based intra-frame codecs, it is possible to improve rate distortion performance. However, it should be noted that the method and related embodiments are also applicable to end-to-end learning-based inter-frame codecs.

[0121] According to one embodiment, the learning intraframe codec is trained in an end-to-end manner by minimizing D+λR, where D is the distortion loss term, R is the rate loss term, and λ is a weight that controls the balance between the losses.

[0122] The distortion loss term may be calculated based on at least the distortion function, ground truth data, and the output of the decoder included in the trained intraframe codec. The rate loss term may be calculated based on at least an estimate of the bitrate (e.g., the size of the bitstream in bits) of the bitstream output of the encoder included in the trained intraframe codec.

[0123] This optimization process results in a so-called rate-distortion trade-off, where a balance is found between distortion D and rate loss R. Rate loss may indicate the bitrate of the encoded image, and distortion is defined as follows: - Mean squared error (MSE), - Multiscale structural similarity (MS-SSIM), - Multiple strain losses, such as the weighted sum of MSE and MS-SSIM, - Other metrics for evaluating the quality of reconstructed images You may also indicate pixel fidelity distortions such as those mentioned above.

[0124] In one embodiment, the distortion may relate to the performance of one or more computer analysis tasks, or the estimated performance of one or more computer analysis tasks, the one or more computer analysis tasks may include object detection, image segmentation, instance segmentation, etc. In one example, the estimated performance of one or more computer analysis tasks may include distortion calculated based on at least a first set of features extracted from the output of a decoder and a second set of features extracted from the respective ground truth data, the first set of features and the second set of features being output by one or more layers of a pre-trained feature extraction neural network.

[0125] According to one embodiment, training of a learning-type intraframe codec is performed jointly with respect to distortion loss D and rate loss R.

[0126] According to another embodiment, training of a learning intraframe codec is performed in two alternating phases, where only the distortion loss D is used in the first of the two alternating phases, and only the rate loss R is used in the second of the two alternating phases.

[0127] Various embodiments of the implementation options are described below. Note that the terms “intraframe,” “frame,” and “image” may be used interchangeably. These terms may refer to input data to an end-to-end (e2e) learning intraframe codec. For brevity, in at least some embodiments, an image is considered a data type. However, the embodiments described herein and their underlying principles may be extended to other types of data, such as video and audio. Also, in at least some of the embodiments, YUV is considered the input color format. However, the embodiments may be similarly extended to other color formats, such as RGB. In the YUV format, “Y” represents the lightness or “luma” value, and “U” and “V” represent the color or “chroma” value. In one example, the input image may be an image in YUV4:4:4 color format and may be represented as a three-dimensional array (or tensor) having a size of 256 × 256 × 3, with a horizontal size of 256 pixels and a vertical size of 256 pixels, and the three channels corresponding to the Y, U, and V components, respectively. In another example, the input image may be in YUV4:2:0 color format and may be represented by a combination of a 256x256 matrix for the lumens and a 128x128x2 two-dimensional array (or tensor) for the chromens.

[0128] Generally, an end-to-end (e2e) learning intraframe codec is analogous to an end-to-end (e2e) learning image compression system. Typically, it may consist of an image encoder, a quantizer, a probabilistic model, an entropy codec (e.g., an arithmetic encoder / decoder), an inverse quantizer, and an image decoder. The image encoder, probabilistic model, and image decoder may primarily consist of neural network components. The quantizer, entropy codec, and inverse quantizer are typically not based on neural network components, but may still contain them.

[0129] An end-to-end learning intra-frame codec may be used as part of a video codec, and the intra-frame codec may code each of one or more first frames of a video independently of any other frames, and another codec called an inter-frame codec may code each of one or more second frames of a video based on at least one or more other frames, for example, based on at least data obtained from one or more first frames. The embodiments described herein refer to an end-to-end learning intra-frame codec, but at least some embodiments described herein may also apply to an end-to-end learning inter-frame codec used for video compression, or an end-to-end learning video codec that compresses both intra-frames and inter-frames.

[0130] First step codec The input to the codec in the first step may include at least ground truth data (e.g., blocks or the entire image), and at least a portion of the ground truth data will be encoded / compressed. The input may also include one or more additional data such as block / image resolution. Examples of input ground truth data are as follows: - Ground Truth. For example, the ground truth may be an image in the YUV4:4:4 color format represented as a 256×256×3 multidimensional array or tensor, with a horizontal size of 256 pixels, a vertical size of 256 pixels, and three channels corresponding to the Y, U, and V components, respectively. In another example, the ground truth may be an image in the YUV4:2:0 color format represented by a combination of a 256×256 matrix for the lumens component and a 128×128×2 two-dimensional array (or tensor) for the chromens component. In yet another example, the ground truth may include only the lumens component, represented as a one-dimensional array (or tensor) with a size of 256×256×1. In yet another example, the ground truth may include only the chromens component, represented as a two-dimensional array (or tensor) with a size of 128×128×2. - Partial ground truth. For example, the input data may be the result of multiplying the ground truth by a mask or the output of a Gaussian filter.

[0131] Figure 10 illustrates an exemplary implementation of the first step codec according to an embodiment. The input data x is ground truth, which here includes both the lumen and chroma components of the image, and h × w × 3 represents the size of x, having height h, width w, and number of channels 3. The first step codec is used to encode the input data; that is, the bitstream output by the encoder of the first step codec represents the initially encoded input data, and the output of the decoder of the first step codec represents the initial reconstruction of the input data.

[0132] The encoder 1000 of the codec in the first step may comprise a neural encoder 1002, a quantizer, a probabilistic model 1004, and an entropy encoder 1006. In the figure, the quantizer is not shown for simplicity. The neural encoder 1002 may comprise a first convolutional layer ("Conv5×5,48,1", where conv represents convolution, 5×5 is the kernel size, 48 is the number of output channels, and 1 is the stride value), followed by a nonlinear activation function ReLU, followed by a first DSA block, followed by a second convolutional layer, followed by a second DSA block, followed by a third convolutional layer, followed by a third DSA block, followed by a fourth convolutional layer, followed by a fourth DSA block, followed by a fifth convolutional layer. The neural encoder 1002 outputs a latent tensor which may be quantized. A latent tensor or quantized latent tensor is input to a probabilistic model 1004, the dimensions of which may be h / / 16 × w / / 16 × 128, where h / / 16 indicates the height, w / / 16 indicates the width, and 128 indicates the number of channels in the latent tensor. The probabilistic model outputs an estimate of the probability of each element of the (quantized) latent tensor. The probabilistic model may be learned from the data using machine learning techniques, for example, the probabilistic model may be a neural network. On the encoder side, the output of the probabilistic model 1004 is used as one of the inputs to the entropy encoder 1006. The entropy encoder may be an arithmetic encoder. The entropy encoder receives at least the (quantized) latent tensor and the output of the probabilistic model and outputs a bitstream 1008.

[0133] The decoder 1010 of the codec in the first step may comprise an entropy decoder 1012, a probabilistic model 1014, an inverse quantizer, and a neural decoder 1016. In the figure, the inverse quantizer is not shown for simplification. It is assumed that the probabilistic model 1014 and the probabilistic model 1004 are identical or substantially identical, for example, they may be copies of the same probabilistic model. The entropy decoder 1012 may be an arithmetic decoder. The entropy decoder receives at least the output of the bitstream 1008 and the probabilistic model 1014 and outputs a (quantized) decoded latent tensor. The decoded latent tensor may then undergo inverse quantization. The decoded latent tensor or the inversely quantized decoded latent tensor is then input to the neural decoder 1016. The neural decoder may include a first transposed convolutional layer ("UpConv5×5,384,2", where UpConv refers to transposed convolution, 5×5 is the kernel size, 384 is the number of output channels, and 2 is the stride value), a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a fourth DSA block, a nonlinear activation function ReLU, and a convolutional layer.

[0134] Output of neural decoder 1016

[0135]

number

[0136] Codec in the second step The codec in the second step is used to encode the residuals of the ground truth. The residuals are used for the initial reconstruction from the ground truth data (e.g., blocks or the entire image) and the codec in the first step.

[0137]

number

[0138]

number

[0139]

number

[0140]

number

[0141] According to one embodiment, the input data to the codec in the second step may include the output of the codec in the first step. Therefore, the input data is initially reconstructed from the decoder of the codec in the first step.

[0142]

number

[0143]

number

[0144] The residual may be calculated, for example, as the sample-by-sample difference between the ground truth data and the initial reconstruction from the codec in the first step.

[0145] Figure 11 illustrates an exemplary implementation of the codec in the second step according to the embodiment. The input data r is ground truth and initial reconstruction from the codec in the first step.

[0146]

number

[0147] The encoder 1100 of the codec in the second step may comprise a neural encoder (i.e., a first sub-encoder) 1102, a quantizer (not shown), a probabilistic model 1104, and an entropy encoder 1106. Similar to the neural encoder 1002 of the codec in the first step, the neural encoder 1102 of the codec in the second step may also comprise a first convolutional layer, a nonlinear activation function ReLU, a first DSA block, a second convolutional layer, a second DSA block, a third convolutional layer, a third DSA block, a fourth convolutional layer, a fourth DSA block, and a fifth convolutional layer. The dimensions of the latent tensor are h / / 16 × w / / 16 × 128, the height is h / / 16, the width is w / / 16, and the number of channels is 128.

[0148] The (quantized) latent tensor output by the encoder 1102 of the codec in the second step may be input to a probabilistic model. The probabilistic model 1104 may be a neural network. The probabilistic model 1104 outputs estimates of the probabilities of one or more elements of the latent tensor. On the encoder side, the output of the probabilistic model 1104 is used as one of the inputs to the entropy encoder 1106. The entropy encoder may be an arithmetic encoder. The entropy encoder 1106 receives at least the (quantized) latent tensor and the output of the probabilistic model 1106 and outputs a bitstream 1108 representing the encoded residual information.

[0149] The decoder 1110 of the codec in the second step may comprise an entropy decoder 1112, a probabilistic model 1114, an inverse quantizer (not shown in the figure), a neural decoder 1116, and a syntax generation network 1118. The entropy decoder 1112 may be an arithmetic decoder. The entropy decoder 1112 receives at least the bitstream 1108 and the output of the probabilistic model 1114 and outputs a decoded (quantized) latent tensor. The probabilistic model 1114 may need to be the same or substantially the same probabilistic model 1104 available on the encoder side. The decoded (quantized) latent tensor may undergo inverse quantization. After inverse quantization, the inversely quantized decoded latent tensor is input to the neural decoder 1116. The neural decoder 1116 may comprise a first transposed convolutional layer, a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a nonlinear activation function ReLU, and a convolutional layer.

[0150] The syntax generation network 1118 may receive an adaptive signal z as input and generate one or more syntaxes. The syntax may be, for example, a tensor. According to one embodiment, the adaptive signal input to the syntax generation network is obtained based at least on a first bitstream, i.e., a bitstream input to the decoder of the codec in the first step. According to one embodiment, the adaptive signal input to the syntax generation network is an inversely quantized latent tensor from the codec in the first step.

[0151] The syntax generation network 1118 may comprise a first transposed convolutional layer (UpConv), a first DSA block, a second transposed convolutional layer, a second DSA block, a third transposed convolutional layer, a third DSA block, a fourth transposed convolutional layer, a nonlinear activation function ReLU, and a fourth DSA block. Each DSA block may be followed by one generated syntax cd1, cd2, cd3, and cd4, respectively. Subsequently, the generated syntax cd1, cd2, cd3, and cd4 may be injected into the first, second, third, and fourth spatial attention blocks of the neural decoder 1116, respectively.

[0152] Figure 12 shows an example of a spatial attention block. In each spatial attention block, the feature extracted from the neural decoder's DSA block may undergo channel pooling operations, convolutional layers, and a nonlinear activation function sigmoid. The output may then be multiplied by the injected syntax cd, and the feature extracted from the neural decoder's DSA block may be added to it to obtain the refined feature.

[0153]

number

[0154] Using syntax injection, the output of the neural decoder 1116 is a reconstruction of the residual information, which may have a size of h × w × 3. Finally, the reconstructed residual is processed by the combiner 1120 into the initial reconstruction.

[0155]

number

[0156] According to one embodiment, the output of the second decoder is the final reconstruction of the input data. Therefore, the initial reconstruction

[0157]

number

[0158] According to one embodiment, the adaptive signal input to the syntax generation network is an initial reconstruction of ground truth data obtained from the output of the codec in the first step.

[0159] According to one embodiment, the intraframe codec includes an auxiliary encoder on the decoder side, and the adaptive signal input to the syntax generation network is the output of the auxiliary encoder.

[0160] Since the codec in the first step contains sufficient spatial information of the input data, the generated syntax may provide spatial information to the neural decoder of the codec in the second step, allowing it to adapt the reconstruction of the residual portion based on the content.

[0161] The methods presented above relating to encoding and decoding modes may be implemented in the respective encoding and decoding devices. One embodiment of the device includes means for receiving input data including first ground truth data; means for providing the first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; means for obtaining initial encoded input data from the output of the first encoder; means for providing the initial encoded input data to a first decoder; means for obtaining initial reconstruction of the input data from the output of the first decoder; and means for determining residuals based at least on the first ground truth data and the initial reconstruction of the input data. The device comprises a stage and means for providing a second ground truth data including at least residuals to a second encoder, the second encoder comprising at least a first sub-encoder for the second ground truth data and a probabilistic model, means for providing the second ground truth data to the first sub-encoder, means for providing the output of the first sub-encoder to the probabilistic model, means for obtaining one or more first probabilities from the output of the probabilistic model, and means for obtaining encoded residual information from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities.

[0162] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

[0163] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder includes means for converting input data into a plurality of latent tensor elements, the probabilistic model includes means for estimating the probability of one or more of the plurality of latent tensor elements, and the entropy encoder includes means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probability of one or more of the plurality of latent tensor elements.

[0164] Several embodiments are described below with reference to Figures 13 to 16. Embodiments with reference to Figure 17 have been described above. In Figures 13, 14, 15, 16, and / or 17, the terms “first neural encoder,” “first neural decoder,” “first probabilistic model,” “second neural encoder,” “second neural decoder,” and “second probabilistic model” may also be referred to as “neural encoder for the first codec,” “neural decoder for the first codec,” “probabilistic model for the first codec,” “first subencoder,” “first subdecoder,” and “probabilistic model for the second codec,” respectively.

[0165] According to one embodiment, the device includes means for providing an initial reconstruction of input data from the output of the first decoder, or data obtained from the initial reconstruction of input data, to the input of the first sub-decoder of the second decoder. Here, the first sub-decoder may refer to, for example, the "second neural decoder" shown in Figure 13.

[0166] According to one embodiment, the device includes means for providing an initial reconstruction of input data from the output of a first decoder, or data obtained from the initial reconstruction of input data, to the input of a first sub-encoder of a second codec. Here, the first sub-encoder may refer to, for example, the "second neural encoder" shown in Figure 14.

[0167] According to one embodiment, the device includes means for providing an initial reconstruction of input data from the output of the first decoder, or data obtained from the initial reconstruction of input data, to the input of a probabilistic model used in the encoder of the second codec and to the input of a probabilistic model used in the decoder of the second codec. Here, the probabilistic model may refer to, for example, the "second probabilistic model" shown in Figure 15.

[0168] According to one embodiment, the device includes means for providing an initial reconstruction of input data from the output of the first decoder, or data obtained from the initial reconstruction of input data, to the input of the first sub-decoder of the second decoder, the input of the first sub-encoder of the second codec, the input of a probabilistic model used in the encoder of the second codec, and the input of a probabilistic model used in the decoder of the second codec. This embodiment is illustrated in Figure 16.

[0169] An apparatus in another embodiment includes means for receiving an encoder's input data including an initial reconstruction of the input data and an encoded residual, the encoded residual being determined based on at least the encoder's ground truth data, the initial reconstruction of the input data and an adaptive signal to a decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit and a probabilistic model; means for obtaining an entropy-decoded residual, based at least the encoded residual and one or more probabilities of the probabilistic model, the entropy-decoded residual being represented as a plurality of decoded latent tensor elements; means for providing the entropy-decoded residual to the first sub-decoder; means for providing the adaptive signal to the syntax generation unit to generate one or more syntaxes based on the adaptive signal; means for adapting the decoding of the residual in the first sub-decoder based on the one or more syntaxes; means for obtaining a reconstructed residual from the output of the first sub-decoder; and means for obtaining a final reconstruction of the input data by combining the reconstructed residual with the initial reconstruction of the input data.

[0170] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, the decoder's probabilistic model includes means for estimating the probability of one or more elements in a plurality of decoded latent tensor elements, the entropy decoder includes means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of one or more elements in the plurality of latent tensor elements, and the neural decoder includes means for converting the plurality of decoded latent tensor elements into reconstructed chroma components.

[0171] According to one embodiment, the first sub-decoder of the decoder is a neural decoder comprising one or more spatial attention units for receiving the one or more syntax and adapting the decoding of the residual components.

[0172] According to one embodiment, the adaptive signal input to the syntax generation network is an inversely quantized latent tensor output by another codec.

[0173] According to one embodiment, the adaptive signal input to the syntax generation network is an initial reconstruction of the input data.

[0174] According to one embodiment, the adaptive signal input to the syntax generation network is the output of an auxiliary encoder.

[0175] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0176] According to one embodiment, the means for combining is configured to omit the initial reconstruction of the input data once the final reconstruction of the input data is obtained.

[0177] In a further embodiment, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory has code thereon, and when executed by the at least one processor, the code provides input data including first ground truth data, provides the first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data, obtains initial encoded input data from the output of the first encoder, provides the initial encoded input data to a first decoder, obtains initial reconstruction of the input data from the output of the first decoder, and first ground The device is made to at least perform the following: determine residuals based on at least the ground truth data and an initial reconstruction of the input data; provide second ground truth data including at least the residuals to a second encoder, wherein the second encoder comprises at least a first sub-encoder and a probabilistic model for the second ground truth data; provide the second ground truth data to the first sub-encoder; provide the output of the first sub-encoder to the probabilistic model; obtain one or more first probabilities from the output of the probabilistic model; and obtain encoded residual information from the output of the second encoder based on the output of the first sub-encoder and one or more probabilities.

[0178] According to one embodiment, the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

[0179] According to one embodiment, the first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder, wherein the neural encoder comprises code configured to cause the device to convert input data into a plurality of latent tensor elements, the probabilistic model comprises code configured to cause the device to estimate the probability of one or more of the plurality of latent tensor elements, and the entropy encoder comprises code configured to cause the device to output a bitstream encoded at least partially on the plurality of latent tensor elements and the probability of one or more of the plurality of latent tensor elements.

[0180] According to one embodiment, the device includes code configured to provide an initial reconstruction of input data from the output of a first decoder, or data obtained from the initial reconstruction of input data, to the input of a first sub-decoder of a second decoder.

[0181] According to one embodiment, the device includes a code configured to provide an initial reconstruction of input data from the output of a first decoder, or data obtained from an initial reconstruction of input data, to the input of a first sub-encoder of a second encoder.

[0182] According to one embodiment, the device includes code configured to provide an initial reconstruction of input data from the output of a first decoder, or data obtained from an initial reconstruction of input data, to the input of a probabilistic model used by a second encoder and to the input of a probabilistic model used by a second decoder.

[0183] In a further embodiment, a device is provided comprising at least one processor and at least one memory, wherein the at least one memory has code thereon, and when executed by the at least one processor, the code receives input data including an initial reconstruction of the encoder's input data and an encoded residual, the encoded residual being determined based at least on the encoder's ground truth data, the initial reconstruction of the input data and an adaptive signal to the decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit and a probabilistic model, and is based at least on the encoded residual and one or more probabilities of the probabilistic model The device is then made to perform at least the following: obtain an entropy-decoded residual, which is represented as a plurality of decoded latent tensor elements; provide the entropy-decoded residual to a first sub-decoder; provide the adaptive signal to a syntax generation unit to generate one or more syntaxes based on the adaptive signal; adapt the decoding of the residual in the first sub-decoder based on the one or more syntaxes; obtain a reconstructed residual from the output of the first sub-decoder; and obtain a final reconstruction of the input data by combining the reconstructed residual with an initial reconstruction of the input data.

[0184] According to one embodiment, the decoder comprises an entropy decoder and a neural decoder, wherein the decoder's probabilistic model includes code configured to cause the device to estimate the probability of one or more elements in a plurality of decoded latent tensor elements, the entropy decoder includes code configured to cause the device to output a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of one or more elements in a plurality of latent tensor elements, and the neural decoder includes code configured to cause the device to convert the plurality of decoded latent tensor elements into reconstructed chroma components.

[0185] According to one embodiment, the first sub-decoder of the decoder is a neural decoder comprising one or more spatial attention units for receiving the one or more syntax and adapting the decoding of the residual components.

[0186] According to one embodiment, the adaptive signal input to the syntax generation network is an inversely quantized latent tensor output by another codec.

[0187] According to one embodiment, the adaptive signal input to the syntax generation network is an initial reconstruction of the input data.

[0188] According to one embodiment, the adaptive signal input to the syntax generation network is the output of an auxiliary encoder.

[0189] According to one embodiment, the decoder belongs to an end-to-end learning intraframe codec.

[0190] According to one embodiment, the device includes code configured to omit the initial reconstruction of the input data once the final reconstruction of the input data is obtained.

[0191] Such a device may include, for example, a functional unit disclosed in any of Figures 1 to 7 and Figures 10 to 12 for carrying out an embodiment.

[0192] Such a device further comprises code stored in the at least one non-temporary memory, which, when executed by the at least one processor, causes the device to implement one or more of the embodiments disclosed herein.

[0193] While exemplary embodiments are described above with reference to the encoder, it should be understood that the resulting bitstream and decoder may have corresponding elements in them. Similarly, while exemplary embodiments are described with reference to the decoder, it should be understood that the encoder may have a structure and / or computer program for generating the bitstream to be decoded by the decoder. For example, some embodiments are described in relation to generating prediction blocks as part of encoding. Embodiments can be similarly realized by generating prediction blocks as part of decoding, but differ in that coding parameters such as horizontal and vertical offsets are decoded from the bitstream rather than being determined by the encoder.

[0194] The embodiments of the present invention described above describe the codec in the context of separate encoder and decoder devices to aid in understanding the associated processes. However, it will be apparent that the device, structure, and operation may be implemented as a single encoder-decoder device / structure / operation. Furthermore, it is possible that the encoder and decoder may share some or all common elements.

[0195] While the above examples illustrate embodiments of the present invention operating within a codec in an electronic device, it will be apparent that the present invention, as defined in the claims, may be implemented as part of any video codec. Therefore, for example, embodiments of the present invention may be implemented in a video codec that performs video coding over a fixed or wired communication path.

[0196] Therefore, the user device may include a video codec such as the video codec described in the embodiments of the present invention above. It will be clear that the term "user device" is intended to cover any suitable type of wireless user device, such as a mobile phone, portable data processing device, or portable web browser.

[0197] Furthermore, elements of the Public Land Mobile Network (PLMN) may also be equipped with video codecs as described above.

[0198] In general, various embodiments of the present invention may be implemented in hardware, dedicated circuitry, software, logic, or any combination thereof. For example, some embodiments may be implemented in hardware, while others may be implemented in firmware or software that may be executed by a controller, microprocessor, or other computing device, but the present invention is not limited to these. Various embodiments of the present invention may be illustrated and described using block diagrams, flowcharts, or some other illustrations, but it should be understood that these blocks, apparatus, systems, techniques, or methods described herein may, as non-limiting examples, be implemented in hardware, software, firmware, dedicated circuitry or logic, general-purpose hardware or controller or other computing device, or any combination thereof.

[0199] Embodiments of the present invention may be implemented by computer software executable by a data processor of a mobile device such as a processor entity, by hardware, or by a combination of software and hardware. Furthermore, it should be noted that any block of the logic flow shown in the figure may represent a program step, or an interconnected logic circuit, block, and function, or a combination of a program step and a logic circuit, block, and function. The software may be stored on a physical medium such as a memory chip or memory block implemented within the processor, a magnetic medium such as a hard disk or floppy disk, or an optical medium such as a DVD and its data variants, or a CD.

[0200] Memory may be of any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as semiconductor-based memory devices, magnetic memory devices and systems, optical memory devices and systems, fixed memory, and removable memory. The data processor may be of any type suitable for the local technical environment and may include, in non-limiting examples, one or more of general-purpose computers, dedicated computers, microprocessors, digital signal processors (DSPs), and processors based on multi-core processor architectures.

[0201] Embodiments of the present invention may be put into practice in various components, such as integrated circuit modules. Designing integrated circuits is generally a highly automated process. Complex and powerful software tools are available to translate logic-level designs into semiconductor circuit designs prepared for etching and forming on semiconductor substrates.

[0202] Programs such as those offered by Synopsys, Inc. in Mountain View, California, and Cadence Design, Inc. in San Jose, California, use well-established design rules, along with a library of pre-stored design modules, to automatically route conductors and place components on a semiconductor chip. Once the design for the semiconductor circuit is complete, the resulting design may be sent to a semiconductor manufacturing facility or "fab" in a standardized electronic format (e.g., Opus or GDSII).

[0203] The foregoing description, using illustrative and non-limiting examples, has provided a sufficient and useful description of exemplary embodiments of the invention. However, various modifications and adaptations will become apparent to those skilled in the art who have read the foregoing description in conjunction with the accompanying drawings and claims. Nevertheless, all such modifications and similar modifications of the teachings of the invention will still fall within the scope of the invention.

Claims

1. A means for receiving input data including first ground truth data, Means for providing the first ground truth data to a first encoder, wherein the ground truth data includes at least a first portion of the input data; Means for obtaining the initial encoded input data from the output of the first encoder, Means for providing the initial encoded input data to the first decoder, Means for obtaining an initial reconstruction of the input data from the output of the first decoder, Means for determining residuals based at least on the first ground truth data and the initial reconstruction of the input data, Means for providing a second ground truth data including at least the residual to a second encoder, wherein the second encoder comprises at least a first sub-encoder and a probabilistic model, Means for providing the second ground truth data to the first sub-encoder, Means for providing the output of the first sub-encoder to the probabilistic model, Means for obtaining one or more first probabilities from the output of the aforementioned probability model, A means for obtaining encoded residual information from the output of the second encoder based on the output of the first sub-encoder and the one or more probabilities. A device equipped with the following features.

2. The apparatus according to claim 1, wherein the first encoder, the first decoder, and the second encoder belong to an end-to-end learning intraframe codec.

3. The first encoder comprises a neural encoder, a probabilistic model, and an entropy encoder. The neural encoder comprises means for converting the input data into a plurality of latent tensor elements, The aforementioned probability model includes means for estimating the probability of one or more of the plurality of latent tensor elements. The apparatus according to claim 1 or 2, wherein the entropy encoder comprises means for outputting a bitstream encoded at least partially based on the plurality of latent tensor elements and the probabilities of one or more of the plurality of latent tensor elements.

4. The apparatus according to any one of claims 1 to 3, further comprising means for providing the initial reconstruction of the input data from the output of the first decoder, or the data obtained from the initial reconstruction of the input data, to the input of the first sub-decoder of the second decoder.

5. The apparatus according to any one of claims 1 to 4, further comprising means for providing the initial reconstruction of the input data from the output of the first decoder, or data obtained from the initial reconstruction of the input data, to the input of the first sub-encoder of the second encoder.

6. The apparatus according to any one of claims 1 to 5, further comprising means for providing the initial reconstruction of the input data from the output of the first decoder, or data obtained from the initial reconstruction of the input data, to the input of the probabilistic model used in the second encoder and to the input of the probabilistic model used in the second decoder.

7. Receiving input data including the first ground truth data, The first ground truth data is provided to the first encoder, wherein the ground truth data includes at least a first portion of the input data. Obtaining the initial encoded input data from the output of the first encoder, The initial encoded input data is provided to the first decoder, Obtaining an initial reconstruction of the input data from the output of the first decoder, The residual is determined based at least on the first ground truth data and the initial reconstruction of the input data, To provide a second ground truth data including at least the residual to a second encoder, wherein the second encoder comprises at least a first sub-encoder and a probabilistic model. The second ground truth data is provided to the first sub-encoder, The output of the first sub-encoder is provided to the probabilistic model, Obtaining one or more first probabilities from the output of the aforementioned probability model, Based on the output of the first sub-encoder and the one or more probabilities, encoded residual information is obtained from the output of the second encoder. Methods that include...

8. Means for receiving an encoder, including an initial reconstruction of the input data and an encoded residual, wherein the encoded residual is determined based at least on the ground truth data of the encoder, the initial reconstruction of the input data, and an adaptive signal to a decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit, and a probabilistic model, A means for obtaining an entropy-decoded residual, based at least on the encoded residual and one or more probabilities of the probabilistic model, wherein the entropy-decoded residual is represented as a plurality of decoded latent tensor elements; Means for providing the entropy-decoded residual to the first sub-decoder, Means for providing the adaptive signal to the syntax generation unit in order to generate one or more syntaxes based on the adaptive signal, Means for adapting the decoding of the residual in the first sub-decoder based on one or more syntaxes, A means for obtaining the reconstructed residual from the output of the first sub-decoder, A means for obtaining a final reconstruction of the input data by combining the reconstructed residual with the initial reconstruction of the input data. A device equipped with the following features.

9. The decoder comprises an entropy decoder and a neural decoder. The probability model of the decoder comprises means for estimating the probability of one or more elements among the plurality of decoded latent tensor elements, The entropy decoder comprises means for outputting a plurality of decoded latent tensor elements based at least partially on the input data and the probabilities of one or more elements among the plurality of latent tensor elements, The apparatus according to claim 8, wherein the neural decoder comprises means for converting the plurality of decoded latent tensor elements into the reconstructed chroma components.

10. The apparatus according to claim 8 or 9, wherein the first sub-decoder of the decoder is a neural decoder comprising one or more spatial attention units for receiving the one or more syntaxes and adapting the decoding of the residual components.

11. The apparatus according to any one of claims 8 to 10, wherein the adaptive signal input to the syntax generation network is an inversely quantized latent tensor output by another codec.

12. The apparatus according to any one of claims 8 to 10, wherein the adaptive signal input to the syntax generation network is the initial reconstruction of the input data.

13. The apparatus according to any one of claims 8 to 10, wherein the adaptive signal input to the syntax generation network is the output of an auxiliary encoder.

14. The device according to any one of claims 8 to 13, wherein the decoder belongs to an end-to-end learning intraframe codec.

15. The apparatus according to any one of claims 8 to 14, wherein the means for combining is configured to omit the initial reconstruction of the input data once the final reconstruction of the input data is obtained.

16. The receiving of an encoder involves initial reconstruction of the input data, receiving the input data including encoded residuals, wherein the encoded residuals are determined based at least on the ground truth data of the encoder, the initial reconstruction of the input data, and an adaptive signal to the decoder, the decoder comprising at least a first sub-decoder, a syntax generation unit, and a probabilistic model. Obtaining an entropy-decoded residual based on at least the encoded residual component and one or more probabilities of the probabilistic model, wherein the entropy-decoded residual is represented as a plurality of decoded latent tensor elements, The entropy-decoded residual is provided to the first sub-decoder, To provide the adaptive signal to the syntax generation unit in order to generate one or more syntaxes based on the adaptive signal, Adapting the decoding of the residual in the first sub-decoder based on one or more of the above syntaxes, Obtain the reconstructed residual from the output of the first sub-decoder, The reconstructed residual is combined with the initial reconstruction of the input data to obtain the final reconstruction of the input data. Methods that include...

Images