Conditional image compression

Abstract

To provide a technique relating to conditional coding of an image component.SOLUTION: There is provided a method of encoding at least a portion of an image, including the steps of: encoding a primary component of the image independently of at least one secondary component; and using information from the primary component to encode the at least one secondary component of the image. Further, there is provided a method for encoding at least a portion of the image, including the steps of: providing a residual including a primary residual component about the primary component of the image and at least secondary residual component about the at least one secondary component of the image different from the primary component; encoding the primary residual component independently of the at least one secondary residual component; and using information from the primary residual component to encode the at least one secondary residual component.SELECTED DRAWING: Figure 14

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE This disclosure relates generally to the field of image and video coding, and more particularly to image and video coding with conditional image compression. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat and video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders in security applications.

[0003] The amount of video data required to represent even a relatively short video can be substantial, which can present challenges when the data is streamed or otherwise communicated over communications networks with limited bandwidth capacity. Thus, video data is typically compressed before being communicated over modern telecommunications networks. Video size can also be an issue when the video is stored on a storage device, where memory resources may be limited. Video compression devices often use software and / or hardware at the source to code the video data before transmission or storage, thereby reducing the amount of data needed to represent a digital video image. The compressed data is then received at the destination by a video decompression device, which decodes the video data. Compression techniques are also suitably applied in the context of still image coding.

[0004] Due to limited network resources and ever-increasing demands for higher video quality, improved compression and decompression techniques that improve compression ratios with little or no sacrifice in image quality are desirable.

[0005] Currently, neural network (NN) and deep learning (DL) techniques that utilize artificial neural networks have been used for some time in the technical field of encoding and decoding videos, images (e.g., still images), etc.

[0006] It is desirable to further improve the efficiency of such image coding (video coding or still image coding) based on trained networks, taking into account limitations in available memory and / or processing speed.

[0007] In particular, conventional constrained image compression coding suffers from poor parallelization suitability and challenging memory requirements. Summary of the Invention [Problem to be solved by the invention]

[0008] In particular, the present invention relates to a method and apparatus for coding image or video data by means of a neural network, such as the neural network described in the detailed description below. The use of a neural network allows reliable encoding and decoding and estimation of entropy models in a self-learning manner, which can result in high accuracy of images reconstructed from encoded compressed input data. [Means for solving the problem]

[0009] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0010] According to a first aspect, there is provided a method of encoding at least a portion of an image (e.g. one or more blocks, slices, tiles, etc.), the method comprising (for at least said portion of the image) encoding a primary component (selected from the components of the image) of the image independently from at least one secondary (non-primary) component (selected from the components of the image) of the image, and encoding said at least one secondary component of the image using information from the primary component.

[0011] In principle, an image may be a still image or an intraframe of a video sequence. Here, and in the following description, it should be understood that an image has components, particularly luminance and color components. Components may be thought of as dimensions of an orthogonal basis describing a full-color image. For example, if an image is represented in YUV space, the components are luma Y, chroma U, and chroma V. One of the components of an image is selected as the primary component, and one or more other components are selected as secondary (non-primary) components. The terms "secondary component" and "non-primary component" are used interchangeably herein to refer to components coded using auxiliary information provided by the primary component. Encoding and decoding the secondary component(s) using auxiliary information provided by the primary component results in a high accuracy of the reconstructed image obtained after the decoding process.

[0012] Image encoding of the disclosed type allows for high parallelism (due to encoding the primary component independently from the secondary component) and reduces memory requirements compared to the prior art. In particular, the primary component and the at least one secondary component can be encoded concurrently.

[0013] According to one implementation, the primary component of the image is a luma component and the at least one secondary component of the image is a chroma component. For example, two secondary components of the image are encoded simultaneously, one of which is a chroma component and the other of which is another chroma component. According to another implementation, the primary component of the image is a chroma component and at least one secondary component of the image is a luma component. Thus, a high degree of flexibility is provided in actually conditioning one component with another component.

[0014] Global encoding may comprise processing in the latent space, which may allow, inter alia, processing of downsampled input data and thus faster processing with lower processing load. Note that the terms "downsampling" and "upsampling" are used herein to mean reducing and increasing the size of a tensor representation of data, respectively.

[0015] Regarding the processing in latent space by a certain implementation, a) encoding the first component by: representing the first-order component by a first tensor; converting the first tensor into a first latent tensor; processing the first latent tensor to generate a first bitstream; b) encoding the at least one secondary component comprises: representing the at least one second-order component by a second tensor different from the first tensor; concatenating the second tensor and the first tensor to obtain a concatenated tensor; converting the coupling tensor into a second latent tensor; Processing the second latent tensor to generate a second bitstream.

[0016] At least one of the sizes in the height or width dimension of the first latent tensor may be smaller than the corresponding size in the height or width dimension of the first tensor, and / or the size in the height or width dimension of the second latent tensor may be smaller than the corresponding size in the height or width dimension of the concatenated tensor. For example, a reduction ratio by a factor of 16 or 32 in the height and / or width dimension may be used.

[0017] The size or sub-pixel offset of the samples of the second tensor in at least one of the height and width dimensions of the tensor may be different from the size or sub-pixel offset of the samples in at least one of the height and width dimensions of the first tensor. Thus, according to another implementation, a) Encoding the primary component is: Represent the primary component by a first tensor with height and width dimensions; converting the first tensor into a first latent tensor; processing the first latent tensor to generate a first bitstream; b) encoding the at least one secondary component comprises: representing the at least one secondary component by a second tensor different from the first tensor and having height and width dimensions; determining whether the size or sub-pixel offset of samples of the second tensor in at least one of the height and width dimensions differs from the size or sub-pixel offset of samples of the first tensor in at least one of the height and width dimensions, and if it is determined that the size or sub-pixel offset of samples of the second tensor differs from the size or sub-pixel offset of samples of the first tensor, adjusting sample positions of the first tensor to match the sample positions of the second tensor, thereby obtaining an adjusted first tensor; concatenating the second tensor and the adjusted first tensor to obtain a concatenated tensor only if it is determined that the size or sub-pixel offset of the samples of the second tensor is different from the size or sub-pixel offset of the samples of the first tensor, and otherwise concatenating the second tensor and the first tensor to obtain a concatenated tensor; converting the coupling tensor into a second latent tensor; and processing the second latent tensor to generate a second bitstream.

[0018] Again, at least one of the size in the height or width dimension of the first latent tensor may be smaller than the corresponding size in the height or width dimension of the first tensor, and / or the size in the height or width dimension of the second latent tensor may be smaller than the corresponding size in the height or width dimension of the concatenated tensor. For example, a reduction ratio by a factor of 16 or 32 in the height and / or width dimension may be used. Adjusting the sample locations of the first tensor to match the sample locations of the second tensor may include, for example, downsampling by a factor of 2 in the width and height of the first tensor.

[0019] According to one implementation, the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and the size of the first latent tensor in the channel dimension is one of greater than, less than, and equal to the size of the second latent tensor in the channel dimension. If the first-order component is considered more important than the second-order component(s), as is typically the case, the channel length of the first-order component may be greater than the channel length of the second-order component(s). If the signals of the first-order component are relatively clear and the signals of the non-first-order components are relatively noisy, the channel length of the first-order component may be shorter than the channel length of the second-order component. Numerical experiments show that shorter channel lengths can be used compared to conventional techniques without significant degradation in the quality of the reconstructed image, thereby reducing memory requirements.

[0020] In general, a first tensor may be transformed into a first latent tensor by a first neural network, and a concatenated tensor may be transformed into a second latent tensor by a second neural network different from the first neural network. In this case, the first and second neural networks may be trained collaboratively to determine the size of the first latent tensor in the channel dimension and the size of the second latent tensor in the channel dimension. The channel length determination may be performed by exhaustive search or in a content-adaptive manner. A set of models may be trained, each based on a different number of channels for encoding primary and non-primary components. This may allow the neural network to optimize the channel length involved.

[0021] The determined channel length must also be used by a decoder used to reconstruct the encoded components. Thus, according to one implementation, the size of a first latent tensor in the channel dimension may be signaled in a first bitstream, and the size of a second latent tensor in the channel dimension may be signaled in a second bitstream. The signaling may be performed explicitly or implicitly, allowing the decoder to be directly informed about the channel length in a bit-saving manner.

[0022] According to one implementation, the first bitstream is generated based on a first entropy model and the second bitstream is generated based on a second entropy model that is different from the first entropy model, such an entropy model allowing reliable estimation of statistical properties used in the process of converting the tensor representation of the data into the bitstream.

[0023] The disclosed method can be advantageously implemented in the context of a hyperprior architecture that provides useful side information for coding of (parts of) an image in order to improve the accuracy of the (parts of) the reconstructed image. According to a particular implementation, the method comprises: A) converting the first latent tensor into a first hyperlatent tensor; processing the first hyperlatent tensor based on a third entropy model to generate a third bitstream; decoding the third bitstream using the third entropy model to obtain a recovered first hyperlatent tensor; Transforming the recovered first hyperlatent tensor into a first hyper-decoded hyperlatent tensor; generating the first entropy model based on the first hyper-decoded hyper-latent tensor and the first latent tensor; B) transforming the second latent tensor into a second hyperlatent tensor different from the first hyperlatent tensor; processing the second hyperlatent tensor based on a fourth entropy model to generate a fourth bitstream; decoding the fourth bitstream using the fourth entropy model to obtain a recovered second hyperlatent tensor; Transforming the recovered second hyperlatent tensor into a second hyper-decoded hyperlatent tensor; Further comprising generating the second entropy model based on the second hyper-decoded hyper-latent tensor and the second latent tensor.

[0024] Converting the first latent tensor to a first hyperlatent tensor may include downsampling the first latent tensor, and converting the second latent tensor to a second hyperlatent tensor may include downsampling the second latent tensor, for example, by a factor of 2 or 4, to further reduce the processing load.

[0025] The first and second entropy models thus generated, used to encode the latent representations of the primary components and connectivity tensors, respectively, can be autoregressive entropy models.

[0026] Neural networks may also be used to generate entropy models. For example, a third entropy model may be generated by a third neural network different from the first and second neural networks, a fourth entropy model may be generated by a fourth neural network different from the first, second, and third neural networks, a third bit stream may be generated by a fifth neural network different from the first through fourth neural networks and decoded by a sixth neural network different from the first through fifth neural networks, and a fourth bit stream may be generated by a seventh neural network different from the first through sixth neural networks and decoded by an eighth neural network different from the first through seventh neural networks. Additionally, the first entropy model may be generated by a ninth neural network distinct from the first through eighth neural networks used to encode the latent representations of the primary components, and the second entropy model used to encode the latent representations of the concatenated tensors may be generated by a tenth neural network distinct from the first through ninth neural networks.

[0027] According to a second aspect, a method for encoding at least a portion of an image is provided, the method comprising: providing a residual (for at least the portion of the image) including a first-order residual component for a first-order component of the image and at least one second-order residual component for at least one second-order component of the image different from the first-order component; encoding the first-order residual component independently of the at least one second-order residual component; and encoding the at least one second-order residual component using information from the first-order residual component. The image processed in this manner may be an interframe or still image of a video sequence. The residual results from subtracting a current portion of the image (or a portion of the image) from a predicted one, resulting in one residual component for each component of the image. Constrained residual encoding according to this method may be performed with the same advantages as the method according to the first aspect described above. Compared to constrained residual coding known in the art (see detailed description below), memory requirements can be reduced because a shorter channel length can be used for data representation in the latent space without significantly losing accuracy in image reconstruction.

[0028] The primary residual component and the at least one secondary residual component may be encoded concurrently. The primary component of an image may be a luma component, and the at least one secondary component of an image may be a chroma component. In this case, the at least one secondary residual component may include a residual component for a chroma component and another residual component for another chroma component. Alternatively, the primary component of an image may be a chroma component, and the at least one secondary component of an image may be a luma component.

[0029] Again, the processing can be performed in latent space. According to one implementation of the method according to the second aspect, a) encoding the first-order residual component: expressing the first-order residual component by a first tensor; Transforming the first tensor into a first latent tensor (e.g., of a smaller size in width and / or height dimensions compared to the first tensor); processing the first latent tensor to generate a first bitstream; b) encoding the at least one secondary residual component comprises: representing the at least one second-order residual component by a second tensor different from the first tensor; concatenating the second tensor and the first tensor to obtain a concatenated tensor; transforming the connectivity tensor into a second latent tensor (e.g., of a smaller size in width and / or height dimensions compared to the connectivity tensor); Processing the second latent tensor to generate a second bitstream.

[0030] According to another implementation of the method according to the second aspect, a) encoding the first-order residual component: representing the first-order residual component by a first tensor having height and width dimensions; converting the first tensor into a first latent tensor; processing the first latent tensor to generate a first bitstream; b) encoding the at least one secondary residual component comprises: representing the at least one second-order residual component by a second tensor distinct from the first tensor and having height and width dimensions; determining whether the size or sub-pixel offset of samples of the second tensor in at least one of the height dimension and the width dimension differs from the size or sub-pixel offset of samples of the first tensor in at least one of the height dimension and the width dimension, and if it is determined that the size or sub-pixel offset of samples of the second tensor differs from the size or sub-pixel offset of samples of the first tensor, adjusting sample positions of the first tensor to match the sample positions of the second tensor, thereby obtaining an adjusted first tensor; concatenate the second tensor and the adjusted first tensor to obtain a concatenated tensor only if it is determined that the size or sub-pixel offset of the samples of the second tensor is different from the size or sub-pixel offset of the samples of the first tensor, and otherwise concatenate the second tensor and the first tensor to obtain a concatenated tensor; converting the coupling tensor into a second latent tensor; Processing the second latent tensor to generate a second bitstream.

[0031] Again, at least one of the sizes in the height or width dimension of the first latent tensor may be smaller than the corresponding size in the height or width dimension of the first tensor, and / or the size in the height or width dimension of the second latent tensor may be smaller than the corresponding size in the height or width dimension of the concatenated tensor.

[0032] According to another implementation of the method according to the second aspect, the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and the size of the first latent tensor in the channel dimension is one of greater than, less than, and equal to the size of the second latent tensor in the channel dimension.

[0033] Neural networks may also be advantageously used in the method according to the second aspect. Thus, a first tensor may be transformed into a first latent tensor by a first neural network, and the concatenated tensor may be transformed into a second latent tensor by a second neural network different from the first neural network. In this case, the first and second neural networks may be trained cooperatively to determine the size of the first latent tensor in the channel dimension and the size of the second latent tensor in the channel dimension. The determined size of the first latent tensor in the channel dimension may be signaled in the first bitstream, and the size of the second latent tensor in the channel dimension may be signaled in the second bitstream.

[0034] According to another implementation of the method according to the second aspect, the first bitstream is generated based on a first entropy model and the second bitstream is generated based on a second entropy model different from the first entropy model.

[0035] In the disclosed conditional residual coding according to the second aspect, a hyperprior pipeline may also be used. Thus, the method according to the second aspect comprises: A) converting the first latent tensor into a first hyperlatent tensor; processing the first hyperlatent tensor based on a third entropy model to generate a third bitstream; decoding the third bitstream using the third entropy model to obtain a recovered first hyperlatent tensor; Transforming the recovered first hyperlatent tensor into a first hyper-decoded hyperlatent tensor; generating the first entropy model based on the first hyper-decoded hyper-latent tensor and the first latent tensor; B) transforming the second latent tensor into a second hyperlatent tensor different from the first hyperlatent tensor; processing the second hyperlatent tensor based on a fourth entropy model to generate a fourth bitstream; decoding the fourth bitstream using the fourth entropy model to obtain a recovered second hyperlatent tensor; Transforming the recovered second hyperlatent tensor into a second hyper-decoded hyperlatent tensor; generating the second entropy model based on the second hyper-decoded hyper-latent tensor and the second latent tensor; It may further include:

[0036] Converting the first latent tensor to a first hyperlatent tensor may include downsampling the first latent tensor, and converting the second latent tensor to a second hyperlatent tensor may include downsampling the second latent tensor, for example, by a factor of 2 or 4.

[0037] The third entropy model may be generated by a third neural network different from the first and second neural networks, the fourth entropy model may be generated by a fourth neural network different from the first, second, and third neural networks, the third bit stream may be generated by a fifth neural network different from the first through fourth neural networks and decoded by a sixth neural network different from the first through fifth neural networks, the fourth bit stream may be generated by a seventh neural network different from the first through sixth neural networks and decoded by an eighth neural network different from the first through seventh neural networks, the first entropy model may be generated by a ninth neural network different from the first through eighth neural networks, and the second entropy model may be generated by a tenth neural network different from the first through ninth neural networks.

[0038] It should be noted that in the aspects and implementations described above, tensors that are converted to bitstreams may undergo quantization prior to the conversion process, which compresses a range of values ​​into a single value in order to reduce the amount of data to be processed.

[0039] Corresponding to the encoding method described above, there is also provided herein a method for reconstructing at least a portion of an image based on conditional coding, with the same or similar advantages as described above. The reconstruction of at least a portion of the image can be facilitated by the use of a neural network, for example a neural network as described in the detailed description below.

[0040] According to a third aspect, there is provided a method for reconstructing at least a portion of an image, the method comprising: processing a first bitstream (for at least the portion of the image) based on a first entropy model to obtain a first latent tensor; and processing the first latent tensor to obtain a first tensor representing a first-order component of the image. The method further comprises processing a second bitstream (for at least the portion of the image) different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; and processing the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order component of the image. In principle, the image can be a still image or an intraframe of a video sequence.

[0041] The first and second entropy models can be provided by the hyperprior pipeline described above.

[0042] The first latent tensor can be processed independently of the processing of the second latent tensor. In fact, the encoded first-order components can be recovered even if data for the second-order components is lost. In this way, the original compressed image data can be reconstructed reliably and quickly due to the possible parallel processing of the first and second bitstreams.

[0043] The primary component of the image may be a luma component and the at least one secondary component of the image may be a chroma component. In particular, the second tensor may represent two secondary components, one of which is a chroma component and the other of which is another chroma component. Alternatively, the primary component of the image may be a chroma component and the at least one secondary component of the image may be a luma component.

[0044] According to one implementation of the method of the third aspect, processing the first latent tensor includes converting the first latent tensor to the first tensor, and processing the second latent tensor includes concatenating the second latent tensor with the first latent tensor to obtain a concatenated tensor and converting the concatenated tensor to the second tensor. At least one of these conversions may include upsampling. Thus, processing in latent space may be performed at a lower resolution, as necessary for accurate reconstruction of the components in YUV space or any other space appropriately used for image representation.

[0045] According to another implementation of the method of the third aspect, each of the first and second latent tensors has height and width dimensions, and processing the first latent tensor includes converting the first latent tensor to the first tensor, and processing the second latent tensor includes determining whether a size or sub-pixel offset of samples of the second latent tensor in at least one of the height and width dimensions differs from a size or sub-pixel offset of samples in at least one of the height and width dimensions of the first latent tensor. When it is determined that the size or sub-pixel offset of samples of the second latent tensor differs from the size or sub-pixel offset of samples of the first latent tensor, sample positions of the first latent tensor are adjusted to match the sample positions of the second latent tensor, thereby obtaining an adjusted first latent tensor. Further, only if it is determined that the size or sub-pixel offset of the samples of the second latent tensor is different from the size or sub-pixel offset of the samples of the first latent tensor, the second latent tensor and the adjusted first latent tensor are concatenated to obtain a concatenated latent tensor; otherwise, a concatenation of the second latent tensor and the first latent tensor is performed to obtain a concatenated latent tensor, and the concatenated latent tensor is transformed into the second tensor.

[0046] The first bit stream may be processed by a first neural network, the second bit stream may be processed by a second neural network different from the first neural network, the first latent tensor may be transformed by a third neural network different from the first and second networks, and the concatenated latent tensor may be transformed by a fourth neural network different from the first, second, and third networks.

[0047] According to another implementation of the method according to the third aspect, the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and a size of the first latent tensor in the channel dimension is one of greater than, less than, and equal to a size of the second latent tensor in the channel dimension. Information about the sizes of the first and second latent tensors in the channel dimension can be obtained from information signaled in the first and second bitstreams, respectively.

[0048] According to a fourth aspect, there is provided a method for reconstructing at least a portion of an image. The method includes processing a first bitstream (for at least the portion of the image) based on a first entropy model to obtain a first latent tensor, and processing the first latent tensor to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image. The method further includes processing a second bitstream (for at least the portion of the image) different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor, and processing the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order residual component of a residual for at least one second-order component of the image. Thus, a residual is obtained that includes a first residual component for the first component and a second residual component for at least one second-order component. In principle, the image can be a still image or an inter-frame of a video sequence.

[0049] The first and second entropy models can be provided by the hyperprior pipeline described above.

[0050] The first latent tensor may be processed independently from the processing of the second latent tensor.

[0051] The primary component of the image may be a luma component and the at least one secondary component of the image may be a chroma component. In this case, the second tensor may represent two residual components for the two secondary components, one of which is a chroma component and the other of which is another chroma component. Alternatively, the primary component of the image may be a chroma component and the at least one secondary component of the image may be a luma component.

[0052] According to one implementation of the method according to the fourth aspect, processing the first latent tensor includes converting the first latent tensor to a first tensor, and processing the second latent tensor includes concatenating the second latent tensor with the first latent tensor to obtain a concatenated tensor, and converting the concatenated tensor to the second tensor.

[0053] At least one of these transformations may involve upsampling, so that processing in the latent space may be performed at a lower resolution, as required for accurate reconstruction of the components in YUV space or any other space suitably used for image representation.

[0054] According to another implementation of the method according to the fourth aspect, each of the first and second latent tensors has height and width dimensions, and processing the first latent tensor includes converting the first latent tensor to the first tensor, and processing the second latent tensor includes determining whether a size or sub-pixel offset of samples of the second latent tensor in at least one of the height and width dimensions differs from a size or sub-pixel offset of samples in at least one of the height and width dimensions of the first latent tensor. When it is determined that the size or sub-pixel offset of samples of the second latent tensor differs from the size or sub-pixel offset of samples of the first latent tensor, sample positions of the first latent tensor are adjusted to match the sample positions of the second latent tensor, thereby obtaining an adjusted first latent tensor. Further, only if it is determined that the size or sub-pixel offset of the samples of the second latent tensor is different from the size or sub-pixel offset of the samples of the first latent tensor, the second latent tensor and the adjusted first latent tensor are concatenated to obtain a concatenated latent tensor; otherwise, a concatenation of the second latent tensor and the first latent tensor is performed to obtain a concatenated latent tensor. Further, the concatenated latent tensor is transformed into a second tensor.

[0055] The first bit stream may be processed by a first neural network, the second bit stream may be processed by a second neural network different from the first neural network, the first latent tensor may be transformed by a third neural network different from the first and second networks, and the concatenated latent tensor may be transformed by a fourth neural network different from the first, second, and third networks.

[0056] According to another implementation of the method according to the fourth aspect, the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and the size of the first latent tensor in the channel dimension is one of greater than, less than, and equal to the size of the second latent tensor in the channel dimension.

[0057] The processing of the first bitstream may include obtaining information regarding the size of a first latent tensor in a channel dimension signaled in the first bitstream, and the processing of the second bitstream may include obtaining information regarding the size of a second latent tensor in a channel dimension signaled in the second bitstream.

[0058] Any of the above-described exemplary implementations may be combined as deemed appropriate. A method according to any of the above-described aspects and implementations may be implemented in an apparatus.

[0059] According to a fifth aspect, there is provided an apparatus for encoding at least a portion of an image, the apparatus having one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the apparatus to perform a method according to any one of the first and second aspects and corresponding implementations described above.

[0060] According to a sixth aspect, there is provided an apparatus for reconstructing at least a portion of an image, the apparatus having one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the apparatus to perform a method according to any one of the third and fourth aspects and corresponding implementations described above.

[0061] According to a seventh aspect, there is provided a processing apparatus for encoding at least a part of an image, the processing apparatus comprising processing circuitry configured to: (for at least the part of the image) encode a primary component of the image independently from at least one secondary component of the image; and (for at least said part of the image) encode said at least one secondary component of the image using information from the primary component.

[0062] The processing device is configured to perform the steps of the method according to the first aspect, and may also be configured to perform the steps of one or more of the corresponding implementations described above.

[0063] According to an eighth aspect, there is provided a processing apparatus for encoding at least a part of an image, the processing apparatus comprising: processing circuitry configured to provide a residual comprising a primary residual component for a primary component of the image and at least one secondary residual component for at least one secondary component of the image that is different from the primary component; encoding the primary residual component independently of the at least one secondary residual component; and encoding the at least one secondary residual component using information from the primary residual component.

[0064] The processing device is configured to perform the steps of the method according to the second aspect, and may also be configured to perform the steps of one or more of the corresponding implementations described above.

[0065] According to a ninth aspect, there is provided a processing device for reconstructing at least a portion of an image, the processing device comprising: a processing circuit configured to: process a first bitstream based on a first entropy model to obtain a first latent tensor, process the first latent tensor to obtain a first tensor representing a first component of the image, process a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor, and process the second latent tensor using information from the first latent tensor to obtain a second tensor representing the at least one second component of the image.

[0066] The processing device is configured to perform the steps of the method according to the third aspect, and may also be configured to perform the steps of one or more of the corresponding implementations described above.

[0067] According to a tenth aspect, there is provided a processing device for reconstructing at least a portion of an image, the processing device comprising: processing circuitry configured to: process a first bitstream based on a first entropy model to obtain a first latent tensor; process the first latent tensor to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image; process a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; and process the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order residual component of the residual for at least one second-order component of the image.

[0068] The processing device is configured to perform the steps of the method according to the fourth aspect, and may also be configured to perform the steps of one or more of the corresponding implementations described above.

[0069] Furthermore, according to an eleventh aspect, there is provided a computer program stored on a non-transitory medium, comprising code that, when executed on one or more processors, performs the steps of the method according to any of the above aspects and implementations. [Brief explanation of the drawings]

[0070] Hereinafter, the technical background and embodiments of the present invention will be described in more detail with reference to the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram showing channels processed by layers of a neural network. [Figure 2] A schematic diagram showing an autoencoder type of neural network. [Figure 3] Schematic diagram showing a network architecture including the Hyperprior model. [Figure 4] FIG. 1 is a block diagram showing the structure of a cloud-based solution for machine-based tasks, such as machine vision tasks. [Figure 5] Figure 1 shows a block diagram of the structure of an end-to-end trainable video compression framework. [Figure 6] FIG. 1 is a block diagram illustrating a network for motion vector (MV) compression. [Figure 7] FIG. 1 is a block diagram illustrating a learned image compression configuration of the technology. [Figure 8] FIG. 2 is a block diagram illustrating another learned image compression arrangement of the art. [Figure 9] FIG. 1 illustrates the concept of conditional coding. [Figure 10] FIG. 1 illustrates the concept of residual coding. [Figure 11] FIG. 1 illustrates the concept of residual constrained coding. [Figure 12] FIG. 1 illustrates constrained intra-coding according to an embodiment of the present invention. [Figure 13] FIG. 1 illustrates constrained residual coding according to an embodiment of the present invention. [Figure 14] FIG. 1 illustrates conditional coding according to an embodiment of the present invention. [Figure 15] FIG. 2 illustrates conditional intra-coding for input data in YUV420 format according to an embodiment of the present invention. [Figure 16] FIG. 2 illustrates conditional intra-coding for input data in YUV444 format according to an embodiment of the present invention. [Figure 17] FIG. 2 illustrates constrained residual coding for input data in YUV420 format according to an embodiment of the present invention. [Figure 18] FIG. 2 illustrates constrained residual coding for input data in YUV444 format according to an embodiment of the present invention. [Figure 19] FIG. 10 illustrates constrained residual coding for input data in YUV420 format according to another embodiment of the present invention. [Figure 20] FIG. 10 illustrates constrained residual coding for input data in YUV444 format according to another embodiment of the present invention. [Figure 21] 4 is a flowchart illustrating an exemplary method for encoding at least a portion of an image, according to an embodiment of the present invention. [Figure 22] 5 is a flowchart illustrating an exemplary method for encoding at least a portion of an image, according to another embodiment of the present invention. [Figure 23] 4 is a flowchart illustrating an exemplary method for reconstructing at least a portion of an image, according to an embodiment of the present invention. [Figure 24] 5 is a flowchart illustrating an exemplary method for reconstructing at least a portion of an image, according to another embodiment of the present invention. [Figure 25] 1 illustrates a processing device configured to perform a method for encoding or reconstructing at least a portion of an image according to an embodiment of the present invention; [Figure 26]1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Figure 27] FIG. 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention. [Figure 28] FIG. 1 is a block diagram illustrating an example of an encoding device or a decoding device. [Figure 29] FIG. 10 is a block diagram illustrating another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE INVENTION

[0071] In the following description, reference is made to the accompanying drawings. The drawings form a part of the disclosure and show, by way of example, specific aspects of embodiments of the present invention or in which embodiments of the present invention may be used. It is understood that these embodiments of the present invention may be used in other aspects and may include structural or logical changes not shown in the drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0072] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform that method, and vice versa. For example, when one or more specific method steps are described, a corresponding device may include one or more units, e.g., functional units, for performing the described one or more method steps (e.g., one unit performing the one or more steps, or multiple units each performing one or more of the multiple steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, when a specific apparatus is described based on one or more units, e.g., functional units, a corresponding method may include a step for performing the function of the one or more units (e.g., one step performing the function of the one or more units, or multiple steps each performing the function of one or more units of the multiple units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that features of the various exemplary embodiments and / or aspects described herein can be combined with each other, unless specifically stated otherwise.

[0073] The following provides an overview of some of the technical terms used.

[0074] Artificial Neural Networks An artificial neural network (ANN) or connectionist system is a computing system loosely inspired by the biological neural networks that make up animal brains. Such systems "learn" to perform tasks by considering examples, generally without being programmed with task-specific rules. For example, in image recognition, a system might learn to identify images containing cats by analyzing example images that have been manually labeled as "cat" or "non-cat" and using the results to identify cats in other images. This is done without any prior knowledge of cats, such as whether they have fur, tails, whiskers, or cat-like faces. Instead, the system automatically generates discriminative features from the examples it processes.

[0075] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely model neurons in a biological brain. Each connection can send a signal to other neurons, just like a synapse in a biological brain. When an artificial neuron receives a signal, it can process it and send a signal to the neurons connected to it.

[0076] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0077] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, resulting in a deviation from biology. ANNs have been used in a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even in activities traditionally considered exclusively human, such as painting.

[0078] Convolutional Neural Networks The name "convolutional neural network" (CNN) indicates that the network employs a mathematical operation called convolution, which is a special kind of linear operation. A convolutional network is simply a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0079] Figure 1 shows the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is the layer to which input (e.g., a portion of an image shown in Figure 1) is provided for processing.

[0080] The hidden layers of a CNN typically consist of a series of convolutional layers that convolve using multiplication or other dot products. The result of a layer is one or more feature maps (f.maps in Figure 1), also called channels. There may be subsampling across some or all layers. As a result, feature maps may be smaller, as shown in Figure 1. The activation function in a CNN is typically a RELU (Rectified Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. These layers are colloquially called convolutions, but this is simply by convention. Mathematically, the technical term is sliding dot products or cross-correlations. The index in the matrix is ​​important in that it affects how the weights are determined at a particular index point.

[0081] When programming a CNN to process images, as shown in Figure 1, the input is a tensor with the shape (number of images) × (image width) × (image height) × (image depth). Then, after passing through a convolutional layer, the image is abstracted into a feature map with the shape (number of images) × (feature map width) × (feature map height) × (feature map channels). A convolutional layer in a neural network must have the following attributes: A convolution kernel defined by its width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) should be equal to the number of channels of the input feature map (depth).

[0082] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the fully connected nature of the nodes, they suffer from high dimensionality and do not scale well to higher-resolution images. A 1000 x 1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed at a fully connected scale. Furthermore, such network architectures do not take into account the spatial structure of the data, treating distant input pixels the same as pixels that are close to each other. This ignores the locality of reference in the image data, both computationally and semantically. Therefore, fully connected neurons are useless for purposes such as image recognition, which are dominated by spatially localized input patterns.

[0083] Convolutional neural networks are biologically inspired variants of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges presented by MLP architectures by exploiting the strong spatial local correlations present in natural images. The convolutional layer is the core building block of a CNN. Its parameters consist of a collection of learnable filters (the aforementioned kernels). These filters have small receptive fields but extend to the full depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, computing dot products between the filter and the input entries, generating a two-dimensional activation map for that filter. As a result, the network learns filters that become active when it detects some particular type of feature at some spatial location in the input.

[0084] Stacking the activation maps for all filters along the depth dimension forms the complete output volume of the convolutional layer. Thus, every entry in the output volume can be interpreted as the output of a neuron that sees a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation for a given filter. Feature map and activation are synonymous. In some papers, it is called an activation map because it is a mapping that corresponds to the activation of different parts of the image, and it is also called a feature map because it is a mapping of where certain features are found in the image. High activation means that certain features are found.

[0085] Another important concept in CNNs is pooling, a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. Max pooling partitions the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0086] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers serve to progressively reduce the spatial size of the representation, reducing the number of parameters, memory footprint, and computational effort in the network, and thus also controlling overfitting. In CNN architectures, it is common to insert pooling layers periodically between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0087] A pooling layer operates independently on every depth slice of the input, spatially resizing the slice. The most common form is a pooling layer with a filter of size 2x2, applied with a stride of 2 downsamples on every depth slice of the input by 2 along both width and height, discarding 75% of the activations. In this case, all max operations are on 4 numbers. The depth dimension is not changed.

[0088] In addition to max pooling, pooling units can use other functions such as average pooling and l2-norm pooling. Average pooling was historically popular but has recently fallen out of favor compared to max pooling, which performs better in practice. Due to the aggressive reduction in representation size, the recent trend is toward using smaller filters or discarding the pooling layer entirely. "Region of Interest" pooling (also known as ROI pooling) is a variant of max pooling, where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0089] The aforementioned ReLU stands for rectified linear unit, which applies a non-saturating activation function. By setting negative values ​​to 0, it effectively removes negative values ​​from the activation map. This increases the nonlinearity of the decision function and the overall network without affecting the receptive fields of the convolutional layers. Other functions are also used to increase nonlinearity, such as saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty in generalization accuracy.

[0090] After several convolutional and max-pooling layers, high-level inference in neural networks occurs via fully connected layers. Neurons in fully connected layers have connections to all activations in the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can therefore be computed as affine transformations followed by matrix multiplications followed by bias offsets (vector additions of learned or fixed bias terms).

[0091] The "loss layer" specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions suitable for different tasks can be used: softmax loss is used to predict a single class out of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; and Euclidean loss is used for regression to real-valued labels.

[0092] In summary, FIG. 1 illustrates the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map containing several channels corresponding to the number of filters in the layer's set of learnable filters. The feature map is then subsampled, for example, using a pooling layer to reduce the dimensionality of each channel in the feature map. Next, data comes to another convolutional layer, which may have a different number of output channels leading to a different number of channels in the feature map. As mentioned above, the number of input and output channels is a hyperparameter of the layer. To establish network connectivity, these parameters need to be synchronized between two connected layers; for example, the number of input channels for the current layer should be equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., images, the number of input channels is typically equal to the number of channels in the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation.

[0093] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram is shown in Figure 2. The goal of an autoencoder is to learn a representation (encoding) of a set of data, typically for dimensionality reduction, by training the network to ignore signal "noise." Along with the reduction side, a reconstruction side is learned, where the autoencoder attempts to generate a representation from the reduced encoding that is as close as possible to the original input, hence the name.

[0094] In the simplest case, given a single hidden layer, the encoder stage of an autoencoder takes an input x and maps it to h. h=σ(Wx+b). This image h is typically called a code, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the rectified linear unit. W is a weight matrix, and b is a bias vector. The weights and biases are typically initialized randomly and then iteratively updated during training through backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' that has the same shape as x. x'=σ'(W'h'+b') Here, σ′, W′, b′ for the decoder may be unrelated to the corresponding σ, W, b for the encoder.

[0095] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach for learning latent representations, resulting in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. This is because the data is represented by a directed graphical model p θ (x|h), and the encoder generates the posterior distribution p θ Approximation q to (h|x) φ Suppose we are trying to learn (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much better than a standard autoencoder. The objective function of a VAE has the form:

number

number

[0096] Recent advances in the field of artificial neural networks, and convolutional neural networks in particular, have enabled researchers to focus on applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed. Data compression is considered a fundamental and well-studied problem in engineering, typically formulated with the goal of designing a code with minimal entropy for a given discrete data ensemble. This solution relies heavily on knowledge of the data's probabilistic structure; thus, the problem is closely related to probabilistic source modeling. However, because all practical codes must have finite entropy, continuous-valued data (such as vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces error. In this context, known as the lossy compression problem, a tradeoff must be made between two competing costs: the entropy (rate) of the discretized representation and the error (distortion) resulting from quantization. Different compression applications, such as data storage and transmission over limited-capacity channels, require different rate-distortion tradeoffs. Joint rate-distortion optimization is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable. For this reason, most existing image compression methods operate by linearly transforming a data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using a reversible entropy code. This approach is called transform coding because of the central role of the transform. For example, JPEG uses a discrete cosine transform on blocks of pixels, while JPEG2000 uses a multiscale orthogonal wavelet decomposition. Typically, the three components of a transform coding method—the transform, the quantizer, and the entropy code—are optimized separately (often through manual parameter tuning).Modern video compression standards such as HEVC, VVC, and EVC also use transform representations to code the residual signal after prediction, and several transforms such as the discrete cosine transform (DCT, DST) and the low frequency non-separable manually optimized transform (LFNST) are used for this purpose.

[0097] variational image compression In [Non-Patent Document 1], the authors proposed a framework for end-to-end optimization of image compression models based on nonlinear transforms. Previously, the authors demonstrated that a model consisting of a linear-nonlinear block transform optimized for a perceptual distortion metric exhibits visually superior performance compared to a model optimized for mean squared error (MSE). Here, the authors optimize for MSE but use a more flexible transform constructed from a cascade of linear convolutions and nonlinearities. In particular, the authors use a generalized divisive normalization (GDN) joint nonlinearity, inspired by models of neurons in the biological visual system and proven effective in Gaussianizing image densities. This cascade transform is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform. [Non-Patent Document 1] J. Balle, L. Valero Laparra, and E. P. Simoncelli (2015), "Density Modeling of Images Using a Generalized Normalization Transformation," arXiv e-prints, presented at the 4th Int. Conf. for Learning Representations, 2016 (hereafter referred to as "Balle")

[0098] For any desired point along the rate-distortion curve, the parameters of both the analytical and synthetic transforms are jointly optimized using stochastic gradient descent. To achieve this in the presence of quantization (which produces zero gradients almost everywhere), the authors replace the quantization step with additive uniform noise and use a proxy loss function based on continuous relaxation of a probabilistic model. The relaxed rate-distortion optimization problem is somewhat similar to those used to fit generative image models, particularly variational autoencoders, but the authors impose different constraints to ensure approximating a discrete problem along the entire rate-distortion curve. Finally, rather than reporting differential or discrete entropy estimates, the authors implement an entropy code and report performance using actual bit rates, thus demonstrating the feasibility of their solution as a fully lossy compression method.

[0099] J. Balle describes an end-to-end trainable model for image compression based on a variational autoencoder. The model incorporates hyperpriors to effectively capture spatial dependencies in the latent representation. The hyperpriors are related to side information that is also transmitted to the decode side, a concept universal to virtually all modern image codecs but rarely explored in image compression using ANNs. Unlike existing autoencoder compression methods, this model trains a complex prior (prior) jointly with the underlying autoencoder. The authors demonstrate that this model delivers state-of-the-art image compression when measuring visual quality using the popular MS-SSIM index, and outperforms published ANN-based methods in rate-distortion performance when evaluated using a more traditional metric based on squared error (PSNR).

[0100] Figure 3 shows the network architecture including the Hyperprior model. a ,g s ) shows the image autoencoder architecture, and the right side (h a ,h s ) corresponds to an autoencoder that implements hyperpriors. Factorized-prior models are based on the analytical transformation g a and synthetic transformation g s The same architecture is used for the quantization. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder converts the input image x into g a to generate a response y (a latent representation) with a spatially varying standard deviation. a contains multiple convolutional layers with subsampling and generalized division normalization (GDN) as the activation function.

[0101] The response is h ato summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. The encoder then uses the quantized vector ^z (^z; and so on) to estimate the spatial distribution of standard deviations ^σ, which is used to obtain probability (or frequency) values ​​for arithmetic coding (AE), and uses it to compress and transmit the quantized image representation ^y (or latent representation). The decoder first recovers ^z from the compressed signal, and then uses h s to obtain ^y, which also provides a correct probability estimate for successfully recovering ^y. Then, ^y is converted to g s to obtain the reconstructed image.

[0102] In further work, hyperprior probability modeling was further improved by introducing autoregressive models, e.g., based on the PixelCNN++ architecture, which allow exploiting the context of already decoded symbols in the latent space for better probability estimation of further symbols to be decoded, as shown, e.g., in Fig. 2 of [2]. [Non-patent document 2] L. Zhou, Zh. Sun, X. Wu, J. Wu, "End-to-end Optimized Image Compression with Attention Mechanism," CVPR 2019 (hereafter referred to as "Zhou")

[0103] Cloud Solutions for Machine Tasks Video Coding for Machines (VCM) is another popular direction in computer science today. The main idea behind this approach is to transmit coded representations of image or video information intended for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video coding, which are intended for human perception, the quality characteristic is not the reconstructed quality, but rather the performance of the computer vision task, e.g., object detection accuracy. This is illustrated in Figure 4.

[0104] Machine-based video coding, also known as collaborative intelligence, is a relatively new paradigm for the efficient deployment of deep neural networks across mobile-cloud infrastructures. By splitting the network between mobile and cloud, it is possible to distribute the computational workload so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which neural network processing is distributed among two or more distinct computational nodes (e.g., devices, but generally, any functionally defined node). Here, the term "node" does not refer to the nodes of the neural network described above. Rather, a (computational) node here refers to a separate device / module (physically or at least logically) that implements parts of the neural network. Such devices may be different servers, different end-user devices, a mix of servers and / or user devices and / or clouds and / or processors, etc. In other words, computational nodes can be considered as nodes that belong to the same neural network and communicate with each other to transfer data encoded within and for the neural network. For example, to perform complex computations, one or more layers may run on a first device and one or more layers may run in another device. However, the distribution may be finer, with a single layer running on multiple devices. In this disclosure, the term "multiple" refers to two or more. In some existing solutions, portions of the neural network function run on a device (such as a user device or edge device) or multiple such devices, and then the output (feature map) is passed to the cloud. The cloud is a collection of processing or computing systems located outside the device running the portion of the neural network. The concept of collaborative intelligence has also been extended to model training.In this case, data flows in both directions: from cloud to mobile during backpropagation in training, and from mobile to cloud during forward passes in training and inference.

[0105] Several studies have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization was demonstrated, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) from the mobile part to the cloud rather than sending the compressed natural image data to the cloud and performing object detection using the reconstructed image. Efficient compression of feature maps benefits image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, such as arithmetic coding, are popular approaches to compressing deep features (i.e., feature maps).

[0106] Today, video content contributes to over 80% of Internet traffic, and this proportion is expected to continue to increase. Therefore, it is important to build efficient video compression systems that produce higher-quality frames within a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of the compressed video. Therefore, efficient video compression can benefit other computer vision tasks. Meanwhile, techniques in video compression are also useful for action recognition and model compression. However, in the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. Although each module is well-designed, the entire compression system has not been optimized end-to-end. It is desirable to further improve video compression performance by jointly optimizing the entire compression system.

[0107] End-to-end image or video compression Recently, deep neural network (DNN)-based autoencoders for image compression have achieved performance comparable to or even better than traditional image codecs such as JPEG, JPEG2000, or BPG. One possible explanation is that DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations not used in traditional methods. However, directly applying these techniques to build end-to-end learning systems for video compression is not trivial. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences. A straightforward solution is to represent motion information using learning-based optical flow. However, current learning-based optical flow methods aim to generate flow fields as accurate as possible. Accurate optical flow is often suboptimal for a particular video task. Furthermore, the amount of optical flow data increases significantly compared to motion information in traditional compression systems, and directly applying existing compression methods to compress optical flow values ​​significantly increases the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective function for both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher-quality reconstructed frames (i.e., less distortion) given the number of bits (or bitrate) for compression. RDO is critical for video compression performance. To leverage the power of end-to-end training for learning-based compression systems, an RDO strategy that optimizes the entire system is needed.

[0108] In Non-Patent Document 3, the authors proposed an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding. [Non-patent document 3] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao, "DVC: An End-to-end Deep Video Compression Framework", Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.11006-11015

[0109] Such an encoder is shown in Figure 5. In particular, Figure 5 shows the overall structure of an end-to-end trainable video compression framework. To compress motion information, a CNN was designed to convert the optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder-type network was used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 6. The network architecture is somewhat similar to that of the ga / gs network in Figure 3. In particular, the optical flow is input to a series of convolution operations and nonlinear transformations, including the GDN and IGDN. The number of output channels for convolution (deconvolution) is 128, except for the last deconvolution layer, which is equal to 2. Given an optical flow with a size of M × N × 2, the MV encoder generates a motion representation with a size of M / 16 × N / 16 × 128. The motion representation is then quantized, entropy coded, and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0110] In particular, the following definitions hold:

[0111] Picture Size (Image Size; the terms "image" and "picture" are used interchangeably herein) : Refers to the width or height of a picture, or a width-height pair. Image width and height are typically measured in number of luma samples. Downsampling : Downsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is reduced. Upsampling :Upsampling is a process in which the sampling rate (sampling interval) of a discrete input signal is increased. Cropping Cropping: Cropping the outer edges of a digital image. Cropping can be used to make an image smaller (in number of samples) and / or to change the aspect ratio (length vs. width) of an image. Padding :Padding refers to increasing the size of an input image (or images) by generating new samples at the borders of the image by using predefined sample values ​​or by using sample values ​​at locations within the input image.

[0112] Convolution : The convolution is given by the following general formula: In the following, f() can be defined as the input signal and g() can be defined as the filter.

number

[0113] Autoencoder : A model that transforms signals into a (compressed) latent space and then back into the original signal space. Encoder : Downsample the image to a latent tensor (y) using convolutional layers with nonlinear activations and / or residuals. decoder : Upsample the latent tensor (y) to the original image size using convolutional layers with nonlinear activation and / or residuals. Hyperencoder : Further downsample the latent tensor to a smaller latent tensor (z) using convolutional layers with nonlinear activation and / or residuals. Hyper Decoder : For entropy estimation, upsample the smaller latent tensor (z) using convolutional layers with nonlinear activation and / or residuals. AE / AD (Arithmetic Encoder / Decoder) : Encode the latent tensor into a bitstream or decode the latent tensor from the bitstream using a given statistical prior. Autoregressive Entropy Estimation The process of estimating the statistical prior distribution of the latent tensor is sequentially Q: Quantization block ^y, ^z: quantized versions of the corresponding latent tensors Masked Convolution (MaskedConv): A type of convolution that masks certain latent tensor elements so that the model can only predict based on latent tensor elements that it has already seen.

[0114] H,W : Height and width of the input image Block / Patch : A subset of latent tensors on a rectangular grid Information sharing : The process of coordination of information from different patches P: Size of the rectangular patch K: kernel size that defines the number of neighboring patches included in the information sharing L: kernel size that defines how many of the previously encoded latent tensor elements are included in the information sharing Masked Convolution (MaskedConv) : A type of convolution that masks certain latent tensor elements so that the model can only make predictions based on latent tensor elements that it has already seen. PixelCNN : A convolutional network containing one or more layers of masked convolutions component : One dimension of an orthogonal basis for describing full-color images channel : A layer in a neural network Intra Codec The first frame or key frame of a video is treated as an intra-frame and is typically treated as an image. Inter-Codec After intra-coding, the video compression system performs inter-prediction: first, a motion estimation tool calculates the motion vector of an object, and then a motion compensation tool uses the motion vector to predict the next frame. Residual Codec The predicted frame is not always the same as the current frame, and the difference between the current frame and the predicted frame is the residual. A residual codec compresses the residual to compress the image.

[0115] signal conditioning: A training procedure in which an additional signal is used to aid in NN inference, but the additional signal is not present in the output and is very different from the output. conditional codec : A codec that uses signal conditioning to assist (guide) compression and reconstruction. In SOTA, conditional codecs are used for compressing video streams rather than images, since the auxiliary information needed for conditioning is not part of the input signal.

[0116] The following references provide details on some aspects of coding in the art: [Non-patent document 4] Ball´e, Johannes, Valero Laparra, and Eero P. Simoncelli, "End-to-end optimized image compression," 5th International Conference on Learning Representations, ICLR 2017, 2017. This paper teaches learned image compression. [Non-Patent Document 5] Ball´e, Johannes, et al., "Variational image compression with a scale hyperprior", International Conference on Learning Representations, 2018. This paper teaches the hyperprior model. [Non-patent document 6] Minnen, David, Johannes Ball´e, and George Toderici, "Joint Autoregressive and Hierarchical Priors for Learned Image Compression", NeurIPS, 2018. This paper teaches serial autoregressive context modeling. [Non-Patent Document 7] Th´eo Ladune, Pierrick Philippe, Wassim Hamidouche, Lu Zhang and Olivier D´eforges, paper "Optical Flow and Mode Selection for Learning-based Video Coding", IEEE 22nd International Workshop on Multimedia Signal Processing (MMSP), 2020, teaches conditional codecs. [Non-patent document 8] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao, "DVC: An End-to-end Deep Video Compression Framework," CVPR 2019, 2019, teaches a deep neural network-based video codec.

[0117] FIG. 7 is a block diagram illustrating a particular learned image compression configuration comprising an autoencoder and a hyperprior component known in the art that may be improved in accordance with this disclosure. The input image to be compressed is represented as a 3D tensor of size H×W×C, where H and W are the height and width (dimensions) of the image, respectively, and C is the number of components (e.g., a luma component and two chroma components). The input image is passed through an encoder 71, which downsamples the input image by applying multiple convolutions and nonlinear transformations to generate a latent tensor y. Note that in the context of deep learning, the terms "downsampling" and "upsampling" do not refer to resampling in the classical sense, but rather are general terms for resizing the H and W dimensions of a tensor. The latent tensor y output by encoder 71 represents the image in latent space and has size (H / D e )×(W / D e )×C e where D eis the downsampling factor of the encoder 71, and C e is the number of channels (e.g., the number of neural network layers involved in transforming the tensor representing the input image).

[0118] The latent tensor y is further downsampled by the hyperencoder 72 via convolution and nonlinear transformation to a hyperlatent tensor z. The hyperlatent tensor z has size (H / D h )×(W / D h )×C h The hyperlatent tensor z is quantized by block Q to obtain the quantized hyperlatent tensor ^z. The statistical properties of the values ​​of the quantized hyperlatent tensor ^z are estimated by a factorized entropy model. The arithmetic encoder AE uses these statistical properties to create a bitstream representation of the tensor ^z. All elements of the tensor ^z are written to the bitstream without the need for an autoregressive process.

[0119] The factorized entropy model serves as a codebook whose parameters are available to the decoder. The arithmetic decoder AD recovers the hyperlatent tensor ^z from the bitstream by using the factorized entropy model. The recovered hyperlatent tensor ^z is upsampled by the hyperdecoder 73 by applying multiple convolution operations and nonlinear transformations. The upsampled recovered hyperlatent tensor is denoted by ψ. The entropy of the quantized latent tensor ^y is estimated autoregressively based on the upsampled recovered hyperlatent tensor ψ. The autoregressive entropy model thus obtained is used to estimate the statistical properties of the quantized latent tensor ^y.

[0120] The arithmetic encoder AE uses these estimated statistical properties to create a bitstream representation of the quantized latent tensor ^y. In other words, the arithmetic encoder AE of the autoencoder component compresses the image information in the latent space by entropy encoding based on the side information provided by the hyperprior component. The latent tensor y is recovered from the bitstream by the arithmetic decoder AD on the receiver side by an autoregressive entropy model. The recovered latent tensor y is then upsampled by the decoder 74 by applying multiple convolution operations and nonlinear transformations to obtain a tensor representation of the reconstructed image.

[0121] Figure 8 shows a modification of the architecture shown in Figure 7. The processing of the encoder 81 and decoder 84 of the autoencoder component is similar to that of the encoder 71 and decoder 74 of the autoencoder component shown in Figure 7, and the processing of the encoder 82 and decoder 83 of the hyperprior component is similar to that of the encoder 72 and decoder 73 of the hyperprior component shown in Figure 7. Note that each of these encoders 71, 81, 72, 82 and decoders 73, 83, 74, 84 may have or be connected to a neural network, respectively. Furthermore, a neural network may be used to provide the entropy model involved.

[0122] Unlike the configuration shown in Figure 7, in the configuration shown in Figure 8, the quantized latent tensor ^y is subjected to masked convolution to obtain a tensor Φ with a reduced number of elements compared to ^y. An entropy model is obtained based on the concatenated tensors Φ and ψ (the upsampled recovered hyperlatent tensor). The entropy model thus obtained is used to estimate the statistical properties of the quantized latent tensor ^y.

[0123] Conditional coding refers to a specific type of coding in which side information is used to improve the quality of the reconstructed image. Figure 9 illustrates the principle idea of ​​conditional coding. Side information A is concatenated with the input frame x and jointly processed by an encoder 91. The quantized encoded information in latent space is written into a bitstream by an arithmetic encoder and recovered from the bitstream by an arithmetic decoder AD. The recovered encoded information in latent space must be decoded by a decoder 92 to obtain the reconstructed frame X. In this decoding stage, a latent representation a of side information A needs to be applied to the input of decoder 92. A latent representation a of side information A is provided by another encoder 93 and concatenated with the output of decoder 92.

[0124] In the context of video compression, a conditional codec is implemented to compress the residual used for inter-prediction of a current block of a current frame, as shown in Figure 10. The residual is calculated by subtracting the current block from its predicted version. The residual is encoded by an encoder 101 to obtain a residual bitstream. The residual bitstream is decoded by a decoder 102. A prediction block is obtained by a prediction unit 103 by using information from the previous frame / block. The prediction block has the same size and dimensions as the current block, and is therefore processed in a similar manner. The reconstructed residual is added to the prediction block to provide a reconstructed block.

[0125] The prior art constrained residual coding (CodeNet) is shown in Figure 11. The structure is similar to that shown in Figure 9. The constrained encoder 111 uses the predicted frame as side information to condition the codec.

number

number

number

number

number

[0126] CodeNet uses predicted frames, but does not use the explicit difference (residual) between the predicted frame and the current frame. Coding the current frame while taking all the information from the predicted frame can advantageously result in less information to be transmitted compared to residual coding.

[0127] However, CodeNet does not allow highly parallel processing because it involves entropy prediction, and it also requires a large memory space. The present disclosure can reduce memory demands and improve overall processing execution time.

[0128] This disclosure provides conditional coding in which a primary component of an image is encoded independently of one or more non-primary components, and the one or more non-primary components are encoded using information from the primary component. Here and below, the primary component may be a luma component and one or more non-primary components may be chroma components, or the primary component may be a chroma component and a single non-primary component may be a luma component. The primary component may be encoded and decoded independently of the non-primary component(s). This allows decoding even if the non-primary component(s) are lost for some reason. The one or more non-primary components may be jointly and concurrently encoded, and they may be encoded concurrently with the primary component. Decoding of the one or more non-primary components utilizes information from a latent representation of the primary component. Such conditional coding may be applied to intra-prediction and inter-prediction processing of video sequences. It may also be applied to still image coding.

[0129] Figure 12 illustrates the basics of constrained intra prediction according to an example embodiment. A tensor representation x of an input image / frame i is quantized and provided to encoding device 121. Note here, and in the following description, that the entire image or only a portion of the image, e.g., one or more blocks, slices, tiles, etc., may be coded.

[0130] Prior to the encoding device 121, a separation of the tensor representation x into a primary intra component and at least one non-primary (secondary) intra component is performed, where the primary intra component is converted into a primary intra component bitstream and the at least one non-primary intra component is converted into at least one non-primary intra component bitstream. These bitstreams represent compressed information about the components used by the decoding device 122 for reconstruction of the components. The two bitstreams can be interleaved with each other. The encoding device 121 may be referred to as a conditional color separation (CCS) encoding device. The encoding of the at least one non-primary intra component is based on information from the primary intra component, as described in more detail below. The respective bitstreams are decoded by the decoding device 122 to reconstruct the image / frame. The decoding of the at least one non-primary intra component is based on information from a latent representation of the primary intra component, as described in more detail below.

[0131] FIG. 13 illustrates the basics of residual coding according to an example embodiment. A tensor representation x′ of an input image / frame i′ is quantized, and a residual is calculated and provided to an encoding device 131. Prior to the encoding device 131, a separation of the residual into a primary residual component and at least one non-primary residual component is performed, where the primary residual component is converted into a primary residual component bitstream and the at least one non-primary residual component is converted into at least one non-primary residual component bitstream. The encoding device 131 may be referred to as a constrained color separation (CCS) encoding device. The encoding of the at least one non-primary residual component is based on information from the primary residual component, as will be described in more detail later. The respective bitstreams are decoded by a decoding device 132 to reconstruct the image / frame. The decoding of the at least one non-primary residual component is based on information from a latent representation of the primary residual component, as will be described in more detail later. The prediction required for the calculation of the residual and the reconstructed image / frame is provided by a prediction unit 133.

[0132] In the configurations shown in FIGS. 12 and 13, the encoding devices 121 and 131 and the decoding devices 131 and 132 may include or be connected to respective neural networks. The encoding devices 121 and 131 may include a variational autoencoder. Processing the primary component may involve a different number of channels / neural network layers compared to processing at least one non-primary component. The encoding devices 121 and 131 may determine the appropriate number of channels / neural network layers by performing an exhaustive search or in a content-adaptive manner. A set of models may be trained, where each model is based on a different number of channels for encoding the primary component and the non-primary component. During processing, the best-performing filter may be determined by the encoding devices 121 and 131. The neural networks of the encoding devices 121 and 131 may be trained collaboratively to determine the number of channels used to process the primary component and the non-primary component. In some applications, the number of channels used to process the primary components may be greater than the number of channels used to process the non-primary components. In other applications, for example, if the primary component signal is less noisy than the non-primary component signal, the number of channels used to process the primary components may be less than the number of channels used to process the non-primary components. In principle, the selection of the number of channels may be the result of optimizing for processing speed and signal distortion. Extra channels can reduce distortion but result in a higher processing load. Experiments have shown that a suitable number of channels may be, for example, 128 for the primary components and 64 for the non-primary components, or 128 for both the primary and non-primary components, or 192 for the primary components and 64 for the non-primary components.

[0133] The number of channels / neural network layers used for the encoding process may be implicitly or explicitly signaled to the decoding devices 122 and 132, respectively.

[0134] 14 shows an embodiment of the conditional coding of an image (a frame of a video sequence or a still image) in more detail. The encoder 141 generates a video signal of size H P ×W P ×C P where H P denotes the height dimension of the image, and W P denotes the width dimension of the image, and C P denotes the input channel dimension. In the following, a tensor with size A×B×C will usually be referred to simply as the tensor A×B×C for short.

[0135] Exemplary sizes of the height, width, and channel dimensions of the tensor output by the encoder 141 are H P / 16×W P / 16×128.

[0136] It should be noted that the encoders 141 and 142 may be included in the encoding devices 121 and 131 .

[0137] Based on the output of the encoder 141, i.e., the representation of the tensor representation of the primary components of the image in the latent space, a bitstream is generated and transformed back to the latent space to produce the recovered tensor

number

[0138] a tensor representation H of at least one non-primary component NP of the image NP ×W NP ×C NP (where HNP denotes the height dimension of the image and W NP denotes the width dimension of the image, and C NP denotes the input channel dimension) is a tensor representation H of the primary component P. P ×W P ×C P and then input to another encoder 142 (so that the other encoder 142 receives the tensor H NP ×W NP ×(C NP and +C P ) are input). Exemplary sizes of the height, width, and channel dimensions of the tensor output by encoder 142 are H P / 16×W P / 16×64 or H P / 32×W P / 32×64.

[0139] Before concatenation, the tensor representation H of the first-order component P P ×W P ×C P The sample locations of are represented by a tensor representation H of at least one non-primary component NP. NP ×W NP ×C NP This is the case when the sizes or sub-pixel offsets of the samples of those tensors differ from each other. Based on the output of the other encoder 142, i.e., the representation of the concatenated tensor image in latent space, a bitstream is generated and transformed back to latent space to produce the recovered concatenated tensor ̂H in latent space. NP ×^W NP ×^C NP is obtained.

[0140] On the first side, the recovered tensor ^H in the latent space P ×^W P ×^C P is the reconstructed tensor representation H P ×W P ×C P is input to a decoder 143 to reconstruct the primary component P of the image based on

[0141] Furthermore, in the latent space, the tensor ^H NP ×^W NP ×^C NP , tensor ^H P ×^W P ×^C P Again, if the samples of these tensors to be concatenated differ in size or sub-pixel offset, some adjustment of the sample positions is required. On the non-primary side, the tensor ^H resulting from this concatenation is NP ×^W NP ×(^C P +^C NP ) is input to another decoder 144. At least one non-primary component NP of the image is decoded into a reconstructed tensor representation H NP ×W NP ×C NP This is to reconstruct it based on the

[0142] The coding described above may be performed for the primary component P independently of at least one non-primary component NP. For example, the coding of the primary component P and at least one non-primary component NP may be performed concurrently. This may increase overall processing parallelism compared to current techniques. Furthermore, numerical experiments have shown that shorter channel lengths may be used, and therefore memory requirements may be reduced, compared to current techniques, without significant degradation in the quality of the reconstructed image.

[0143] In the following, an exemplary implementation of conditional coding of the components of an image represented in YUV space (one luma component Y and two chroma components U and V) is described with reference to Figures 15 to 20. It goes without saying that the disclosed conditional coding is also applicable to any other (color) space that may be used to represent an image.

[0144] In the embodiment shown in Figure 15, input data in YUV420 format is processed, where Y indicates the luma component of the current image to be processed, UV indicates the chroma components U and V of the current image to be processed, and 420 indicates that the size of the luma component Y in the height and width dimensions is four times larger than the size of the chroma components UV (twice as high and twice as wide). In the embodiment shown in Figure 15, Y is selected to be a primary component that is processed independently of UV, and UV is selected to be a non-primary component. The UV components are processed together.

[0145] The YUV representation of the image to be processed is separated into a (primary) Y component and a (non-primary) UV component. The encoder 151, which includes a neural network, receives a tensor representing the Y component of the image to be processed, with size (H / 1) × (W / 1) × 1, where H, W are the height and width dimensions, and the depth (i.e., number of channels) of the input is 1 (for one luma component). The output of the encoder 151 is a tensor of size (H / 16) × (W / 16) × C y where C y is the number of channels allocated to the Y component. In this embodiment, four downsampling layers in encoder 151 reduce (downsample) both the height and width of the input tensor by a factor of 16, reducing the number of channels C y is 128. The resulting latent representation of the Y component is processed by the hyperprior Y pipeline.

[0146] The UV component of the image to be processed is represented by the tensor (H / 2) × (W / 2) × 2. Again, H,W are the height and width dimensions, and the number of channels is 2 (for the two chroma components). Conditional encoding of the UV component requires auxiliary information from the Y component. If the plane size (H and W) of the Y component differs from the size of the UV component, a resampling unit is used to align the sample positions in the tensor representing the Y component with the sample positions in the tensor representing the UV component. Similarly, if there is an offset between the sample positions in the tensor representing the Y component and the sample positions in the tensor representing the UV component, an alignment must be performed.

[0147] The aligned tensor representation of the Y component is concatenated with the tensor representation of the UV component to obtain a tensor (H / 2)×(W / 2)×3. The encoder 152, which includes a neural network, converts this concatenated tensor into a latent tensor (H / 32)×(W / 32)×C. uv where C uv is the number of channels allocated to the UV components. In this embodiment, the five downsampling layers in encoder 152 reduce (downsample) both the height and width of the input tensor by a factor of 32, resulting in a number of channels of 64. The resulting latent representation of the UV components is processed by a Hyperprior UV pipeline, which is similar to the Hyperprior Y pipeline (see also the description of Figure 7 above for the operation of these pipelines). Note that both the Hyperprior UV pipeline and the Hyperprior Y pipeline may comprise neural networks.

[0148] The hyperprior Y pipeline provides the entropy model used for entropy coding of the (quantized) latent representation of the Y component. The hyperprior Y pipeline comprises a (hyper)encoder 153, an arithmetic encoder, an arithmetic decoder, and a (hyper)decoder 154.

[0149] Latent tensor (H / 16)×(W / 16)×C representing the Y component in the latent space y is further downsampled by the (hyper)encoder 153 via convolution and nonlinear transformation to obtain a hyperlatent tensor. The hyperlatent tensor is converted into a bitstream by the arithmetic encoder AE (possibly after quantization, not shown in FIG. 15; in fact, here and below, the quantization performed by the quantization unit Q, if any, is optional). The statistical properties of the (quantized) hyperlatent tensor are estimated by an entropy model, e.g., a factorized entropy model, and the arithmetic encoder AE in the hyperprior Y pipeline uses these statistical properties to create the bitstream. All elements of the (quantized) hyperlatent tensor can be written to the bitstream without the need for an autoregressive process.

[0150] The (factorized) entropy model serves as a codebook whose parameters are available at the decoder side. The arithmetic decoder AD of the hyperprior Y pipeline recovers the hyper-latent tensor from the bitstream by using the (factorized) entropy model. The recovered hyper-latent tensor is upsampled by the (hyper)decoder 154 by applying multiple convolution operations and nonlinear transformations. The latent tensor (H / 16) × (W / 16) × C, which represents the Y component in the latent space, is y is subjected to quantization by the quantization unit Q of the hyperprior Y pipeline, and the entropy of the quantized latent tensor is autoregressively estimated based on the upsampled reconstructed hyperlatent tensor output by the (hyper)decoder 154.

[0151] Latent tensor (H / 16)×(W / 16)×C representing the Y component in the latent space yis also quantized before being converted into a bitstream (that can be transmitted from the transmitter side to the receiver side) by another arithmetic encoder AE that uses the estimated statistical properties of that tensor provided by the hyperprior Y pipeline. The latent tensor (H / 16) × (W / 16) × C y is recovered from the bitstream by another arithmetic decoder AD using the autoregressive entropy model provided by the hyperprior Y pipeline. The recovered latent tensor (H / 16)×(W / 16)×C y is upsampled by decoder 155 by applying multiple convolution operations and nonlinear transformations to obtain a tensor representation of the reconstructed Y component of the image, with size (H / 1)×(W / 1)×1.

[0152] The hyperprior UV pipeline outputs the encoder 152, i.e., the latent tensor (H / 32) × (W / 32) × C. uv This latent tensor is further downsampled by the (hyper)encoder 156 of the hyperprior UV pipeline via convolution and nonlinear transformation to obtain a hyperlatent tensor, which is then converted to a bitstream by the arithmetic encoder AE of the hyperprior UV pipeline (possibly after quantization, not shown in FIG. 15 ). The statistical properties of the (quantized) hyperlatent tensor are estimated by an entropy model, e.g., a factorized entropy model, and the arithmetic encoder AE of the hyperprior Y pipeline uses these statistical properties to create the bitstream. All elements of the (quantized) hyperlatent tensor can be written to the bitstream without the need for an autoregressive process.

[0153] The (factorized) entropy model serves as a codebook whose parameters are available at the decoder side. The arithmetic decoder AD of the hyperprior UV pipeline recovers the hyperlatent tensor from the bitstream by using the (factorized) entropy model. The recovered hyperlatent tensor is upsampled by the (hyper)decoder 157 of the hyperprior UV pipeline by applying multiple convolution operations and nonlinear transformations. The latent tensor (H / 32) × (W / 32) × C representing the UV components is uv is subjected to quantization by the quantization unit Q of the hyperprior UV pipeline, and the entropy of the quantized latent tensor is autoregressively estimated based on the upsampled restored hyperlatent tensor output by the (hyper)decoder 157.

[0154] Latent tensor (H / 32)×(W / 32)×C representing the UV components in the latent space uv is also quantized before being converted into a bitstream (that can be transmitted from the transmitter side to the receiver side) by another arithmetic encoder AE that uses the estimated statistical properties of that tensor provided by the hyperprior UV pipeline. The latent tensor (H / 32) × (W / 32) × C represents the UV components in the latent space. uv The sigma-based decoder (AD) is then reconstructed from the bitstream by a separate arithmetic decoder (AD) using the autoregressive entropy model provided by the Hyperprior UV pipeline.

[0155] The recovered latent tensor (H / 32)×(W / 32)×C represents the UV components in the latent space. uv is the recovered latent tensor (H / 16)×(W / 16)×C y and the latter are concatenated after downsampling, i.e., the recovered latent tensor (H / 32) × (W / 32) × C uv is the tensor (H / 32)×(W / 32)×C y(as auxiliary information required for decoding UV components) and is concatenated into the tensor (H / 32) × (W / 32) × (C y +C uv ) is obtained, which is input to a decoder 158 on the UV processing side, which upsamples it by applying multiple convolution operations and nonlinear transformations to obtain a tensor representation of the reconstructed UV components of the image with size (H / 2) × (W / 2) × 2. The tensor representation of the reconstructed UV components of the image is combined with the tensor representation of the reconstructed Y component of the image to obtain a reconstructed image in YUV space.

[0156] FIG. 16 shows an embodiment similar to that shown in FIG. 15 , but for processing input data in YUV444 format, where the sizes of the tensors representing the Y and UV components, respectively, are the same in the height-width dimension. Encoder 161 converts the tensor (H / 1)×(W / 1)×1, representing the Y component of the image to be processed, into latent space. The auxiliary information does not need to be resampled according to this embodiment, so the tensor (H / 1)×(W / 1)×2, representing the UV component of the image to be processed, can be directly concatenated with the tensor (H / 1)×(W / 1)×1, representing the Y component, and the concatenated tensor (H / 1)×(W / 1)×3 is converted into latent space by UV-side encoder 162. The hyperprior Y pipeline, comprising (hyper)encoder 163 and (hyper)decoder 164, and the hyperprior UV pipeline, comprising (hyper)encoder 166 and (hyper)decoder 167, operate in a similar manner to that described above with reference to FIG. 15 . Since the recovered latent representations of the U and UV components have the same size in height and width, they can be concatenated together in the latent space without resampling. The recovered latent representation of the U component is (H / 16) × (w / 16) × C y is upsampled by the decoder 165 to obtain the recovered concatenated latent representations (H / 16)×(w / 16)×(C y +C uv) is upsampled by decoder 168 and the outputs of decoders 165 and 168 are combined to obtain the reconstructed image in YUV space.

[0157] Figures 17 and 18 illustrate an embodiment in which constrained residual coding is provided. Residual constrained coding may be used for inter-prediction or still image coding of a current frame of a video sequence. Unlike the embodiments shown in Figures 15 and 16, the residual is processed with residual components in YUV space. The residual is separated into a residual Y component for the Y component and a residual UV component for the UV component. The processing of the residual components is similar to the processing of the Y and UV components described above with reference to Figures 15 and 16. According to the embodiment shown in Figure 17, the input data is in YUV 420 format. Therefore, the residual Y component must be downsampled before being concatenated with the residual UV component. Encoders 171 and 172 provide respective latent representations. The hyperprior Y pipeline, comprising (hyper)encoder 173 and (hyper)decoder 174, and the hyperprior UV pipeline, comprising (hyper)encoder 176 and (hyper)decoder 177, operate in the same manner as described above with reference to Figure 15. On the residual Y component side, decoder 175 outputs a reconstructed representation of the residual Y component. On the residual UV side, decoder 178 outputs a reconstructed representation of the residual UV component based on auxiliary information provided in latent space, where downsampling of the reconstructed latent representation of the residual Y component is required. The outputs of decoders 175 and 178 are combined to obtain a reconstructed residual in YUV space that can be used to obtain (part of) a reconstructed image.

[0158] According to the embodiment shown in Fig. 18, the input data is in YUV 444 format. No downsampling of the side information is required. The processing of the residual Y and UV components is similar to that of the Y and UV components described above with reference to Fig. 16. The encoder 181 converts the tensor (H / 1) x (W / 1) x 1 representing the residual Y component of the image to be processed into latent space. The tensor (H / 1) x (W / 1) x 2 representing the residual UV component of the image to be processed can be directly concatenated with the tensor (H / 1) x (W / 1) x 1 representing the residual Y component, and the concatenated tensor (H / 1) x (W / 1) x 3 is converted into latent space by the residual UV side encoder 182.

[0159] The hyperprior Y pipeline, comprising (hyper)encoder 183 and (hyper)decoder 184, and the hyperprior UV pipeline, comprising (hyper)encoder 186 and (hyper)decoder 187, operate in a similar manner as described above with reference to FIG. 15.

[0160] Since the reconstructed latent representations of the residual U and UV components have the same size in height and width, they can be concatenated together without resampling. The reconstructed latent representation of the residual U component is (H / 16) × (w / 16) × C y is upsampled by the decoder 185 to produce the recovered concatenated latent representation of the residual Y and UV components (H / 16)×(w / 16)×(C y +C uv ) is upsampled by decoder 188 and the outputs of decoders 185 and 188 are combined to obtain a reconstructed residual of the image in YUV space that can be used to obtain (part of) the reconstructed image.

[0161] Figure 19 shows an alternative embodiment to that shown in Figure 17. The only difference is that in the configuration shown in Figure 19, an autoregressive entropy model is not used. The representation of the residual Y component, represented by the tensor (H / 1) x (w / 1) x 1, is transformed in latent space by encoder 191. The residual Y component is used as side information for coding the residual UV components, represented by the tensor (H / 2) x (w / 2) x 2, by encoder 192, which outputs the tensor (H / 2) x (w / 2) x 3. The hyperprior Y pipeline, including (hyper)encoder 193 and (hyper)decoder 194, then converts the latent representation of the residual Y component, (H / 16) x (w / 16) x C, into a latent space. y The decoder 195 outputs the reconstructed residual Y component represented by the tensor (H / 1)×(w / 1)×1. The hyperprior UV pipeline, including the (hyper)encoder 196 and the (hyper)decoder 197, generates a latent representation of the tensor (H / 2)×(w / 2)×3 output by the encoder 192, i.e., the tensor (H / 32 )×(w / 32 )×C uv The decoder 198 provides side information used for coding the connected tensor (H / 32)×(w / 32)×(C y +C uv ) and outputs the reconstructed residual UV components represented by the tensor (H / 2)×(w / 2)×2.

[0162] Figure 20 shows an alternative embodiment to that shown in Figure 18. Again, the only difference is that the configuration shown in Figure 20 does not use an autoregressive entropy model.

[0163] The representation of the residual Y component, represented by the tensor (H / 1)×(w / 1)×1, is transformed in latent space by encoder 201. The residual Y component is used by encoder 202 as side information for coding the residual UV components, represented by the tensor (H / 2)×(w / 2)×2, which outputs a tensor (H / 2)×(w / 2)×3. A hyperprior Y pipeline, including (hyper)encoder 203 and (hyper)decoder 204, transforms the latent representation of the residual Y component, (H / 16)×(w / 16)×C y The decoder 205 outputs the reconstructed residual Y component represented by the tensor (H / 1)×(w / 1)×1. The hyperprior UV pipeline, including the (hyper)encoder 206 and the (hyper)decoder 207, generates a latent representation of the tensor (H / 2)×(w / 2)×3 output by the encoder 202, i.e., the tensor (H / 16)×(w / 16)×C. uv The decoder 208 provides the reconstructed representation of the residual UV components in latent space, (H / 16)×(w / 16)×C uv and outputs the reconstructed residual UV components represented by the tensor (H / 2)×(w / 2)×2.

[0164] Processing without using the autoregressive entropy model can reduce the overall processing complexity and still provide sufficient accuracy of the restored image, depending on the actual application.

[0165] Specific embodiments of a method for encoding at least a portion of an image are shown in FIGS. 21 and 22, and specific embodiments of a method for reconstructing at least a portion of an image are shown in FIGS. 23 and 24.

[0166] The method for encoding at least a portion of an image shown in FIG. 21 includes step S212 of encoding a primary component of the image independently of at least one secondary (non-primary) component of the image, and step S214 of encoding at least one secondary component of the image using information from the primary component. The primary component provides auxiliary information for the process of encoding the secondary component. In the embodiment shown in FIGS. 22-24, the image includes a luma component and a chrominance component, one of which is selected to be the primary component and at least one of the other components is selected to be the at least one secondary component. For example, in YUV space, the Y component is selected as the primary component and one or both of the chrominance components U and V are selected as secondary components. Alternatively, one of the chrominance components U and V is selected as the primary component and the luma component Y is selected as the secondary component.

[0167] The method of encoding at least a portion of an image shown in Figure 22 includes providing a residual including a first-order residual component for a first-order component of the image and at least one second-order residual component for at least one second-order component of the image that is different from the first-order component (S222). The first-order residual component is encoded independently of the at least one second-order residual component (S224), and the at least one second-order residual component is encoded using information from the first-order residual component (S226).

[0168] 23, a method for reconstructing at least a portion of an image includes processing a first bitstream based on a first entropy model (S232) to obtain a first latent tensor, and processing the first latent tensor (S234) to obtain a first tensor representing a first-order component of the image. Further, a second bitstream different from the first bitstream is processed based on a second entropy model different from the first entropy model (S236) to obtain a second latent tensor different from the first latent tensor, and processing the second latent tensor using information from the first latent tensor (S238) to obtain a second tensor representing at least one second-order component of the image.

[0169] 24, a method for reconstructing at least a portion of an image includes processing a first bitstream based on a first entropy model (S242) to obtain a first latent tensor, and processing the first latent tensor (S244) to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image. Further, a second bitstream different from the first bitstream is processed based on a second entropy model different from the first entropy model (S246) to obtain a second latent tensor different from the first latent tensor, and processing the second latent tensor using information from the first latent tensor (S248) to obtain a second tensor representing at least one second-order residual component of a residual for at least one second-order component of the image.

[0170] The methods illustrated in Figures 21-24 may be applied in the context of intra-prediction, inter-prediction, and / or still image coding, as appropriate. Furthermore, in specific implementations, the methods illustrated in Figures 21-24 may utilize the processes (units) described with reference to Figures 12-20.

[0171] In particular, the methods illustrated in FIGS. 21-24 may be implemented in a processing device 250 including processing circuitry 255 configured to perform the steps of these methods as illustrated in FIG.

[0172] Thus, the processing device 250 may be a processing device 250 for encoding at least a portion of an image, the processing device 250 comprising a processing circuit 255 configured to encode a primary component of the image independently from at least one secondary component of the image (for at least said portion of the image) and to encode at least one secondary component of the image using information from the primary component (for at least said portion of the image).

[0173] Alternatively, the processing device 250 may be a processing device 250 that encodes at least a portion of an image, the processing device 250 comprising processing circuitry 255 configured to provide a residual comprising a first-order residual component for a first-order component of the image and at least one second-order residual component for at least one second-order component of the image that is different from the first-order component, to encode the first-order residual component independently from the at least one second-order residual component, and to encode the at least one second-order residual component using information from the first-order residual component.

[0174] Alternatively, the processing device 250 may be a processing device 250 for reconstructing at least a portion of an image, the processing device 250 comprising processing circuitry 255 configured to: process a first bitstream based on a first entropy model to obtain a first latent tensor; process the first latent tensor to obtain a first tensor representing a first component of the image; process a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; and process the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second component of the image.

[0175] Alternatively, the processing device 250 may be a processing device 250 for reconstructing at least a portion of an image, the processing device 250 comprising processing circuitry 255 configured to: process a first bitstream based on a first entropy model to obtain a first latent tensor; process the first latent tensor to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image; process a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; and process the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order residual component of the residual for at least one second-order component of the image.

[0176] Some Example Implementations in Hardware and Software A corresponding system in which the above-described encoder-decoder processing chain may be deployed is shown in FIG. 26. FIG. 26 is a schematic block diagram illustrating an example coding system, e.g., a video, image, audio, and / or other coding system (or coding system, for short), that may utilize the techniques of the present application. The video encoder 20 (or encoder 20, for short) and the video decoder 30 (or decoder 30, for short) of the video coding system 10 represent examples of devices that may be configured to perform the techniques according to various examples described herein. For example, the video coding and decoding may employ neural networks such as those shown in FIGS. 1-6, which may be distributed and may apply the above-described bitstream parsing and / or bitstream generation to communicate feature maps among the distributed computational nodes (two or more).

[0177] As shown in FIG. 26, the coding system 10 includes a source device 12 configured to provide encoded picture data 21 to, for example, a destination device 14 for decoding the encoded picture data 13.

[0178] The source device 12 comprises an encoder 20 and may further, i.e. optionally, comprise a picture source 16 , a pre-processor (or pre-processing unit) 18 , for example a picture pre-processor 18 , and a communication interface or unit 22 .

[0179] Picture source 16 may comprise or be any kind of picture capture device, such as, for example, a camera that captures real-world pictures, and / or any kind of picture generation device, such as, for example, a computer graphics processor for generating computer-animated pictures, or any kind of other device that acquires and / or provides real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any kind of memory or storage device that stores any of the aforementioned pictures.

[0180] To distinguish between the pre-processing unit 18 and the processing performed by the pre-processing unit 18, the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17.

[0181] The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the preprocessing unit 18 may be an optional component. It is noted that the preprocessing may also employ a neural network (such as in any of FIGS. 1-7) that uses presence indicator signaling.

[0182] A video encoder 20 is configured to receive the pre-processed picture data 19 and provide encoded picture data 21 .

[0183] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) over the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0184] The destination device 14 comprises a decoder 30 (e.g., a video decoder 30), and may additionally, i.e., optionally, comprise a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0185] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g., directly from the source device 12 or from any other source, e.g., a storage device, e.g., a storage device for the encoded picture data, and to provide the encoded picture data 21 to a decoder 30.

[0186] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0187] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, e.g., packets, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0188] The communications interface 28, which forms a counterpart of the communications interface 22, may for example be configured to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0189] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in Figure 26, or as bidirectional communication interfaces, e.g., configured to send and receive messages, e.g., set up connections, receive, acknowledge, and exchange any other information related to the communication link and / or data transmission, e.g., transmission of encoded picture data. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or a decoded picture 31 (e.g., using a neural network based on one or more of Figures 1-7).

[0190] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by, for example, a display device 34.

[0191] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, e.g., to display the picture to a user or viewer. Display device 34 may be or have any type of display for displaying the reconstructed picture, e.g., an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0192] 26 depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices may include both or both functionality, i.e., source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0193] As will be apparent to those skilled in the art based on this document, the presence and (exact) division of functions or features of various units within source device 12 and / or destination device 14 as shown in FIG. 26 may vary depending on the actual device and application.

[0194] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to embody various modules, including neural networks such as those shown in any of FIGS. 1-6 or portions thereof. Decoder 30 may be implemented via processing circuitry 46 to embody various modules discussed with respect to FIGS. 1-7 and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the techniques are implemented partially in software, a device may store instructions for the software on a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 or video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example as shown in FIG.

[0195] Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of handheld or stationary device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0196] In some cases, the video coding system 10 shown in FIG. 26 is merely an example, and the techniques herein may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory, streamed over a network, or the like. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that simply encode data to memory and / or retrieve data from memory and decode it without communicating with each other.

[0197] 28 is a schematic diagram of a video coding device 2000 according to an embodiment of the present invention. The video coding device 2000 is suitable for implementing the disclosed embodiments as described herein. In an embodiment, the video coding device 2000 may be a decoder, such as the video decoder 30 of FIG. 26, or an encoder, such as the video encoder 20 of FIG. 26.

[0198] The video coding device 2000 comprises an ingress port 2010 (or input port 2010) and a receiver unit (Rx) 2020 for receiving data; a processor, logic unit, or central processing unit (CPU) 2030 for processing the data; a transmitter unit (Tx) 2040 and an egress port 2050 (or output port 2050) for transmitting the data; and a memory 2060 for storing the data. The video coding device 2000 may also comprise optical-electrical (OE) and electro-optical (EO) components coupled to the ingress port 2010, the receiver unit 2020, the transmitter unit 2040, and the egress port 2050 for the egress or ingress of optical or electrical signals.

[0199] The processor 2030 is implemented in hardware and software. The processor 2030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 2030 communicates with the ingress port 2010, the receiver unit 2020, the transmitter unit 2040, the egress port 2050, and the memory 2060. The processor 2030 includes a coding module 2070. The coding module 2070 implements the disclosed embodiments described above. For example, the coding module 2070 performs, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 2070 provides substantial improvements to the functionality of the video coding device 2000 and results in the transformation of the video coding device 2000 into different states. Alternatively, the coding module 2070 is implemented as instructions stored in the memory 2060 and executed by the processor 2030.

[0200] The memory 2060 may comprise one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device for storing programs when such programs are selected for execution, as well as for storing instructions and data read during program execution. The memory 2060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0201] FIG. 29 is a simplified block diagram of an apparatus 800 that may be used as either or both of the source device 12 and the destination device 14 from FIG. 26, according to an exemplary embodiment.

[0202] The processor 2102 in the device 2100 may be a central processing unit. Alternatively, the processor 2102 may be any other type of device or devices, now existing or later developed, that are capable of manipulating or processing information. While the disclosed embodiments may be implemented using a single processor as shown, e.g., processor 2102, advantages in speed and efficiency may be achieved using more than one processor.

[0203] The memory 2104 in the apparatus 2100 may, in some implementations, be a read-only memory (ROM) device or a random-access memory (RAM) device. Any other suitable type of storage device may be used as the memory 2104. The memory 2104 may include code and data 2106 that is accessed by the processor 2102 using a bus 2112. The memory 2104 may further include an operating system 2108 and application programs 2110, which include at least one program that enables the processor 2102 to perform the methods described herein. For example, the application programs 2110 may include applications 1 through N, which may further include a video coding application that performs the methods described herein.

[0204] The apparatus 2100 may also include one or more output devices, such as a display 2118. The display 2118, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 2118 may be coupled to the processor 2102 via the bus 2112.

[0205] Although shown here as a single bus, bus 2112 of device 2100 may be comprised of multiple buses. Additionally, secondary storage may be directly coupled to other components of device 2100 or accessed over a network, and may comprise a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, device 2100 may be implemented in a wide variety of configurations.

[0206] Furthermore, the processing device 250 shown in FIG. 25 may include the source device 12 or destination device 14 shown in FIG. 26, the video coding system 40 shown in FIG. 27, the video coding device 2000 shown in FIG. 28, or the device 2100 shown in FIG. 29.

Claims

1. 1. A method for reconstructing at least a portion of an image, comprising: processing the first bitstream based on a first entropy model to obtain a first latent tensor (S232); processing the first latent tensor to obtain a first tensor representing a first component of the image (S234); processing a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor (S236); and processing the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one quadratic component of the image (S238). method.

2. The method of claim 1 , wherein the first latent tensor is processed independently from the processing of the second latent tensor.

3. The method of claim 1 or 2, wherein the primary component of the image is a luma component and the at least one secondary component of the image is a chroma component.

4. The method of claim 1 or 2, wherein the primary component of the image is a chroma component and the at least one secondary component of the image is a luma component.

5. The method of claim 3 , wherein the second tensor represents two quadratic components, one of which is a chroma component and the other of which is another chroma component.

6. Processing the first latent tensor (S234) includes transforming the first latent tensor into the first tensor; Processing the second latent tensor (S238) includes concatenating the second latent tensor with the first latent tensor to obtain a concatenated tensor, and converting the concatenated tensor to the second tensor.

5. The method according to any one of claims 1 to 4.

7. each of the first and second latent tensors has height and width dimensions; Processing the first latent tensor (S234) includes transforming the first latent tensor into the first tensor; The processing of the second latent tensor (S238) includes determining whether a size or sub-pixel offset of samples of the second latent tensor in at least one of height and width dimensions differs from a size or sub-pixel offset of samples of the first latent tensor in at least one of height and width dimensions, and adjusting sample positions of the first latent tensor to match the sample positions of the second latent tensor when it is determined that the size or sub-pixel offset of samples of the second latent tensor differs from the size or sub-pixel offset of samples of the first latent tensor, thereby obtaining an adjusted first latent tensor; concatenate the second latent tensor and the adjusted first latent tensor to obtain a concatenated latent tensor only if it is determined that a size or sub-pixel offset of samples of the second latent tensor is different from a size or sub-pixel offset of samples of the first latent tensor, otherwise concatenate the second latent tensor and the first latent tensor to obtain a concatenated latent tensor; transforming the concatenated latent tensor into the second tensor.

6. The method according to any one of claims 1 to 5.

8. 7. The method of claim 1, wherein the first bitstream is processed by a first neural network and the second bitstream is processed by a second neural network different from the first neural network.

9. 8. The method of claims 6 and 7, wherein the first latent tensor is transformed by a third neural network distinct from the first and second networks, and the concatenated latent tensor is transformed by a fourth neural network distinct from the first, second, and third networks.

10. 10. The method of claim 1, wherein the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and a size of the first latent tensor in the channel dimension is one of larger than, smaller than, and equal to a size of the second latent tensor in the channel dimension.

11. 11. The method of claim 10, wherein processing the first bitstream includes obtaining information signaled in the first bitstream about the size of the first latent tensor in a channel dimension, and processing the second bitstream includes obtaining information signaled in the second bitstream about the size of the second latent tensor in a channel dimension.

12. 1. A method for reconstructing at least a portion of an image, comprising: processing the first bitstream based on a first entropy model to obtain a first latent tensor (S242); processing the first latent tensor to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image (S244); processing a second bitstream different from the first bitstream based on a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor (S246); and processing the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order residual component of the residual for at least one second-order component of the image (S248). method.

13. The method of claim 12 , wherein the first latent tensor is processed independently of the processing of the second latent tensor.

14. 14. The method of claim 12 or 13, wherein the primary component of the image is a luma component and the at least one secondary component of the image is a chroma component.

15. 14. The method of claim 12 or 13, wherein the primary component of the image is a chroma component and the at least one secondary component of the image is a luma component.

16. 15. The method of claim 14, wherein the second tensor represents two residual components for two secondary components, one of which is a chroma component and the other of which is another chroma component.

17. Processing the first latent tensor (S244) includes transforming the first latent tensor into the first tensor; Processing the second latent tensor (S248) includes concatenating the second latent tensor with the first latent tensor to obtain a concatenated tensor and transforming the concatenated tensor into the second tensor.

17. The method according to any one of claims 12 to 16.

18. each of the first and second latent tensors has height and width dimensions; Processing the first latent tensor (S244) includes transforming the first latent tensor into the first tensor; The processing of the second latent tensor (S248) includes determining whether a size or sub-pixel offset of samples of the second latent tensor in at least one of height and width dimensions differs from a size or sub-pixel offset of samples of the first latent tensor in at least one of height and width dimensions, and adjusting sample positions of the first latent tensor to match the sample positions of the second latent tensor when it is determined that the size or sub-pixel offset of samples of the second latent tensor differs from the size or sub-pixel offset of samples of the first latent tensor, thereby obtaining an adjusted first latent tensor; concatenate the second latent tensor and the adjusted first latent tensor to obtain a concatenated latent tensor only if it is determined that a size or sub-pixel offset of samples of the second latent tensor is different from a size or sub-pixel offset of samples of the first latent tensor, otherwise concatenate the second latent tensor and the first latent tensor to obtain a concatenated latent tensor; transforming the concatenated latent tensor into the second tensor.

17. The method according to any one of claims 12 to 16.

19. 19. The method of claim 12, wherein the first bitstream is processed by a first neural network and the second bitstream is processed by a second neural network different from the first neural network.

20. 20. The method of claim 19, wherein the first latent tensor is transformed by a third neural network distinct from the first and second networks, and the concatenated latent tensor is transformed by a fourth neural network distinct from the first, second, and third networks.

21. 21. The method of claim 12, wherein the first latent tensor includes a channel dimension, the second latent tensor includes a channel dimension, and a size of the first latent tensor in the channel dimension is one of greater than, less than, and equal to a size of the second latent tensor in the channel dimension.

22. 22. The method of claim 21 , wherein processing the first bitstream includes obtaining information signaled in the first bitstream about the size of the first latent tensor in a channel dimension, and processing the second bitstream includes obtaining information signaled in the second bitstream about the size of the second latent tensor in a channel dimension.

23. A computer program stored on a non-transitory medium having code for performing the steps of the method of any one of claims 1 to 22 when the computer program is run on one or more processors.

24. A processing device (40, 250, 2000, 2100) for reconstructing at least a portion of an image, the processing device (40, 250, 2000, 2100) comprising: one or more processors (43, 255, 2030, 2102); a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the apparatus to perform the method of any one of claims 1 to 22. Processing equipment.

25. A processing device (250) for reconstructing at least a portion of an image, the processing device (40, 250, 2000, 2100) comprising: processing the first bitstream based on a first entropy model to obtain a first latent tensor; processing the first latent tensor to obtain a first tensor representing the first component of the image; processing a second bitstream different from the first bitstream according to a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; Using information from the first latent tensor, process the second latent tensor to obtain a second tensor representing the at least one secondary component of the image. a processing circuit (255) configured to Processing equipment.

26. A processing device (40, 250, 2000, 2100) for reconstructing at least a portion of an image, the processing device comprising: processing the first bitstream based on a first entropy model to obtain a first latent tensor; processing the first latent tensor to obtain a first tensor representing a first-order residual component of a residual for a first-order component of the image; processing a second bitstream different from the first bitstream according to a second entropy model different from the first entropy model to obtain a second latent tensor different from the first latent tensor; Processing the second latent tensor using information from the first latent tensor to obtain a second tensor representing at least one second-order residual component of the residual for at least one second-order component of the image. a processing circuit (255) configured to Processing equipment.

Images