Resampling in image compression

Abstract

The invention relates to tensor resampling in image compression of images. The invention provides a method for reconstructing at least one part of an image, and the method comprises the steps: analyzing a first code stream, so as to obtain a first prior tensor; the first priori tensor is processed to obtain a first tensor representing a major component of the image. The method further comprises the following steps: analyzing a second code stream different from the first code stream to obtain a second prior tensor different from the first prior tensor; performing resampling on the first prior tensor according to an integer factor to obtain a resampled first prior tensor; and obtaining a second tensor representing at least one secondary component of the image according to the second prior tensor and the resampled first prior tensor.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to International Application No. PCT / EP2023 / 067439, filed on June 27, 2023. The entire disclosure of the aforementioned patent application is incorporated herein by reference. Technical Field

[0003] This disclosure generally relates to the field of image and video decoding, and more particularly to image and video decoding including resampling in image compression. Background Technology

[0004] Video decoding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital TV, video transmission over the Internet and mobile networks, real-time conversational applications such as video chat and video conferencing, DVD and Blu-ray discs, video content capture and editing systems, and portable cameras for security applications.

[0005] Even relatively short videos require a large amount of video data to describe, which can be challenging when streaming or otherwise transmitting data over communication networks with limited bandwidth. Therefore, video data is typically compressed before transmission over modern telecommunications networks. The size of the video can also be an issue when storing it on storage devices due to potentially limited memory resources. Video compression devices typically decode the video data at the source using software and / or hardware before transmitting or storing it, thereby reducing the amount of data required to represent the digital video image. A video decompression device, which then decodes the video data, receives the compressed data at the destination. Compression techniques also apply to the context of still image decoding.

[0006] Given the limited network resources and the ever-growing demand for higher video quality, there is a need to improve compression and decompression technologies that can increase compression ratios with minimal impact on image quality.

[0007] Today, neural networks (NN) and deep learning (DL) technologies that use artificial neural networks have been used for some time in the fields of encoding and decoding videos, images (e.g., still images).

[0008] The aim is to further improve the efficiency of this image decoding (video decoding or still image decoding) based on a network trained to take into account limitations in available memory and / or processing speed.

[0009] Specifically, traditional image compression is costly and causes actual damage to compression performance. Summary of the Invention

[0010] This invention relates to methods and apparatus for decoding image or video data, particularly using neural networks such as the neural networks described below. Using neural networks allows for reliable encoding and decoding and the estimation of entropy models in a self-learning manner, thereby enabling high-precision reconstruction of images from compressed input data.

[0011] The foregoing and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0012] According to a first aspect, a method for reconstructing at least a portion of an image is provided, the method comprising: (for the at least a portion of the image) parsing a first bitstream to obtain a first prior tensor; processing the first prior tensor to obtain a first tensor representing a major component of the image. Furthermore, the method comprises: (for the at least a portion of the image) parsing a second bitstream different from the first bitstream to obtain a second prior tensor different from the first prior tensor; resampling the first prior tensor according to an integer factor to obtain a resampled first prior tensor; and obtaining a second tensor representing at least one minor component of the image based on the second prior tensor and the resampled first prior tensor.

[0013] Because integer factors are used to obtain the first prior tensor after resampling, this is highly beneficial in terms of both compression performance and simplicity.

[0014] In principle, an image can be a still image or an intra-frame of a video sequence. In this and the following description, it should be understood that an image comprises multiple components, particularly luminance and chrominance components. A component can be considered as the dimension of an orthogonal basis describing a full-color image. For example, when an image is represented in YUV space, the components are luminance Y, chrominance U, and chrominance V. One component of the image is selected as the principal component, and one or more of the remaining components are selected as one or more minor components (non-principal components). The terms "minor component" and "non-principal component" are used interchangeably herein to refer to the component decoded using the auxiliary information provided by the principal component. Encoding and decoding one or more minor components using the auxiliary information provided by the principal component results in a highly accurate reconstructed image after decoding.

[0015] The processing of the first prior tensor can be independent of the processing of the second prior tensor. In fact, even if the data of the minor components is lost, the encoded major components can be restored. Since the first and second bitstreams can be processed in parallel, this method can reliably and quickly reconstruct the compressed original image data.

[0016] In one implementation, the primary component of the image is a luminance component, and the at least one secondary component of the image is a chrominance component. For example, two secondary components of the image are decoded simultaneously, with one component being a chrominance component and the other being a different chrominance component. In another implementation, the primary component of the image is a chrominance component, and the at least one secondary component of the image is a luminance component. This provides a high degree of flexibility in which one component is actually adjusted by another component.

[0017] Overall decoding can include processing in the prior space, especially processing downsampled input data, thus enabling fast processing with a lower processing load. It should be noted that the terms "downsampling" and "upsampling" here refer to reducing and increasing the size of the tensor representation of the data, respectively.

[0018] The size of the first prior tensor in either the height or width dimension can be smaller than the corresponding size of the first tensor in either the height or width dimension, and / or the size of the second prior tensor in either the height or width dimension can be smaller than the corresponding size of the cascaded tensor in either the height or width dimension. For example, a reduction factor of 16 or 32 can be used in the height and / or width dimensions.

[0019] It is possible that the size or subpixel offset of the sample in the second tensor at least in the height and width dimensions of the tensor is different from the size or subpixel offset of the sample in the first tensor at least in the height and width dimensions.

[0020] Similarly, the size of the first prior tensor in either the height or width dimension can be smaller than the corresponding size in the height or width dimension of the first tensor, and / or the size of the second prior tensor in either the height or width dimension can be smaller than the corresponding size in the height or width dimension of the cascaded tensor. For example, a reduction factor of 16 or 32 can be used in the height and / or width dimensions. Adjusting the sample positions of the first tensor to match the sample positions of the second tensor can include, for example, performing downsampling with a factor of 2 in the width and height of the first tensor.

[0021] According to one implementation, the first prior tensor includes a channel dimension, the second prior tensor includes a channel dimension, and the size of the first prior tensor in the channel dimension is greater than, less than, or equal to the size of the second prior tensor in the channel dimension. If the primary component is considered significantly more important than one or more secondary components (which is often the case), the channel length of the primary component can be greater than the channel length of one or more secondary components. If the signal of the primary component is relatively sharp, while the signal of one or more non-primary components is relatively noisy, the channel length of the primary component can be less than the channel length of one or more secondary components. Numerical experiments show that shorter channel lengths compared to existing techniques can be used without significantly reducing the quality of the reconstructed image, thus reducing memory requirements.

[0022] Typically, a first tensor can be transformed into a first prior tensor using a first neural network, and a cascaded tensor can be transformed into a second prior tensor using a second neural network different from the first neural network. In this case, the first and second neural networks can be trained collaboratively to determine the size of the first and second prior tensors in the channel dimension. The channel length can be determined through exhaustive search or in a content-adaptive manner. A set of models can be trained, where each model decodes the primary and non-primary components based on a different number of channels. Therefore, the neural network may be able to optimize the channel lengths involved.

[0023] The determined channel length must also be used by the decoder to reconstruct the coded components. Therefore, according to one implementation, the size of the first prior tensor in the channel dimension can be indicated in the first bitstream, and the size of the second prior tensor in the channel dimension can be indicated in the second bitstream. The indication can be explicit or implicit, and the channel length can be directly communicated to the decoder in a bit-saving manner.

[0024] In one implementation, the first bitstream is parsed according to a first entropy model, while the second bitstream is parsed according to a second entropy model, which differs from the first entropy model. Such entropy models can reliably estimate the statistical properties used in converting the tensor representation of data into a bitstream.

[0025] The disclosed method can be advantageously implemented in the context of a super-prior architecture that provides side information useful for decoding a portion of an image to improve the accuracy of reconstructing that portion.

[0026] Resampling can be performed without interpolation (a computationally expensive operation), which can improve decoding efficiency accordingly.

[0027] Integer factors can be calculated based on a first scaling factor of the primary component and a second scaling factor of at least one secondary component. For example, the integer factor is calculated as follows: ,in, Indicates the first scaling factor. This represents the second scaling factor.

[0028] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0029] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0030] As another example, integer factors It can exist in the bitstream or be set as the default value.

[0031] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0032] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0033] Then, Export as ,in, This represents the scaling factor for at least one minor component.

[0034] According to one implementation of the first aspect, the first scaling factor may include a first horizontal scaling factor and a first vertical scaling factor. Similarly, the second scaling factor may include a second horizontal scaling factor and a second vertical scaling factor.

[0035] The resampling is performed through nearest neighbor upsampling or nearest neighbor downsampling.

[0036] For nearest neighbor upsampling:

[0037] Represented as The size received by this layer is Tensor input, and output size is The tensor output.

[0038] This downsampling does not require interpolation; simply copy it.

[0039]

[0040] Where s represents an integer factor.

[0041] For nearest neighbor downsampling:

[0042] Represented as The layer receives a size of Tensor input, and output size is The tensor output, through this process, reduces the spatial resolution of each tensor channel.

[0043] This downsampling does not require interpolation; simply copy it.

[0044]

[0045] Where s represents an integer factor.

[0046] According to one implementation of the method of the first aspect, processing the first prior tensor includes transforming the first prior tensor into the first tensor.

[0047] According to one implementation of the method of the first aspect, obtaining the second tensor representing at least one minor component of the image includes:

[0048] The second prior tensor is concatenated with the resampled first prior tensor to obtain a concatenated tensor, and the concatenated tensor is transformed back into the second tensor. At least one of these transformations may include upsampling. Therefore, processing in the prior space can be performed at a lower resolution, as this is necessary for accurate component reconstruction in YUV space or any other space suitable for image representation.

[0049] According to another implementation of the method in the first aspect, each of the first prior tensor and the second prior tensor has a height and a width dimension, and processing the first prior tensor includes transforming the first prior tensor into the first tensor, and processing the second prior tensor includes determining whether the size or subpixel offset of a sample in at least one of the height and width dimensions of the second prior tensor is different from the size or subpixel offset of a sample in at least one of the height and width dimensions of the first prior tensor. When it is determined that the size or subpixel offset of a sample in the second prior tensor is different from the size or subpixel offset of a sample in the first prior tensor, the sample position of the first prior tensor is adjusted to match the sample position of the second prior tensor. This yields the adjusted first prior tensor. Furthermore, when the sample size or sub-pixel offset of the second prior tensor is different from that of the sample size or sub-pixel offset of the first prior tensor, the second prior tensor and the adjusted first prior tensor are concatenated to obtain a concatenated prior tensor; otherwise, the second prior tensor and the first prior tensor are concatenated to obtain a concatenated prior tensor, and the concatenated prior tensor is transformed into the second tensor.

[0050] The first bitstream can be processed by the first neural network, and the second bitstream can be processed by the second neural network, which is different from the first neural network. The first prior tensor can be transformed by the third neural network, which is different from the first and second networks, and the cascaded prior tensor can be transformed by the fourth neural network, which is different from the first, second, and third networks.

[0051] According to another implementation of the method described in the first aspect, the first prior tensor includes a channel dimension, the second prior tensor includes a channel dimension, and the size of the first prior tensor in the channel dimension is greater than, less than, or equal to the size of the second prior tensor in the channel dimension. Information regarding the size of the first and second prior tensors in the channel dimension can be obtained from information indicated in the first and second bitstreams, respectively.

[0052] According to another implementation of the method of the first aspect, the first tensor represents the main residual component of the residual of the main component of the image, and the second tensor uses information from the first prior tensor to represent at least one minor residual component of the residual of at least one minor component of the image.

[0053] According to a second aspect, a method for reconstructing at least a portion of an image is provided, the method comprising: (for the at least a portion of the image) for example, parsing a first bitstream according to a first entropy model to obtain a first prior tensor; processing the first prior tensor to obtain a first tensor representing a principal residual component of the residuals of the principal components of the image. Furthermore, the method comprises: (for the at least a portion of the image) for example, parsing a second bitstream different from the first bitstream according to a second entropy model different from the first entropy model to obtain a second prior tensor different from the first prior tensor; resampling the first prior tensor according to an integer factor to obtain a resampled first prior tensor; and obtaining a second tensor representing at least one minor residual component of the residuals of at least one minor component of the image based on the second prior tensor and the resampled first prior tensor.

[0054] Therefore, a residual is obtained, which includes a first residual component of the primary component and a second residual component of at least one secondary component. In principle, the image can be a still image or an inter-frame sequence of a video sequence.

[0055] The first entropy model and the second entropy model can be provided by the aforementioned super-prior pipeline.

[0056] The processing of the first prior tensor can be independent of the processing of the second prior tensor.

[0057] The primary component of an image can be the luminance component, and at least one secondary component can be the chrominance component. In this case, the second tensor can represent two residual components of the two secondary components, one of which is the chrominance component and the other is another chrominance component. Alternatively, the primary component of an image can be the chrominance component, and at least one secondary component can be the luminance component.

[0058] Because integer factors are used to obtain the first prior tensor after resampling, this is highly beneficial in terms of both compression performance and simplicity.

[0059] In principle, an image can be a still image or an intra-frame of a video sequence. In this and the following description, it should be understood that an image comprises multiple components, particularly luminance and chrominance components. A component can be considered as the dimension of an orthogonal basis describing a full-color image. For example, when an image is represented in YUV space, the components are luminance Y, chrominance U, and chrominance V. One component of the image is selected as the principal component, and one or more of the remaining components are selected as one or more minor components (non-principal components). The terms "minor component" and "non-principal component" are used interchangeably herein to refer to the component decoded using the auxiliary information provided by the principal component. Encoding and decoding one or more minor components using the auxiliary information provided by the principal component results in a highly accurate reconstructed image after decoding.

[0060] The processing of the first prior tensor can be independent of the processing of the second prior tensor. In fact, even if the data of the minor components is lost, the encoded major components can be restored. Since the first and second bitstreams can be processed in parallel, this method can reliably and quickly reconstruct the compressed original image data.

[0061] In one implementation, the primary component of the image is a luminance component, and the at least one secondary component of the image is a chrominance component. For example, two secondary components of the image are decoded simultaneously, with one component being a chrominance component and the other being a different chrominance component. In another implementation, the primary component of the image is a chrominance component, and the at least one secondary component of the image is a luminance component. This provides a high degree of flexibility in which one component is actually adjusted by another component.

[0062] Overall decoding can include processing in the prior space, especially processing downsampled input data, thus enabling fast processing with a lower processing load. It should be noted that the terms "downsampling" and "upsampling" here refer to reducing and increasing the size of the tensor representation of the data, respectively.

[0063] The size of the first prior tensor in either the height or width dimension can be smaller than the corresponding size of the first tensor in either the height or width dimension, and / or the size of the second prior tensor in either the height or width dimension can be smaller than the corresponding size of the cascaded tensor in either the height or width dimension. For example, a reduction factor of 16 or 32 can be used in the height and / or width dimensions.

[0064] It is possible that the size or subpixel offset of the sample in the second tensor at least in the height and width dimensions of the tensor is different from the size or subpixel offset of the sample in the first tensor at least in the height and width dimensions.

[0065] Similarly, the size of the first prior tensor in either the height or width dimension can be smaller than the corresponding size in the height or width dimension of the first tensor, and / or the size of the second prior tensor in either the height or width dimension can be smaller than the corresponding size in the height or width dimension of the cascaded tensor. For example, a reduction factor of 16 or 32 can be used in the height and / or width dimensions. Adjusting the sample positions of the first tensor to match the sample positions of the second tensor can include, for example, performing downsampling with a factor of 2 in the width and height of the first tensor.

[0066] According to one implementation of the second aspect, the first prior tensor includes a channel dimension, the second prior tensor includes a channel dimension, and the size of the first prior tensor in the channel dimension is greater than, less than, or equal to the size of the second prior tensor in the channel dimension. If the primary component is considered significantly more important than one or more secondary components (which is often the case), the channel length of the primary component can be greater than the channel length of one or more secondary components. If the signal of the primary component is relatively clear, while the signal of one or more non-primary components is relatively noisy, the channel length of the primary component can be less than the channel length of one or more secondary components. Numerical experiments show that shorter channel lengths compared to existing techniques can be used without significantly reducing the quality of the reconstructed image, thus reducing memory requirements.

[0067] Typically, a first tensor can be transformed into a first prior tensor using a first neural network, and a cascaded tensor can be transformed into a second prior tensor using a second neural network different from the first neural network. In this case, the first and second neural networks can be trained collaboratively to determine the size of the first and second prior tensors in the channel dimension. The channel length can be determined through exhaustive search or in a content-adaptive manner. A set of models can be trained, where each model decodes the primary and non-primary components based on a different number of channels. Therefore, the neural network may be able to optimize the channel lengths involved.

[0068] The determined channel length must also be used by the decoder to reconstruct the coded components. Therefore, according to one implementation, the size of the first prior tensor in the channel dimension can be indicated in the first bitstream, and the size of the second prior tensor in the channel dimension can be indicated in the second bitstream. The indication can be explicit or implicit, and the channel length can be directly communicated to the decoder in a bit-saving manner.

[0069] According to one implementation of the second aspect, the first bitstream is parsed according to a first entropy model, and the second bitstream is parsed according to a second entropy model, which is different from the first entropy model. Such entropy models can reliably estimate the statistical properties used in the process of converting the tensor representation of data into a bitstream.

[0070] The disclosed method can be advantageously implemented in the context of a super-prior architecture that provides side information useful for decoding a portion of an image to improve the accuracy of reconstructing that portion.

[0071] Resampling can be performed without interpolation (a computationally expensive operation), which can improve decoding efficiency accordingly.

[0072] Integer factors can be calculated based on a first scaling factor of the primary component and a second scaling factor of at least one secondary component. For example, the integer factor is calculated as follows: ,in, Indicates the first scaling factor. This represents the second scaling factor.

[0073] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0074] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0075] As another example, integer factors It can exist in the bitstream or be set as the default value.

[0076] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0077] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0078] Then, Export as ,in, This represents the scaling factor for at least one minor component.

[0079] According to one implementation of the second aspect, the first scaling factor may include a first horizontal scaling factor and a first vertical scaling factor. Similarly, the second scaling factor may include a second horizontal scaling factor and a second vertical scaling factor.

[0080] The resampling is performed through nearest neighbor upsampling or nearest neighbor downsampling.

[0081] For nearest neighbor upsampling:

[0082] Represented as The size received by this layer is Tensor input, and output size is The tensor output.

[0083] This downsampling does not require interpolation; simply copy it.

[0084]

[0085] Where s represents an integer factor.

[0086] For nearest neighbor downsampling:

[0087] Represented as The layer receives a size of Tensor input, and output size is The tensor output, through this process, reduces the spatial resolution of each tensor channel.

[0088] This downsampling does not require interpolation; simply copy it.

[0089]

[0090] Where s represents an integer factor.

[0091] According to one implementation of the method of the second aspect, processing the first prior tensor includes transforming the first prior tensor into the first tensor.

[0092] According to one implementation of the method of the second aspect, obtaining a second tensor representing at least one minor residual component of the residual of at least one minor component of the image includes:

[0093] The second prior tensor is concatenated with the resampled first prior tensor to obtain a concatenated tensor, and the concatenated tensor is transformed back into the second tensor. At least one of these transformations may include upsampling. Therefore, processing in the prior space can be performed at a lower resolution, as this is necessary for accurate component reconstruction in YUV space or any other space suitable for image representation.

[0094] According to another implementation of the method in the second aspect, each of the first prior tensor and the second prior tensor has a height and a width dimension, and processing the first prior tensor includes transforming the first prior tensor into the first tensor, and processing the second prior tensor includes determining whether the size or subpixel offset of a sample in at least one of the height and width dimensions of the second prior tensor is different from the size or subpixel offset of a sample in at least one of the height and width dimensions of the first prior tensor. When it is determined that the size or subpixel offset of a sample in the second prior tensor is different from the size or subpixel offset of a sample in the first prior tensor, the sample position of the first prior tensor is adjusted to match the sample position of the second prior tensor. This yields the adjusted first prior tensor. Furthermore, when the sample size or sub-pixel offset of the second prior tensor is different from that of the sample size or sub-pixel offset of the first prior tensor, the second prior tensor and the adjusted first prior tensor are concatenated to obtain a concatenated prior tensor; otherwise, the second prior tensor and the first prior tensor are concatenated to obtain a concatenated prior tensor, and the concatenated prior tensor is transformed into the second tensor.

[0095] The first bitstream can be processed by the first neural network, and the second bitstream can be processed by the second neural network, which is different from the first neural network. The first prior tensor can be transformed by the third neural network, which is different from the first and second networks, and the cascaded prior tensor can be transformed by the fourth neural network, which is different from the first, second, and third networks.

[0096] According to another implementation of the method described in the second aspect, the first prior tensor includes a channel dimension, the second prior tensor includes a channel dimension, and the size of the first prior tensor in the channel dimension is greater than, less than, or equal to the size of the second prior tensor in the channel dimension. Information regarding the size of the first and second prior tensors in the channel dimension can be obtained from information indicated in the first and second bitstreams, respectively.

[0097] Any of the exemplary implementations described above can be combined as appropriate. A method according to any of the above aspects and implementations can be implemented in the apparatus.

[0098] According to a third aspect, an apparatus for reconstructing at least a portion of an image is provided, the apparatus comprising one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing a program executable by the one or more processors, wherein, when the program is executed by the one or more processors, the apparatus performs the method according to any one of the first and second aspects and corresponding implementations described above.

[0099] According to a fourth aspect, a processing apparatus for reconstructing at least a portion of an image is provided, the processing apparatus including a processing circuit for performing the method according to any one of the first and second aspects and their respective implementations.

[0100] Furthermore, according to a fifth aspect, a computer program stored on a non-transitory medium is provided, the computer program comprising code that, when executed on one or more processors, performs the steps of the method according to any one of the foregoing aspects and implementations. Attached Figure Description

[0101] The technical background and embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0102] Figure 1 A schematic diagram of the channels processed by the layers of a neural network is shown.

[0103] Figure 2 A schematic diagram of encoder types for neural networks is shown.

[0104] Figure 3 A schematic diagram of a network architecture including a hyperprior model is shown.

[0105] Figure 4 A block diagram illustrating the structure of a cloud-based scheme for machine-based tasks such as machine vision is shown.

[0106] Figure 5 A block diagram illustrating the structure of an end-to-end trainable video compression framework is shown.

[0107] Figure 6 A block diagram of a network for motion vector (MV) compression is shown.

[0108] Figure 7 A block diagram of a learning image compression configuration in this field is shown.

[0109] Figure 8 A block diagram of another learning image compression configuration in this field is shown.

[0110] Figure 9The concept of conditional decoding is illustrated.

[0111] Figure 10 The concept of residual decoding is illustrated.

[0112] Figure 11 The concept of conditional residual decoding is illustrated.

[0113] Figure 12 Conditional intra-frame decoding according to an embodiment of the present invention is illustrated.

[0114] Figure 13 Conditional residual decoding according to an embodiment of the present invention is illustrated.

[0115] Figure 14 The conditional decoding according to an embodiment of the present invention is shown, which means that the prior space sample position is restricted to an integer sampling factor when adjusting.

[0116] Figure 15 Conditional intra-frame decoding of input data in YUV420 format according to an embodiment of the present invention is illustrated.

[0117] Figure 16 Conditional intra-frame decoding of input data in YUV444 format according to an embodiment of the present invention is illustrated.

[0118] Figure 17 Conditional residual decoding of input data in YUV420 format according to an embodiment of the present invention is illustrated.

[0119] Figure 18 Conditional residual decoding of input data in YUV444 format according to an embodiment of the present invention is illustrated.

[0120] Figure 19 Conditional residual decoding of input data in YUV420 format according to another embodiment of the present invention is shown.

[0121] Figure 20 Conditional residual decoding of input data in YUV444 format according to another embodiment of the present invention is shown.

[0122] Figure 21 A flowchart illustrating an exemplary method for reconstructing at least a portion of an image according to an embodiment of the present invention is shown.

[0123] Figure 22 This is a decoder architecture according to an embodiment of the present invention.

[0124] Figure 23 This is a decoder architecture according to another embodiment of the present invention.

[0125] Figure 24 This is a signal decoder architecture according to an embodiment of the present invention.

[0126] Figure 25 A processing apparatus for performing a method of encoding or reconstructing at least a portion of an image is shown according to an embodiment of the present invention.

[0127] Figure 26 A block diagram is shown as an example of a video decoding system for implementing embodiments of the present invention.

[0128] Figure 27 A block diagram is shown of another example of a video decoding system for implementing embodiments of the present invention.

[0129] Figure 28 A block diagram showing an example of an encoding or decoding device is provided.

[0130] Figure 29 A block diagram of another example of an encoding or decoding device is shown. Detailed Implementation

[0131] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure, illustrating by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0132] For example, it should be understood that the disclosure relating to the described method is also applicable to the corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of multiple steps respectively), even if the one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units, such as functional units, the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of multiple units respectively), even if the one or more units are not explicitly described or shown in the drawings. Furthermore, it should be understood that, unless otherwise expressly stated, features of the various exemplary embodiments and / or aspects described herein can be combined with each other.

[0133] The following is an overview of some of the technical terms used.

[0134] Artificial Neural Networks

[0135] Artificial neural network (ANN) or connectionist systems are computational systems inspired by the fuzzy logic of the biological neural networks that make up the animal brain. These systems "learn" to perform tasks by example, typically without being programmed using task-specific rules. For instance, in image recognition, these systems might learn to recognize images containing cats by analyzing exemplary images manually labeled "cat" or "no cat" and using the results to identify cats in other images. These systems do this without prior knowledge about cats, such as that they have fur, a tail, whiskers, and a cat's face. Instead, they automatically generate recognizable features from the examples they process.

[0136] ANNs are based on a collection of connection units or nodes called artificial neurons, which loosely mimic neurons in a biological brain. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. The artificial neuron that receives the signal processes it and can then send signals to neurons connected to it.

[0137] In ANN implementations, the "signal" at a connection is a real number, and the output of each neuron is calculated based on a nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. These weights increase or decrease the strength of the signal at the connection. Neurons can have a threshold such that a signal is only sent when the aggregated signal exceeds that threshold. Typically, neurons are aggregated into layers. Different layers can perform different transformations on their inputs. The signal travels from the first layer (input layer) to the last layer (output layer), but this may occur after multiple traversals of these layers.

[0138] The initial goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention shifted to performing specific tasks, leading to a departure from biology. ANNs have been applied to a wide variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board games and video games, medical diagnosis, and even activities traditionally considered solely human, such as painting.

[0139] Convolutional Neural Networks

[0140] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. Simply put, a convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0141] Figure 1 This schematically illustrates the general concept of processing performed by neural networks such as CNNs. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is where input is provided (e.g., ...). Figure 1 (A portion of the image shown) is processed by layers. Hidden layers in a CNN typically consist of a series of convolutional layers that perform convolutions through multiplication or other dot products. The result of a layer is one or more feature maps ( Figure 1 In the context of f.maps, feature maps are sometimes also called channels. Subsampling may be involved in some or all layers. Therefore, feature maps may become smaller, such as... Figure 1 As shown. The activation function in a CNN is typically a Rectified Linear Unit (RELU) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers. These convolutions are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. While these layers are colloquially referred to as convolutions, this is merely a convention. Mathematically, convolution is technically a sliding dot product or cross-correlation. This is significant for indexing in a matrix, as the matrix influences how weights are determined at specific index points.

[0142] When configuring a CNN to process images, such as Figure 1 As shown, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth). Then, after passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network should have the following properties: A convolutional kernel defined by its width and height (hyperparameters). The number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0143] In the past, traditional multilayer perceptron (MLP) models have been applied to image recognition. However, due to the full connectivity between nodes, traditional MLP models are affected by high dimensionality and cannot scale well when processing high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high for efficient and feasible large-scale processing with full connectivity. This network architecture also does not consider the spatial structure of the data, causing input pixels that are far apart to be processed in the same way as pixels that are close together. This ignores the principle of locality in image data both computationally and semantically. Therefore, the full connectivity of neurons is useless for purposes such as image recognition, which is dominated by spatially local input patterns.

[0144] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons, specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. Convolutional layers are the core building blocks of CNNs. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above), each with a small receptive field that extends across the entire depth of the input image. During forward propagation, each filter is convolved across the width and height of the input image, computing the dot product between the elements of that filter and the input, and generating a two-dimensional activation map of that filter. Thus, when the network detects a particular type of feature at a spatial location in the input, it learns to activate the corresponding filter.

[0145] Stacking activation maps of all filters along the depth dimension forms the complete output image of the convolutional layer. Therefore, each element in the output image can also be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature maps and activations have the same meaning. In some papers, it's called an activation map because it's a mapping corresponding to the activations of different parts of the image, and also a feature map because it's also a mapping indicating the location of a certain feature in the image. High activation indicates that a certain feature has been found.

[0146] Another important concept in CNNs is pooling, which is a form of non-linear downsampling. Several non-linear functions can be used to implement pooling, the most common being max pooling. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each sub-region.

[0147] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the reason behind the use of pooling in convolutional neural networks. Pooling layers are used to gradually reduce the size of the representation space, thereby reducing the number of parameters, memory footprint, and computational cost in the network, and thus also controlling overfitting. In CNN architectures, it is common practice to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0148] Pooling layers operate independently on each depth strip in the input, spatially adjusting the size of the depth strips. The most common form is a 2×2 pooling layer with a stride of 2, which applies a downsampling factor of 2 to both the width and height of each depth strip in the input, discarding 75% of the activations. In this case, each max operation is performed on four numbers. The depth dimension remains unchanged.

[0149] Besides max pooling, other functions can also be used for pooling units, such as average pooling or... Normative pooling. Average pooling was frequently used in the past, but has recently fallen short of max pooling because max pooling performs better in practice. Due to the significant reduction in representation size, there has recently been a trend towards using smaller filters or discarding pooling layers entirely. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a single parameter. Pooling is an important component of convolutional neural networks used for object detection based on the Fast R-CNN architecture.

[0150] The ReLU mentioned above is an abbreviation for Corrected Linear Unit, which applies a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting negative values ​​to zero. ReLU increases the non-linearity of the decision function and the overall network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the saturating hyperbolic tangent and the sigmoid function. ReLU is generally superior to other functions because it trains neural networks several times faster without significantly reducing generalization accuracy.

[0151] After several convolutional and max-pooling layers, high-level inference in the neural network is accomplished through fully connected layers. As in conventional (non-convolutional) artificial neural networks, neurons in fully connected layers are connected to all activations in the previous layer. Therefore, the activations of these neurons can be computed as affine transformations, which involve matrix multiplication followed by bias shifts (vector addition of learned or fixed bias terms).

[0152] The "loss layer" specifies how training compensates for the bias between the predicted (output) labels and the true labels, and is typically the last layer of a neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0153] In short, Figure 1This illustrates the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map comprising several channels, corresponding to the number of filters in a set of learnable filters for that layer. Then, the feature map is subsampled using pooling layers, which reduces the dimensionality of each channel in the feature map. Next, the data enters another convolutional layer, which may have a different number of output channels, and therefore a different number of channels in the feature map. As mentioned above, the number of input and output channels are hyperparameters of that layer. To establish network connectivity, these parameters need to be synchronized between two connected layers; for example, the number of input channels in the current layer should be equal to the number of output channels in the previous layer. For the first layer processing input data (e.g., an image), the number of input channels is typically equal to the number of channels used for data representation; for example, 3 channels for RGB or YUV representation of an image or video, or 1 channel for grayscale image or video representation.

[0154] Autoencoders and Unsupervised Learning

[0155] An autoencoder is an artificial neural network used to learn efficient data decoding in an unsupervised manner. A schematic diagram of an autoencoder is shown below. Figure 2 As shown. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore signal "noise," typically for dimensionality reduction. Along with the dimensionality reduction side, a reconstruction side is also learned, where the autoencoder attempts to generate a representation from the dimensionality-reduced encoding that is as close as possible to the original input of the autoencoder, hence the name. In the simplest case, given a hidden layer, the encoder stage of the autoencoder... For input and map that input to

[0156] .

[0157] The image This is commonly referred to as encoding, prior variable, or prior representation. Here, It is an element-wise activation function, such as the sigmoid function or the modified linear unit. It is a weight matrix. This is the bias vector. Weights and biases are typically initialized randomly and then iteratively updated via backpropagation during training. Afterwards, the decoder stage of the autoencoder maps h to... Reconstruction with the same shape :

[0158]

[0159] Among them, the decoder , and Possibly corresponding to the encoder , and Irrelevant.

[0160] Variational autoencoder models make strong assumptions about the distribution of prior variables. They use variational methods for prior representation learning, resulting in additional loss components and a specific estimator used to train the algorithm, called a Stochastic Gradient Variational Bayes (SGVB) estimator. Variational autoencoder models assume that the data is generated by a directed graphical model. The generated, and the encoder learns the posterior distribution approximation ,in, and These represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the prior vectors of a VAE is typically closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0161]

[0162] here, This represents the Kullback–Leibler divergence. The prior variable is typically set to a centrally isotropic multivariate Gaussian distribution. The variational and likelihood distributions are typically chosen to be factored Gaussian distributions.

[0163]

[0164]

[0165] in, It is the encoder output, and and This is the decoder output.

[0166] Recent advances in the field of artificial neural networks (especially convolutional neural networks) have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using variational autoencoder-based networks has been proposed. Data compression, therefore, is considered a well-studied fundamental problem in engineering, typically formulated with the goal of designing an encoding for a given set of discrete data that minimizes entropy. This approach relies heavily on an understanding of the probabilistic structure of the data, thus the problem is closely related to probabilistic source modeling. However, since all practical encodings must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, introducing error. In this context, known as the lossy compression problem, one must weigh two competing costs: the entropy of the discretized representation (rate) and the error (distortion) introduced by quantization. Different compression applications, such as data storage or data transmission over channels with limited capacity, require different trade-offs between rate and distortion. Joint optimization of rate and distortion is extremely difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is difficult to solve. For this reason, most existing image compression methods linearly transform the data vector into a suitable continuous-value representation, independently quantize the elements of the continuous-value representation, and then encode the resulting discrete representation using lossless entropy coding. Since the transform plays a central role, this approach is called transform decoding. For example, JPEG uses the discrete cosine transform for pixel blocks, while JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform decoding method—the transform, the quantizer, and the entropy decoding—are optimized separately (usually through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transform representations to decode the predicted residual signal. Several transforms are used for this purpose, such as the discrete cosine transform (DCT) and the discrete sine transform (DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0167] Variational image compression

[0168] In their paper "Density Modeling of Images Using a Generalized Normalization Transformation" (hereinafter referred to as "Balle"), published on the arXiv e-print version at the 4th International Conference on Learning Representations in 2016 by J. Balle, L. Valero Laparra, and EP Simoncelli (2015), the authors proposed an end-to-end optimization framework for an image compression model based on a nonlinear transformation. Previously, they had demonstrated that a model optimized for a measure of perceptual distortion, including linear-nonlinear block transformations, outperformed a model optimized for mean squared error (MSE). Here, the authors optimized MSE but used a more flexible transformation constructed from a concatenation of linear convolutions and nonlinearities. Specifically, they used a joint nonlinearity with generalized divisive normalization (GDN), inspired by neuron models in biological visual systems and proven effective for Gaussianizing image density. This cascaded transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which is essentially a parametric vector quantization performed in the original image space. The compressed image is then reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0169] For any desired point on the rate distortion curve, the parameters of the analytical and synthetic transforms are jointly optimized using stochastic gradient descent. To achieve this in the presence of quantization (which produces zero gradients almost everywhere), the authors use a continuously relaxed surrogate loss function based on a probabilistic model, replacing the quantization step with additive uniform noise. The relaxed rate distortion optimization problem is somewhat similar to the problem used to fit generative image models (especially variational autoencoders), but differs in the constraints imposed by the authors to ensure that the relaxed rate distortion optimization problem approximates a discrete problem along the rate distortion curve. Finally, instead of reporting differential entropy or discrete entropy estimates, the authors implement entropy decoding and report performance using actual bit rates, thus demonstrating the feasibility of the scheme as a complete lossy compression method.

[0170] In J. Balle, an end-to-end trainable model for image compression based on variational autoencoders is described. This model incorporates a super-prior to effectively capture spatial dependencies in the prior representation. This super-prior involves side information sent to the decoder; this concept applies to almost all modern image codecs but has been largely unexplored in image compression using ANNs. Unlike existing autoencoder compression methods, this model jointly trains the complex prior with the underlying autoencoder. The authors demonstrate that this model achieves state-of-the-art image compression when visual quality is measured using the popular MS-SSIM metric, and outperforms published ANN-based methods in terms of rate distortion performance when evaluated using a more traditional metric based on squared error (PSNR).

[0171] Figure 3 The network architecture including the advanced prior model is shown. Left side (g) a g s The image autoencoder architecture is shown on the right (h). a h s This corresponds to the implementation of a super-prior autoencoder. The factorization prior model uses the exact same architecture for analysis and synthesis transformation g. a and g s Q represents quantization, and AE and AD represent arithmetic encoder and arithmetic decoder, respectively. The encoder applies the input image x to g. a This yields the response y (prior representation) with spatially varying standard deviation. The encoding g... a It includes convolutional layers with multiple subsampling and generalized divisive normalization (GDN) as the activation function.

[0172] Input the response into h a The standard deviation distribution of z is summarized. Then, z is quantized, compressed, and sent as side information. The encoder then uses the quantized vector... To estimate That is, the spatial distribution of the standard deviation of the probability values ​​(or frequency values) of arithmetic decoding (AE), and using... To compress and send quantized image representations (or a priori representation). The decoder first restores the signal from the compressed signal. Then, the decoder uses h s Get This provides the decoder with the correct probability estimate, thus enabling successful reconstruction. Then, the decoder will Enter g s In order to obtain the reconstructed image.

[0173] Further research improved the probabilistic modeling via super-prior by introducing autoregressive models (e.g., based on the PixelCNN++ architecture). This allows for better probability estimation of other undecoded symbols by utilizing the context of decoded symbols in the prior space, as seen in L. Zhou, Zh. Sun, X. Wu, and J. Wu's paper "End-to-end Optimized Image Compression with Attention Mechanism" (hereinafter referred to as "Zhou"), published at CVPR 2019. Figure 2 As shown.

[0174] Cloud solutions for machine tasks

[0175] Video Coding for Machines (VCM) is another popular area of ​​computer science. The main idea behind this approach is to send a decoded representation of image or video information for further processing via computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video decoding, which targets human perception, VCM focuses on performance characteristics specific to computer vision tasks, such as object detection accuracy, rather than the quality of the reconstruction. This is because... Figure 4 As shown.

[0176] Video Coding for Machines, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks on mobile cloud infrastructure. By partitioning the network between mobile devices and the cloud, computational workload can be distributed, thereby minimizing the overall energy consumption and / or latency of the system. Generally, collaborative intelligence is a pattern of distributing the processing work of a neural network to two or more different computing nodes (e.g., devices, but typically any functionally defined node). Here, the term "node" does not refer to the neural network node mentioned above. Instead, a (computing) node refers to a (physically or at least logically) independent device / module that implements a portion of the functionality of the neural network. These devices can be different servers, different end-user devices, a hybrid of servers and / or user devices and / or the cloud and / or processors, etc. In other words, computing nodes can belong to the same neural network, and these nodes communicate with each other to transmit decoded data within / for the neural network. For example, to perform complex computations, one or more layers can be executed on one device, and one or more layers can be executed on another device. However, the distribution can also be more granular, with a single layer executing on multiple devices. In this disclosure, the term "multiple" refers to two or more. In some existing schemes, a portion of the neural network function is executed on one or more such devices (such as user devices or edge devices), and the output (feature maps) is then passed to the cloud. The cloud is a collection of processing or computing systems located outside the devices that are operating a portion of the neural network. The concept of collaborative intelligence has also been extended to model training. In this case, data flows bidirectionally: the backpropagation process during training flows from the cloud to the mobile device, and the forward propagation and inference processes during training flow from the mobile device to the cloud.

[0177] Some studies have proposed semantic image compression by encoding deep features and then reconstructing the input image from these features. These include compression based on uniform quantization and context-based adaptive arithmetic coding (CABAC) in H.264. In certain scenarios, sending the output of the hidden layer (deep feature maps) from the moving part to the cloud and using the reconstructed image to perform object detection may be more efficient than sending compressed natural image data to the cloud. Efficient compression of feature maps is beneficial for image and video compression and reconstruction, both for human perception and machine vision. Entropy decoding methods (e.g., arithmetic decoding) are commonly used to compress deep features (i.e., feature maps).

[0178] Today, video content accounts for over 80% of internet traffic, and this proportion is expected to increase further. Therefore, building efficient video compression systems and generating higher-quality frames within a given bandwidth budget is crucial. Furthermore, most video-related computer vision tasks (such as video object detection or video object tracking) are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Simultaneously, techniques in video compression also contribute to action recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules such as block-based motion estimation and Discrete Cosine Transform (DCT) to reduce redundancy in video sequences, as mentioned above. While each module is carefully designed, the entire compression system lacks end-to-end optimization. The aim is to further improve video compression performance by jointly optimizing the entire compression system.

[0179] End-to-end image or video compression

[0180] Recently, image compression autoencoders based on deep neural networks (DNNs) have achieved performance comparable to or even better than traditional image codecs such as JPEG, JPEG2000, or BPG. One possible reason is that DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations, which are not available in traditional methods. However, directly applying these techniques to build an end-to-end learning system for video compression is not easy. First, learning how to generate and compress motion information specifically for video compression remains an open question. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences. A simple approach is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate the most accurate flow field possible. For specific video tasks, accurate optical flow is often not optimal. Furthermore, the amount of data in optical flow is significantly increased compared to motion information in traditional compression systems, and directly applying existing compression methods to compress optical flow values ​​would drastically increase the number of bits required to store motion information. Second, it remains unclear how to build a DNN-based video compression system by minimizing the rate distortion of residuals and motion information. Rate-distortion optimization (RDO) aims to achieve higher-quality reconstructed frames (i.e., less distortion) given a certain number of bits (or bitrate) for compression. RDO is crucial for video compression performance. To leverage the advantages of end-to-end training in learning-based compression systems, RDO strategies are needed to optimize the entire system.

[0181] In their paper "DVC: An End-to-end Deep Video Compression Framework" published on pages 11006-11015 of the proceedings of the 2019 IEEE / CVF Computer Vision and Pattern Recognition (CVPR) conference, Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao proposed an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual decoding.

[0182] This type of encoder is like Figure 5 shown. Specifically, Figure 5 The overall structure of an end-to-end trainable video compression framework is shown. To compress motion information, a CNN is specified to transform optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder-like network is used to compress the optical flow. The motion vector (MV) compression network is as follows: Figure 6 As shown. Network architecture and Figure 3 g in a / g s There are some similarities. Specifically, the optical flow is input into a series of convolutional operations and nonlinear transformations, including GDN and IGDN. The number of output channels for each convolution (deconvolution) is 128, but the number of output channels for the last deconvolutional layer is equal to 2. Assuming the size of the optical flow is M×N×2, the MV encoder generates a motion representation of size M / 16×N / 16×128. The motion representation is then quantized and entropy-decoded and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0183] Specifically, the following definitions apply.

[0184] Image size (the English word for image can be either "picture" or "image," and the two are used interchangeably): refers to the width, height, or width-height pair of an image. The width and height of an image are usually measured in units of brightness samples.

[0185] Downsampling: Downsampling is a process of reducing the sampling rate (sampling interval) of a discrete input signal.

[0186] Upsampling: Upsampling is a process of increasing the sampling rate (sampling interval) of a discrete input signal.

[0187] Cropping: Trimming the outer edges of a digital image. Cropping can be used to reduce the size of an image (in terms of sample size) and / or change the image's aspect ratio.

[0188] Padding: Padding refers to increasing the size of an input image (or image) by generating new samples at the boundaries of the image using predefined sample values ​​or sample values ​​at locations in the input image.

[0189] Convolution: Convolution is given by the following general equation. In the following text, f() can be defined as the input signal, and g() can be defined as the filter.

[0190]

[0191] NN module: A neural network module, that is, a component of a neural network. An NN module can be a layer or a subnetwork within a neural network. A neural network is a series of NN modules. In the context of this document, it is assumed that a neural network is a series of K NN modules.

[0192] Latent space: The intermediate steps in neural network processing. The prior space represents the output of the input layer or one or more hidden layers that should not be observed.

[0193] Lossy NN module: A lossy NN module will cause information loss when processing information, and the information processed by the lossy module cannot be recovered.

[0194] Lossless NN module: The lossless NN module processes information without causing information loss, and the information processed by the lossless module can be recovered.

[0195] Bottleneck: The prior space tensor leading to the lossless decoding module

[0196] An autoencoder is a model that transforms a signal into a (compressed) prior space and then back into the original signal space.

[0197] The encoder uses convolutional layers, nonlinearities, and / or residuals to downsample the image to obtain the prior tensor (y).

[0198] Decoder: Upsamples the prior tensor (y) using convolutional layers, nonlinearities, and / or residuals to bring it up to the size of the original image.

[0199] Hyper-encoder: Also uses convolutional layers, nonlinearities, and / or residuals to downsample the prior tensor to obtain a smaller prior tensor (z).

[0200] Hyper-Decoder: Upsamples a small prior tensor (z) using convolutional layers, nonlinearities, and / or residuals to perform entropy estimation.

[0201] Arithmetic Encoder / Decoder (AE / AD): Encodes a priori tensors into or decodes priori tensors from a bitstream using a given statistical prior.

[0202] Autoregressive Entropy Estimation: The process of sequentially estimating the statistical prior of prior tensors.

[0203] Q: Quantization block.

[0204] , : The quantized version of the corresponding prior tensor.

[0205] Masked Convolution (MaskedConv): A type of convolution that masks certain prior tensor elements so that the model can only make predictions based on the prior tensor elements that have already been seen.

[0206] H, W: Height and width of the input image

[0207] Block / Patch: A subset of prior tensors on a rectangular grid

[0208] Information sharing: the collaborative processing of information from different sources.

[0209] P: Size of the rectangular piece

[0210] K: Core size, defines the number of neighboring slices included in information sharing.

[0211] L: Kernel size, defines how many previously decoded prior tensor elements are included in information sharing.

[0212] Masked Convolution (MaskedConv): A type of convolution that masks certain prior tensor elements so that the model can only make predictions based on the prior tensor elements that have already been seen.

[0213] PixelCNN: A convolutional neural network that contains one or more layers of masked convolutions.

[0214] Component: A dimension that describes the orthogonal basis of a full-color image.

[0215] Channel: A layer in a neural network.

[0216] Intra codec: Treats the first frame or keyframe of a video as an intra frame, typically as an image.

[0217] Inter-frame codec: After the intra-frame codec, the video compression system performs inter-frame prediction. First, the motion estimation tool calculates the motion vectors of the objects, and then the motion compensation tool uses the motion vectors to predict the next frame.

[0218] Residual codec: The predicted frame is not always exactly the same as the current frame; the difference between the current frame and the predicted frame is the residual. The residual codec compresses the residual like compressing an image.

[0219] Signal conditioning: A training process in which an additional signal is used to help the neural network perform inference, but the additional signal is not present in the output and is very different from the output.

[0220] Conditional codec: A codec that uses signal conditioning to aid (guide) compression and reconstruction. Because the auxiliary information required for conditioning is not part of the input signal, in state-of-the-art (SOTA) systems, conditional codecs are used to compress video streams, not images.

[0221] Figure 7 A block diagram of a specific learning image compression configuration that can be improved according to this disclosure is shown, the configuration including an autoencoder and a super-prior component as known in the art. The input image to be compressed is represented as having a size of or 3D tensors, where, and These are the height and width (dimensions) of the image, respectively. This refers to the number of components (e.g., one luminance component and two chrominance components). The input image is processed by encoder 71. The encoder downsamples the input image by applying multiple convolutions and nonlinear transformations, and generates a priori tensors. It's important to note that in the context of deep learning, the terms "downsampling" and "upsampling" do not refer to resampling in the classical sense, but rather to general terms used to change the size of the H and W dimensions of a tensor. "Resampling" can be translated as "re-sample" or "resample". Similarly, "performing resampling" and "after resampling" can be translated as "re-sampling / resampling" and "re-sampled / resampled," respectively.

[0222] Prior tensor output by encoder 71 Represents an image in the prior space, with a size of ,in, It is the downsampling factor of encoder 71. It is the number of channels (e.g., the number of neural network layers used when transforming a tensor representing an input image).

[0223] Prior tensors The super encoder 72 further downsamples the data into a super-prior tensor through convolution and nonlinear transformation. Super-prior tensor The size is Through block Q on the hyperprior tensor Quantization is performed to obtain the quantized super-prior tensor. The quantized hyperprior tensor is estimated using a factorization entropy model. The statistical properties of the values. Arithmetic encoders (AEs) use these statistical properties to create tensors. The bitstream representation. Tensor All elements are written into the bitstream without the need for an autoregressive process.

[0224] The factorization entropy model is used as the codebook, and its parameters are available at the decoder. The arithmetic decoder (AD) recovers the hyperprior tensor from the bitstream using the factorization entropy model. By applying multiple convolution operations and nonlinear transformations, the super decoder recovers 73 pairs of hyperprior tensors. Perform upsampling. The restored hyperprior tensor after upsampling is represented as: Based on the restored hyperprior tensor from upsampling Autoregressive estimation of quantized prior tensors The entropy of the quantized prior tensor is obtained from this. The resulting autoregressive entropy model is used to estimate the quantized prior tensor. Statistical attributes.

[0225] The arithmetic encoder (AE) uses these estimated statistical properties to create quantized prior tensors. The bitstream representation. In other words, the arithmetic encoder (AE) of the autoencoder component compresses the image information in the prior space using entropy coding based on the side information provided by the super-prior component. Prior tensor The arithmetic decoder (AD) at the receiving side reconstructs the prior tensors from the bitstream using an autoregressive entropy model. This is achieved by applying multiple convolution operations and nonlinear transformations to the decoder, which then reconstructs the prior tensors from 74 pairs of data. Upsampling is performed to obtain a tensor representation of the reconstructed image.

[0226] Figure 8 It shows the Figure 7 The architecture shown is modified. The processing of the encoder 81 and decoder 84 of the autoencoder component is similar to... Figure 7 The processing of encoder 71 and decoder 74 in the autoencoder component shown is similar to that of encoder 82 and decoder 83 in the hyper-prior component. Figure 7 The processing of the encoder 72 and decoder 73 of the advanced prior components shown is illustrated. It should be noted that each of these encoders 71, 81, 72, 82 and decoders 73, 83, 74, 84 may individually comprise a neural network or be connected to a neural network. Furthermore, the neural network can be used to provide the entropy model involved.

[0227] and Figure 7 The configurations shown are different, in Figure 8 In the configuration shown, the prior tensor for quantization Perform masked convolution to obtain the result with Compared to the tensor Φ with a reduced number of elements. Based on the cascaded tensor Φ and... (The upsampled, restored hyperprior tensor) yields an entropy model. This entropy model is then used to estimate the quantized prior tensor. Statistical attributes.

[0228] Conditional decoding represents a special type of decoding in which auxiliary information is used to improve the quality of the reconstructed image. Figure 9 The principle of conditional decoding is illustrated. Auxiliary information A is concatenated with the input frame x and jointly processed by encoder 91. The quantized encoded information in the prior space is written into the bitstream by the arithmetic encoder and then reconstructed from the bitstream by the arithmetic decoder (AD). The reconstructed encoded information in the prior space must be decoded by decoder 92 to obtain the reconstructed frame X. During this decoding stage, the prior representation a of auxiliary information A needs to be added to the input of decoder 92. The prior representation a of auxiliary information A is provided by another encoder 93 and concatenated with the output of decoder 92.

[0229] In the context of video compression, conditional codecs are used to compress the residuals of inter-frame predictions for the current block of the current frame, such as... Figure 10 As shown. The residual is calculated by subtracting the current block from the predicted version of the current block. The residual is encoded by encoder 101 to obtain the residual bitstream. The residual bitstream is decoded by decoder 102. The prediction block is obtained by prediction unit 103 using information from the previous frame / block. Since the prediction block has the same size and dimensions as the current block, it is processed in a similar manner. The reconstruction residual is added to the prediction block to provide the reconstruction block.

[0230] Figure 11 Conditional residual decoding (CodeNet) is illustrated in this paper. The configuration is similar to... Figure 9 The configuration shown is used by the condition encoder 111 to utilize data from the prediction frame. The information serves as auxiliary information for adjusting the codec to the current frame x. t Encoding is performed. The quantized encoded information in the prior space is written into the bitstream by the arithmetic encoder and then restored from the bitstream by the arithmetic decoder (AD). The restored encoded information in the prior space is decoded by decoder 112 to obtain the reconstructed frame X. t During this decoding stage, auxiliary information needs to be... Prior representation Add to the input of decoder 112. Auxiliary information. Prior representation It is provided by another encoder 113 and cascaded with the output of decoder 112.

[0231] CodeNet uses predicted frames, but not the explicit difference (residual) between the predicted frame and the current frame. Compared to residual decoding, decoding the current frame while retrieving all information from the predicted frame can advantageously reduce the amount of information to be transmitted.

[0232] However, due to the involvement of entropy prediction, CodeNet cannot be processed in a highly parallel manner and requires a large amount of memory. According to this disclosure, memory requirements can be reduced, and the overall processing time can be shortened.

[0233] This disclosure provides conditional decoding in which the principal component of an image is encoded independently of one or more non-principal components, and one or more non-principal components are encoded using information from the principal component. Herein and hereinafter, the principal component may be a luma component, and the one or more non-principal components may be chroma components, or the principal component may be a chroma component and a single non-principal component may be a luma component. The principal component can be encoded and decoded independently of one or more non-principal components. Therefore, decoding is possible even if one or more non-principal components are lost for some reason. The one or more non-principal components may be jointly and simultaneously encoded, and may be encoded concurrently with the principal component. Decoding of the one or more non-principal components utilizes information from the prior representation of the principal component. This conditional decoding can be applied to intra-frame prediction and inter-frame prediction processing of video sequences. Furthermore, it can also be applied to still image decoding.

[0234] Figure 12 The basic principle of conditional intra-frame prediction according to an exemplary embodiment is illustrated. The tensor representation x of the input image / frame i is quantized and provided to the encoding device 121. It should be noted that, in this description and below, the entire image can be decoded, or only a portion of the image (e.g., one or more blocks, stripes, coded blocks, etc.) can be decoded.

[0235] In the early stages of encoding device 121, the tensor representation x is divided into a primary intra-component and at least one secondary intra-component, and the primary intra-component is converted into a primary intra-component bitstream, while the at least one secondary intra-component is converted into at least one secondary intra-component bitstream. The bitstream represents the compressed component information used by decoding device 122 to reconstruct the components. The two bitstreams can be interleaved. Encoding device 121 can be addressed as a conditional color separation (CCS) encoding device. The encoding of at least one secondary intra-component is based on information from the primary intra-component, as described in detail later. The corresponding bitstream is decoded by decoding device 122 to reconstruct the image / frame. The decoding of at least one secondary intra-component is based on information from a priori representations of the primary intra-component, as described in detail later.

[0236] Figure 13The basic principle of residual decoding according to an exemplary embodiment is illustrated. The tensor representation x' of the input image / frame i' is quantized, residuals are calculated, and the residuals are provided to encoding device 131. In the early stages of encoding device 131, the residuals are divided into a primary residual component and at least one non-primary residual component, and the primary residual component is converted into a primary residual component bitstream, and the at least one non-primary residual component is converted into at least one non-primary residual component bitstream. Encoding device 131 can be addressed as a conditional color separation (CCS) encoding device. The encoding of at least one non-primary residual component is based on information from the primary residual component, as described in detail later. The corresponding bitstream is decoded by decoding device 132 to reconstruct the image / frame. The decoding of at least one non-primary residual component is based on information from a priori representations of the primary residual component, as described in detail later. The predictions required for calculating the residuals and reconstructing the image / frame are provided by prediction unit 133.

[0237] exist Figure 12 and Figure 13 In the illustrated configuration, encoding devices 121 and 131, and decoding devices 131 and 132, may include corresponding neural networks or be connected to corresponding neural networks. Encoding devices 121 and 131 may include variational autoencoders. Processing the principal component may involve a different number of channels / neural network layers compared to processing at least one non-principal component. Encoding devices 121 and 131 can determine the appropriate number of channels / neural network layers by performing an exhaustive search or in a content-adaptive manner. A set of models can be trained, where each model encodes the principal and non-principal components based on a different number of channels. During processing, the optimal filter can be determined by encoding devices 121 and 131. The neural networks of encoding devices 121 and 131 can be trained collaboratively to determine the number of channels used to process the principal component and one or more non-principal components. In some applications, the number of channels used to process the principal component may be greater than the number of channels used to process one or more non-principal components. In other applications, for example, if the signal of the principal component is less noisy than the signals of one or more non-principal components, the number of channels used to process the principal component may be less than the number of channels used to process one or more non-principal components. In principle, the choice of the number of channels may stem from optimizations in processing speed or signal distortion. Additional channels may reduce distortion but lead to a higher processing load. Experiments show that suitable channel numbers could be, for example, 128 for the primary components and 64 for the non-primary components, or 128 for both primary and non-primary components, or 192 for the primary components and 64 for the non-primary components.

[0238] The number of channels / neural network layers used in the encoding process can be implicitly or explicitly indicated to the decoding devices 122 and 132, respectively.

[0239] Figure 14 An embodiment of conditional decoding of images (frames of a video sequence or still images) is shown in more detail. Encoder 141 receives the principal component P of the image, the size of which is... The tensor representation, in which, Represents the height dimension of the image. This represents the width dimension of the image. This indicates the input channel dimension. In the following text, a tensor of size A×B×C is usually simply referred to as tensor A×B×C. Similarly, a tensor of size C×A×B is usually simply referred to as tensor C×A×B.

[0240] The exemplary sizes of the height, width, and channel dimensions of the tensor output by encoder 141 are: .

[0241] It should be noted that encoders 141 and 142 may be included in encoding devices 121 and 131.

[0242] Based on the output of encoder 141, i.e., the tensor representation of the principal components of the image in the prior space, a bitstream is generated and converted back to the prior space to obtain the reconstructed tensor in the prior space. .

[0243] Tensor representation of the principal component P input to another encoder 142 (Therefore, with tensor) After cascading, the tensor representation of at least one non-primary component NP of the image is obtained. The input is fed into another encoder 142, where, Represents the height dimension of the image, This represents the width dimension of the image. This represents the input channel dimension. An example size for the height, width, and channel dimensions of the tensor output by encoder 142 is... or .

[0244] Before cascading, if the sample sizes or sub-pixel offsets of the tensors differ from each other, it may be necessary to represent the tensor of the principal component P. The sample positions are adjusted to the tensor representation of at least one non-principal component NP. The sample location. Based on the output of another encoder 142, i.e., the representation of the concatenated tensor of the image in the prior space, a bitstream is generated and converted back to the prior space to obtain the reconstructed concatenated tensor in the prior space. .

[0245] Under constraints In this case, the sub-pixel offset in the resampling operation is equal to zero, allowing the resampling process to be performed without interpolation (which is a computationally expensive operation). This can correspondingly improve decoding efficiency.

[0246] On the principal side, the reconstructed tensor in the prior space The input is fed into decoder 143 to be used based on the reconstructed tensor representation. The principal component P of the reconstructed image.

[0247] Furthermore, in the prior space, the execution tensor With tensor The cascading of these tensors is also problematic. Similarly, if the sample sizes or sub-pixel offsets of the tensors to be cascaded differ from each other, some adjustments to the sample positions are necessary. On the non-primary side, this cascading produces tensors... The input is fed into another decoder 144 for processing based on the reconstructed tensor representation. Reconstruct at least one non-primary component NP of the image.

[0248] The decoding described above can be performed on the principal component P independently of at least one non-principal component NP. For example, the decoding of the principal component P and at least one non-principal component NP can be performed simultaneously. Compared with existing techniques, the parallelism of the overall processing can be increased. Furthermore, numerical experiments show that a shorter channel length can be used compared to existing techniques without significantly reducing the quality of the reconstructed image, thus reducing memory requirements.

[0249] In the following text, see references Figures 15 to 20 An exemplary implementation of conditional decoding for the components (one luminance component Y and two chrominance components U and V) of an image represented in YUV space is provided. Needless to say, the disclosed conditional decoding is also applicable to any other (color) space that might be used to represent an image.

[0250] exist Figure 15 In the illustrated embodiment, input data in YUV420 format is processed, where Y represents the luminance component of the current image to be processed, and UV represents the chrominance components U and V of the current image to be processed. 420 indicates that the luminance component Y is four times the size of the chrominance component UV in both height and width dimensions (twice the height and twice the width). Figure 15 In the illustrated embodiment, Y is selected as the primary component processed independently of UV, and UV is selected as a non-primary component. The UV components are processed together.

[0251] The YUV representation of the image to be processed is divided into (primary) Y components and (non-primary) UV components. The encoder 151, including a neural network, receives a tensor representing the Y component of the image to be processed, the size of which is... ,in, These are the height and width dimensions; the input depth (i.e., the number of channels) is... (For a single luminance component). The output of encoder 151 is of magnitude... The prior tensor, where, This refers to the number of channels allocated to the Y component. In this embodiment, the four downsampling layers in encoder 151 reduce the height and width of the input tensor to 1 / 16 (downsampling), thus reducing the number of channels. The value is 128. The prior representation of the obtained Y component is processed by the super-prior Y pipeline.

[0252] The UV components of the image to be processed are derived from tensors It means that, among them, Similarly, the height and width dimensions are the same, with a channel count of 2 (for the two chroma components). Conditional encoding of the UV components requires auxiliary information from the Y component. If the planar sizes (H and W) of the Y component differ from those of the UV component, a resampling unit is used to align the sample positions in the tensor representing the Y component with those in the tensor representing the UV component. Similarly, if there is an offset between the sample positions in the tensor representing the Y component and those in the tensor representing the UV component, alignment must be performed.

[0253] Concatenate the aligned tensor representations of the Y components with the tensor representations of the UV components to obtain a tensor. The encoder 152, including the neural network, transforms this cascaded tensor into a priori tensor. ,in, This refers to the number of channels allocated to the UV components. In this embodiment, the five downsampling layers in encoder 152 reduce (downsample) both the height and width of the input tensor to 1 / 32, resulting in 64 channels. The resulting prior representation of the UV components is processed by a super-prior UV pipeline, similar to the super-prior Y pipeline (the pipeline operation is also described above). Figure 7 (Description). It should be noted that both the super-prior UV pipeline and the super-prior Y pipeline can include neural networks.

[0254] The super-prior Y pipeline provides an entropy model for entropy decoding of the (quantized) prior representation of the Y components. The super-prior Y pipeline includes a (super)encoder 153, an arithmetic encoder, an arithmetic decoder, and a (super)decoder 154.

[0255] Prior tensors representing the Y component in prior space Further downsampling by (super)encoder 153 through convolution and nonlinear transformation to obtain (possibly after quantization, Figure 15 (Not shown in the text; in fact, any quantization performed by the quantization unit Q is optional here and below) is converted into a super-prior tensor of the bitstream by the arithmetic encoder (AE). The statistical properties of the (quantized) super-prior tensor are estimated using an entropy model (e.g., a factorization entropy model), which the arithmetic encoder (AE) of the super-prior Y pipeline uses to create the bitstream. All elements of the (quantized) super-prior tensor can be written into the bitstream without an autoregressive process.

[0256] The (factorization) entropy model is used as the codebook, and its parameters are available at the decoder. The arithmetic-decoder (AD) of the super-prior Y pipeline recovers the super-prior tensor from the bitstream using the (factorization) entropy model. The recovered super-prior tensor is upsampled by the (super)decoder 154 by applying multiple convolution operations and nonlinear transformations. The prior tensor representing the Y component in the prior space. The quantization is performed by the quantization unit Q of the super-prior Y pipeline, and the entropy of the quantized prior tensor is estimated autoregressively based on the upsampled restored super-prior tensor output by the (super)decoder 154.

[0257] Prior tensors representing the Y component in the prior space Quantization also occurs before the tensor is converted into a bitstream (potentially from the sender to the receiver) by another arithmetic encoder (AE), which uses estimated statistical properties of the tensor provided by the super-prior Y pipeline. (Prior tensor) The prior tensors are reconstructed from the bitstream by another arithmetic decoder (AD) through an autoregressive entropy model provided by the super-prior Y pipeline. The reconstructed prior tensors are then processed by decoder 155 through multiple convolution operations and nonlinear transformations. Perform upsampling to obtain a size of Tensor representation of the reconstructed Y component of the image.

[0258] The output of the super-prior UV pipelined encoder 152, i.e., the prior tensor, is processed. The prior tensor is further downsampled by the (super)encoder 156 of the super-prior UV pipeline through convolution and nonlinear transformation to obtain (possibly after quantization), Figure 15(Not shown) is a super-prior tensor converted into a bitstream by the arithmetic encoder (AE) of the super-prior UV pipeline. The statistical properties of the (quantized) super-prior tensor are estimated using an entropy model (e.g., a factorization entropy model), which the arithmetic encoder (AE) of the super-prior Y pipeline uses to create the bitstream. All elements of the (quantized) super-prior tensor can be written into the bitstream without an autoregressive process.

[0259] The (factorization) entropy model is used as the codebook, and its parameters are available at the decoder. The arithmetic-decoder (AD) of the super-prior UV pipeline reconstructs the super-prior tensor from the bitstream using the (factorization) entropy model. The reconstructed super-prior tensor is upsampled by the (super)decoder 157 of the super-prior UV pipeline by applying multiple convolution operations and nonlinear transformations. The prior tensor representing the UV components... The quantization is performed by the quantization unit Q of the super-prior UV pipeline, and the entropy of the quantized prior tensor is estimated autoregressively based on the upsampled restored super-prior tensor output by the (super)decoder 157.

[0260] Prior tensors representing UV components in prior space Quantization also occurs before the tensor is converted into a bitstream (potentially from the transmitter to the receiver) by another arithmetic encoder (AE), which uses estimated statistical properties of the tensor provided by the super-prior UV pipeline. The prior tensor representing the UV components in the prior space... The data is then recovered from the bitstream by another arithmetic decoder (AD) using an autoregressive entropy model provided by the super-prior UV pipeline.

[0261] In the restored prior tensor After downsampling, the prior tensor representing the reconstruction of UV components in the prior space With the prior tensor of the reduction To perform cascading, that is, to restore the prior tensor. With tensor (As auxiliary information required for UV component decoding) are concatenated to obtain tensors. Tensor The input is fed into the decoder 158 on the UV processing side, and the decoder 158 upsamples the data by applying multiple convolution operations and nonlinear transformations to obtain a value of [size missing]. Tensor representations of the reconstructed UV components of the image. These tensor representations are then combined with the tensor representations of the reconstructed Y components to obtain the reconstructed image in YUV space.

[0262] Figure 16 It shows something similar to Figure 15 The illustrated embodiment is for processing input data in YUV444 format, where the tensors representing the Y and UV components are of the same size in both the height and width dimensions. Encoder 161 will convert the tensor representing the Y component of the image to be processed... The transformation is to a priori space. According to this embodiment, there is no need to resample auxiliary information; therefore, the tensor representing the UV components of the image to be processed... It can be directly compared with the tensor representing the Y component. Perform cascading, and cascade tensors The UV-side encoder 162 transforms the prior space. The operation of the super-prior Y pipeline, including (super)encoder 163 and (super)decoder 164, and the super-prior UV pipeline, including (super)encoder 166 and (super)decoder 167, is similar to the combination described above. Figure 15 The pipeline is described. Since the prior representations of the reconstructed U and UV components have the same size in height and width, they can be cascaded in the prior space without resampling. The prior representation of the reconstructed U component... The concatenated prior representation of the Y and UV components restored by upsampling by decoder 165. The image is upsampled by decoder 168, and the outputs of decoders 165 and 168 are combined to obtain the reconstructed image in YUV space.

[0263] Figure 17 and Figure 18 An example in which conditional residual decoding is provided is shown. Conditional residual decoding can be used for inter-frame prediction of the current frame of a video sequence or for still image decoding. Figure 15 and Figure 16 The illustrated embodiment differs in that it processes the residuals in the YUV space, which include residual components. The residuals are divided into the residuals of the Y component (Y component) and the residuals of the UV components (UV component). The processing of the residual components is similar to that described above. Figure 15 and Figure 16 The processing of the Y and UV components is described. According to... Figure 17In the illustrated embodiment, the input data is in YUV420 format. Therefore, the residual Y component must be downsampled before being concatenated with the residual UV component. Encoders 171 and 172 provide the corresponding prior representations. The operation of the super-prior Y pipeline including (super)encoder 173 and (super)decoder 174 and the super-prior UV pipeline including (super)encoder 176 and (super)decoder 177 is similar to that described above in combination with... Figure 15 The pipeline is described. On the residual Y component side, decoder 175 outputs a reconstructed representation of the residual Y component. On the residual UV side, decoder 178 outputs a reconstructed representation of the residual UV component based on auxiliary information provided in the prior space, wherein the reconstructed prior representation of the residual Y component needs to be downsampled. The outputs of decoders 175 and 178 are combined to obtain the reconstructed residual in the YUV space, which can be used to obtain the reconstructed image (a portion of the image).

[0264] according to Figure 18 The embodiment shown uses YUV444 format as input data. Downsampling of auxiliary information is not required. The processing of the residual Y component and residual UV component is similar to that described above. Figure 16 The processing of the Y and UV components is described. Encoder 181 generates a tensor representing the residual Y component of the image to be processed. Transformed into prior space. A tensor representing the residual UV components of the image to be processed. It can be directly compared with the tensor representing the residual Y component. Perform cascading, and cascade tensors The residual UV side is transformed into the prior space by encoder 182.

[0265] The operation of the super-prior Y pipeline including (super) encoder 183 and (super) decoder 184 and the super-prior UV pipeline including (super) encoder 186 and (super) decoder 187 is similar to the combination described above. Figure 15 The described production line.

[0266] Since the prior representations of the restored residual U-components and residual UV-components have the same height and width, they can be cascaded without resampling. (Prior representations of the restored residual U-components) The cascaded prior representation of the reconstruction of the residual Y component and residual UV component by upsampling by decoder 185. The image is upsampled by decoder 188, and the outputs of decoders 185 and 188 are combined to obtain the reconstruction residual of the image in YUV space, which can be used to obtain the reconstructed image (a part of the image).

[0267] Figure 19 It shows relative to Figure 17Alternative embodiments to the illustrated embodiments. The only difference is that, in Figure 19 The configuration shown does not employ an autoregressive entropy model. (Based on tensor) The representation of the residual Y component is transformed in the prior space by encoder 191. The residual Y component is used as the output tensor. The encoder has 192 pairs of tensors. The residual UV components are represented by auxiliary information for decoding. The super-prior Y pipeline, including the (super)encoder 193 and (super)decoder 194, provides a priori representations for the residual Y components. The side information is used for decoding. The decoder 195 output consists of tensors. The reconstructed residual Y component is represented. The super-prior UV pipeline, including (super)encoder 196 and (super)decoder 197, provides tensors for the output of encoder 192. The prior representation (i.e., tensor) The decoder 198 receives the side information for decoding. The decoder 198 receives the concatenated tensors in the prior space. And output the tensor The UV component of the reconstructed residual is represented.

[0268] Figure 20 It shows relative to Figure 18 Alternative embodiments of the illustrated embodiments. Again, the only difference is that... Figure 20 The configuration shown does not employ an autoregressive entropy model.

[0269] By tensor The representation of the residual Y component is transformed in the prior space by encoder 201. The residual Y component is used as the output tensor. The encoder 202 pairs tensors The residual UV components are represented by auxiliary information for decoding. The super-prior Y pipeline, including the (super)encoder 203 and (super)decoder 204, provides a priori representations for the residual Y components. The side information is used for decoding. The output of decoder 205 consists of tensors. The reconstructed residual Y component is represented. The super-prior UV pipeline, including the (super)encoder 206 and (super)decoder 207, provides tensors for the output of encoder 202. The prior representation (i.e., tensor) The decoder 208 receives the reconstructed representation of the residual UV components in the prior space. And output the tensor The UV component of the reconstructed residual is represented.

[0270] Processing without using an autoregressive entropy model can reduce the overall processing complexity, and depending on the application, the restored image can still have sufficient accuracy.

[0271] according to Figure 21 The illustrated embodiment provides a method for reconstructing at least a portion of an image. In S231, for example, a first bitstream is parsed according to a first entropy model to obtain a first prior tensor, and in S233, the first prior tensor is processed to obtain a first tensor representing the principal components of the image. Furthermore, for example, in S235, a second bitstream different from the first bitstream is parsed according to a second entropy model different from the first entropy model to obtain a second prior tensor different from the first prior tensor. In S237, the first prior tensor is resampled according to an integer factor to obtain a resampled first prior tensor. Then, in S239, a second tensor representing at least one minor component of the image is obtained based on the second prior tensor and the resampled first prior tensor. The integer factor can also be called an integer scaling factor.

[0272] Because integer factors are used to obtain the first prior tensor after resampling, this is highly beneficial for both compression performance and simplicity. The resampling process can be performed without interpolation (a computationally expensive operation), which can correspondingly improve decoding efficiency.

[0273] Integer factors can be calculated based on a first scaling factor of the primary component and a second scaling factor of at least one secondary component. For example, the integer factor is calculated as follows: ,in, Indicates the first scaling factor. This represents the second scaling factor.

[0274] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0275] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0276] As another example, integer factors It can exist in the bitstream or be set as the default value.

[0277] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 2.

[0278] when When it exists in the bitstream, it is obtained by parsing the bitstream. .when When it does not exist in the bitstream, Set as the default value. The default value can be 1.

[0279] Then, Export as ,in, This represents the scaling factor for at least one minor component.

[0280] In the example above, It is a scaling factor for brightness (horizontal and vertical).

[0281] It is the (horizontal and vertical) scaling factor of chroma.

[0282] In other implementations, different horizontal and vertical scaling factors can be used, for example...

[0283] It is the (horizontal) scaling factor of brightness.

[0284] It is the (vertical) scaling factor of brightness.

[0285] It is the (horizontal) scaling factor of chroma.

[0286] It is the (vertical) scaling factor of chroma.

[0287] Integer factors can also include horizontal factors and vertical factors.

[0288] The resampling is performed through nearest neighbor upsampling or nearest neighbor downsampling.

[0289] For nearest neighbor upsampling:

[0290] Represented as The size received by this layer is tensor and output size is tensor .

[0291] This downsampling does not require interpolation; simply copy it.

[0292]

[0293] Where s represents an integer factor.

[0294] For nearest neighbor downsampling:

[0295] Represented as The layer receives a size of Tensor input, and output size is The tensor output, through this process, reduces the spatial resolution of each tensor channel.

[0296] This downsampling does not require interpolation; simply copy it.

[0297]

[0298] Where s represents an integer factor.

[0299] Figure 22 The decoder architecture shown is the implementation Figure 21 An example of the method is shown. Data (tensors and streams) is shown in the "white" box, the neural network modules required for decoding are shown in the gray shaded box, and the toggleable tools are shown in the purple shaded box.

[0300] For the primary and secondary color components, the bitstream can be parsed independently and reconstructed using modules with the same sequence of neural network layers, the only difference being the size of the input tensor and the number of tensor channels.

[0301] First, it can be achieved by a lossless entropy decoder ( Analysis . The probability distribution assumption for lossless decoding is a Gaussian distribution with pre-trained parameters (part of the training model), and the lossless entropy decoder uses a cumulative distribution function calculated based on these pre-trained parameters.

[0302] Decoding the super-prior tensor It is used as input to two different processes (super decoder and ultra-large scale decoder).

[0303] Then, it can be obtained by a lossless entropy decoder ( Analysis Analysis The probability distribution is assumed to be Gaussian, and zero mean and standard deviation are given as the output of the following steps: the output tensor of the VLSI decoder. Then, based on the rate control parameters within the Sigma scale Scale the tensor to generate Then, masking and scaling are performed based on the RVS parameters within the adaptive Sigma scale to generate... Finally, regarding tensors The values ​​are quantized (converted to indices of a probability distribution table). Some elements of the residual tensor are skipped (not encoded / decoded) and replaced with zeros in the decoder skip module, which receives the parsed set of syntax elements {s} from the tANS decoder, receives mask_sigma from the skip mask generation module, and outputs the reconstructed residual tensor reshaped into a 3D shape. .

[0304] At the decoding end, residual The inverting gain unit is based on the parameters Scaling and generating Then, the residual tensor is scaled in the Inverse Residual and Variance Scale (invRVS) module to form a residual tensor. This is the prior tensor used for reconstruction. .

[0305] The super decoder generates the input to the multi-stage context model (MCM). MCM is an eight-level neural network process that also reconstructs the residual. As input, and output the prior space tensor Following Latent Scaling Before Synthesis (LSBS), the reconstructed prior space tensor... Ready to perform signal reconstruction. The prior tensor reconstructions of the primary and secondary components are independent of each other.

[0306] Reconstructed prior space tensor This is the input to the composition transformation. The other input to the composition transformation is the auxiliary tensor. For minor component synthesis, auxiliary tensors are generated based on the prior tensors reconstructed from the major components. For the major components, no auxiliary tensors are used. The input image height is used as a reference. and width and main components ( ) and secondary components ( The scaling factor of the tensor is shown in Table 1.

[0307] Table 1 Tensor size parameters used for decoding primary and secondary components.

[0308]

[0309] For the principal components, the parameters This means that the synthesis transformation of the primary components does not receive any (independently reconstructed) auxiliary information. For the secondary components... The auxiliary tensor synthesized by the quadratic transformation is By using integer factors Prior space tensor of the reconstructed principal components Perform resampling. Resampling can be done using nearest neighbor upsampling or nearest neighbor downsampling as described above.

[0310] In one embodiment, resampling is performed after the synthesis transform and before filtering. Filtering can be any filtering method disclosed in this application. For example, resampling is performed by receiving the output of the synthesis transform and resampling the output.

[0311] Resampling can be performed using nearest neighbor upsampling or nearest neighbor downsampling as described above.

[0312] In one embodiment, if the size of the minor component of the encoded image parameters is not equal to the color sampling mode of the output image, the minor component is encoded at a lower resolution compared to the output size and upsampled. Table 2 lists the supported color sampling modes for the input and output images, and the corresponding ratios between the major component size and the minor component size. For example, the resampling (S237) step can be performed by resampling the first prior tensor according to an integer factor, where the integer factor is c in Table 2. hor or c ver Or according to c in Table 2 hor or c ver Obtain.

[0313] In one embodiment, the decoding process begins by parsing the image header, which includes the image size, quality parameter β, model identifier, chunking, tool, and the color sampling mode (SSD) of the output image. ver s hor ), encoded image parameters (c ver c hor Information such as the size of secondary components and tool information.

[0314] In one embodiment, the next step is z-stream decoding (z stream decoding). Y For the principal components, z UV The flow is for the minor components. The probability distribution table ('z-table') of the z-flow arithmetic decoder is part of the training model. This process generates the three-dimensional hypertensors for the major and minor components respectively. and .

[0315] In one embodiment, the decoded hypertensor It is used as input to two different processes (super decoder and ultra-large scale decoder).

[0316] In one embodiment, the arithmetic decoder for the quality map takes the residual stream 'q stream' as input and outputs a value of . The quality_map.

[0317] In one embodiment, the entropy and residual decoding operations are quantized, and a multiplier of no more than 8 bits is used to generate 16 bits of data at each layer. The quantization operation ensures that the 32-bit integer register does not overflow, thus guaranteeing precise bit behavior. The ultra-large-scale decoder generates logarithmic scales. The residual signal variant on, whose residual tensor size is (For the primary components) and (For minor components). The VAR-scale decoder is followed by 'VarScale'. Operations on the sigma scale are part of the variable-rate support, outputting variance on a logarithmic scale. This tensor is used in the mask generation processes of SKIP, RVS, and LSBS. Furthermore, It will undergo an adaptive sigma scale ('RVS Scale'). Finally, the variance on a logarithmic scale is... Quantization is performed, providing SigmaIdx (for the primary component, size is...) For secondary components, the size is ), SigmaIdx indicates the probability distribution table used by the arithmetic decoder for residual decoding.

[0318] In one embodiment, the arithmetic decoder for the residuals takes the residual stream 'rstream', SigmaIdx (used to derive the probability distribution table), and SkipMask as input. Tensor elements skipped during encoding based on the SkipMask are replaced with zeros. The arithmetic decoder outputs the residual tensor. (For the primary components) and (For secondary components).

[0319] In one embodiment, the residual is descaled in the inverse gain unit module to support variable rates and further modified in the inverse RVS.

[0320] In one embodiment, the superdecoder generates the prediction tensor. and The predictive tensor in prior tensor reconstruction and the residual and Combined. For the main components, a multi-stage contextual model is used to generate a priori representation of the image for reconstruction: (For the primary component). For the secondary component, only the residuals are added to the prediction, generating... .

[0321] In one embodiment, the primary and secondary color component bitstreams can be parsed, and the prior tensor can be reconstructed independently. and .

[0322] In one embodiment, an LSBS process precedes the synthesis transformation (if the LSBS tool is enabled). Several synthesis transformations with different architectures and parameters are defined. The synthesis transformations for the primary and secondary components are specified by the DecoderID. Any synthesis transformation network can be used based on prior tensors. and Image reconstruction. Sub-clause 10.3 specifies three different synthesis networks (DecoderID=0, 1, 2). The synthesis transform of the principal components only accepts... As input (the primary components can be reconstructed independently). The secondary components are synthesized and transformed. and As input, the output of the synthetic transform network has the size of the output image. (For the primary components) and .

[0323] In one embodiment, These color components enter an optional filter module (see Appendix I), which consists of several chroma filters and a luminance edge filter.

[0324] In one embodiment, the analysis transform network generates a size of... (For the primary components) and The tensor y (for minor components). The size of the supertensor z is (For the primary components) and (For secondary components). Table 2 lists the supported color sampling modes for the input and output images, as well as the corresponding ratios between the primary component size and the secondary component size.

[0325] Table 2 Supported color sampling modes and scaling factors

[0326]

[0327] exist Figure 22 In the example shown, It is the first prior tensor. It is the first prior tensor after resampling. or It is the first tensor representing the principal components of the image.

[0328] It is the second prior tensor. or It is a second tensor representing at least one minor component of an image.

[0329] The synthesis transformation of the primary and secondary components involves the same neural network layers, differing only in the size of the input tensor and the number of tensor channels. The output tensor of the synthesis transformation is... (The size of the tensor is listed in Table 1).

[0330] like Figure 22 As shown, after reconstruction, both the primary and secondary components are resampled (the upsampling module is in...). Figure 22 The Chinese character is represented as Each component has its own scaling factor. After resampling to the original image size, all three color components are passed through an Inter-Channel Correlation Information (ICCI) filter.

[0331] The reconstruction process uses inverse color transformation ( Figure 22 "in ")Finish.

[0332] Figure 23 The decoder architecture shown is the implementation Figure 21 Another example of the method shown. Figure 23 and Figure 22 The difference is that, in Figure 22 China Perform nearest neighbor downsampling, while Figure 23 China Perform nearest neighbor upsampling.

[0333] Figure 22 and Figure 23 Examples of signal decoders in [the following text is incomplete and likely refers to a separate topic:] Figure 24 As shown. The signal decoder can also be called a synthesis transform. Learning-based reconstruction (called a synthesis transform) involves two pipelines with identical neural network architectures but different input sizes and number of channels.

[0334] The inputs to the analysis transform include:

[0335] With auxiliary information tensor The cascaded shape is The prior space tensor of the reconstruction

[0336] Operation point indicator ,

[0337] Size of input / output tensors ,

[0338] The synthetic transform network consists of a pair Defined model parameters.

[0339] The output of the analysis transformation is a tensor of size . Reconstructed color components .

[0340] The sizes of these tensors for the primary and secondary components are listed in Table 1.

[0341] The synthesis transformation from the cascaded main input ( ) and secondary input ( Begin with the operation point indicator (). The decoder performs the following series of steps.

[0342] At the basic operation point ( Under this condition, the number of channels is A lightweight residual block is followed by a clipping layer (step size is...) The depth is correspondingly and ) and kernel size The kernel size of the residual activation unit combination is Two transposed convolutions. The number of output channels in the transposed convolution is correspondingly... and The stride of both transposed convolutions is . The next step in this process is with the residual activation unit (kernel size: The kernel size of the combination is Step size is 1 and the number of channels The invariant regular convolution. Then there exists a stride of... The number of channels will be increased from Increase to convolution This is done to ensure the next step (i.e., the step size is...) The output of (pixel recombination) has a number of channels This process uses trimming layers (step size is...) Depth is )Finish.

[0343] At advanced operation points ( Under this condition, the number of channels is After the two residual blocks, there is a clipping layer (step size is...) The depth is correspondingly and ) and kernel size The kernel size of the residual activation combinatorial is Two transposed convolutions. The number of output channels in both transposed convolutions is... The stride of both transposed convolutions is . The next step in this process is to have a kernel size of [missing information]. The step size is 1 and the number of output channels is The regular convolution. This is done to ensure that the next step (i.e., with a stride of...) is... The output of (pixel recombination) has a number of channels Then, execute the clipping layer (with a stride of 1). Depth is ) and kernel size The residual nonlocal attention block of the residual activation combination (where, This process is based on the kernel size. Step size is And the number of output channels is The transposed convolution and the subsequent clipping layer (with stride of ) Depth is )Finish.

[0344] Figure 25 A processing apparatus 250 is provided for reconstructing at least a portion of an image. The processing apparatus 250 includes components for performing operations such as... Figure 21 The processing circuit 255 of the method shown.

[0345] Some exemplary implementations in hardware and software

[0346] The corresponding systems that can be deployed in the above encoder-decoder processing chain are as follows: Figure 26 As shown. Figure 26 A schematic block diagram of exemplary decoding systems, such as video, image, audio, and / or other decoding systems (or simply decoding systems), that can utilize the techniques of this application is shown. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) in the video decoding system 10 are two examples, i.e., devices that can be used to perform various techniques according to the various examples described in this application. For example, video decoding and decoding can employ methods such as... Figures 1 to 6 The neural network shown can be distributed, and the above-described bitstream parsing and / or bitstream generation can be applied to transmit feature maps between distributed computing nodes (two or more).

[0347] like Figure 26 As shown, the decoding system 10 includes a source device 12, which provides encoded image data 21, for example, to a destination device 14; the destination device 14 decodes the encoded image data 13.

[0348] The source device 12 includes an encoder 20 and may additionally (i.e., optionally) include an image source 16, a preprocessor (or preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22.

[0349] Image source 16 may include or may be any type of image capture device such as a camera for capturing real-world images, and / or any type of image generation device such as a computer graphics processor for generating computer-animated images, or any other device of any type for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). Image source may be any type of memory or storage for storing any of the aforementioned images.

[0350] To distinguish between the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0351] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component. It should be noted that preprocessing may also employ neural networks (such as... Figures 1 to 7 (Any of the diagrams shown in the figure) This neural network uses presence indicator signals.

[0352] The video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0353] The communication interface 22 of the source device 12 can be used to receive encoded image data 21 and send the encoded image data 21 (or any further processed version thereof) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0354] Destination device 14 includes decoder 30 (e.g., video decoder 30) and may additionally (i.e., optionally) include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0355] The communication interface 28 of the destination device 14 is used to receive encoded image data 21 (or any further processed version thereof) directly from the source device 12 or from any other source end (e.g., encoded image data storage device) and provide the encoded image data 21 to the decoder 30.

[0356] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link between source device 12 and destination device 14 (e.g., a direct wired or wireless connection) or via any type of network (e.g., a wired network or wireless network or any combination thereof, any type of private network and public network or any combination thereof).

[0357] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format (e.g., a data packet) and / or process the encoded image data through any type of transmission encoding or processing method for transmission over a communication link or communication network.

[0358] For example, the corresponding part of the communication interface 22, the communication interface 28, can be used to receive the transmitted data and process the transmitted data through any type of corresponding transmission decoding or processing and / or decapsulation method to obtain the encoded image data 21.

[0359] Both communication interface 22 and communication interface 28 can be configured as unidirectional communication interfaces (e.g., Figure 26 The communication channel 13 (as shown by the arrow pointing from source device 12 to destination device 14) or a bidirectional communication interface can be used, for example, to send and receive messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). The decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (e.g., using a communication channel 13 based on source device 12 to destination device 14) or a bidirectional communication interface, and can be used for, for example, sending and receiving messages to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). Figures 1 to 7 (a neural network of one or more graphs in the graph).

[0360] The post-processor 32 in the destination device 14 is used to post-process the decoded image data 31 (also known as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0361] The display device 34 in the destination device 14 is used to receive post-processed image data 33 in order to display the image to a user or viewer. The display device 34 can be or can include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS) display, a digital light processor (DLP), or any other type of display.

[0362] although Figure 26 Source device 12 and destination device 14 are described as separate devices; however, device embodiments may also include two devices or two functions, namely source device 12 or its corresponding function and destination device 14 or its corresponding function. In these embodiments, source device 12 or its corresponding function and destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software or any combination thereof.

[0363] According to the description, Figure 26 The different units or functions that are (accurately) divided in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application, which is obvious to those skilled in the art.

[0364] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry, such as one or more microprocessors, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more discrete logic devices, one or more hardware devices, one or more dedicated video decoding devices, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to incorporate neural networks (e.g., Figures 1 to 6 Various modules (of or part thereof) of the neural network shown in any of the figures. Decoder 30 can be implemented via processing circuitry 46 to embody the combination of... Figures 1 to 7The various modules described herein and / or any other decoder systems or subsystems described herein. Processing circuitry can be used to perform the various operations discussed below. If the above techniques are implemented in part in software, the device can store the instructions for that software in a suitable non-transitory computer-readable storage medium, and these instructions can be executed in hardware by one or more processors to perform the techniques of this disclosure. Either the video encoder 20 and the video decoder 30 can be integrated into a single device as part of a combined encoder / decoder (codec), such as... Figure 27 As shown.

[0365] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or notebook computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0366] In some cases, Figure 26 The video decoding system 10 shown is merely an example, and the techniques in this application can be applied to video decoding setups (e.g., video encoding or video decoding) that do not necessarily include any data communication between encoding and decoding devices. In other examples, data is retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode it.

[0367] Figure 28 This is a schematic diagram of a video decoding device 2000 according to an embodiment of the present disclosure. The video decoding device 2000 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 2000 may be a decoder (e.g., Figure 26 The video decoder 30 or encoder (e.g.) in the video decoder 30 Figure 26 (Video encoder 20 in the middle).

[0368] The video decoding device 2000 includes an input port 2010 (or input port 2010) and a receiving unit (Rx) 2020 for receiving data; a processor, logic unit, or central processing unit (CPU) 2030 for processing data; a transmitting unit (Tx) 2040 and an output port 2050 (or output port 2050) for transmitting data; and a memory 2060 for storing data. The video decoding device 2000 may also include optical-to-electrical (OE) components and electro-optical (EO) components coupled to the input port 2010, the receiving unit 2020, the transmitting unit 2040, and the output port 2050, serving as outputs or inputs for optical or electrical signals.

[0369] Processor 2030 is implemented through hardware and software. Processor 2030 can be implemented as one or more CPU chips, one or more cores (e.g., as a multi-core processor), one or more FPGAs, one or more ASICs, and one or more DSPs. Processor 2030 communicates with ingress port 2010, receiver unit 2020, transmitter unit 2040, egress port 2050, and memory 2060. Processor 2030 includes decoding module 2070. Decoding module 2070 implements the disclosed embodiments described above. For example, decoding module 2070 implements, processes, prepares, or provides various decoding operations. Therefore, including decoding module 2070 provides a substantial improvement to the functionality of video decoding device 2000 and enables transitions of video decoding device 2000 to different states. Alternatively, decoding module 2070 can be implemented with instructions stored in memory 2060 and executed by processor 2030.

[0370] Memory 2060 may include one or more disks, one or more tape drives, and one or more solid-state drives, and may be used as an overflow data storage device to store programs as selected for execution, as well as instructions and data read during program execution. For example, memory 2060 may be volatile and / or non-volatile memory, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0371] Figure 29This is a simplified block diagram of a device 800 according to an exemplary embodiment, wherein the device 800 can be used as Figure 26 The source device 12 and / or the destination device 14.

[0372] The processor 2102 in device 2100 may be a central processing unit. Alternatively, the processor 2102 may be any other type of device or multiple devices, existing or to be developed in the future, capable of operating or processing information. While the disclosed implementation may be carried out using a single processor such as the processor 2102 shown in the figure, using multiple processors can provide advantages in terms of speed and efficiency.

[0373] In one implementation, the memory 2104 in device 2100 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 2104. Memory 2104 may include code and data 2106 accessed by processor 2102 via bus 2112. Memory 2104 may also include an operating system 2108 and an application program 2110, which includes at least one program that causes processor 2102 to perform the methods described herein. For example, application program 2110 may include applications 1 to N, and also includes a video decoding application that performs the methods described herein.

[0374] Device 2100 may also include one or more output devices, such as display 2118. In one example, display 2118 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. Display 2118 may be coupled to processor 2102 via bus 2112.

[0375] Although bus 2112 in device 2100 is described herein as a single bus, bus 2112 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components in device 2100 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 2100 can be implemented in a variety of configurations.

[0376] also, Figure 25 The processing device 250 shown may include Figure 26 The source device 12 or destination device 14 shown Figure 27 The video decoding system 40 shown Figure 28 The video decoding device shown is 2000 or Figure 29 The device 2100 shown.

Claims

1. A method for reconstructing at least a portion of an image, characterized in that, include: Parse (S231) the first bitstream to obtain the first prior tensor; Process (S233) the first prior tensor to obtain a first tensor representing the principal components of the image; Parse (S235) a second bitstream that is different from the first bitstream to obtain a second prior tensor that is different from the first prior tensor; The first prior tensor is resampled according to an integer factor (S237) to obtain the resampled first prior tensor. Based on the second prior tensor and the resampled first prior tensor, obtain (S239) a second tensor representing at least one minor component of the image.

2. The method according to claim 1, characterized in that, The resampling (S237) is performed based on the integer factor without interpolation.

3. The method according to claim 1 or 2, characterized in that, The integer factor is calculated based on a first scaling factor of the primary component and a second scaling factor of the at least one secondary component.

4. The method according to claim 3, characterized in that, The integer factor is calculated as follows: ,in, This represents the first scaling factor. This represents the second scaling factor.

5. The method according to claim 4, characterized in that, It exists in the bitstream or is set as the first default value. It exists in the bitstream or is set as the second default value.

6. The method according to claim 5, characterized in that, The first default value is 1. The second default value is 2.

7. The method according to claim 1 or 2, characterized in that, The integer factor exists in the bitstream or is set as the first default value.

8. The method according to claim 7, characterized in that, It exists in the bitstream or is set as the second default value. Export as , The first scaling factor represents the primary component. The second scaling factor represents the at least one minor component. This represents the integer factor.

9. The method according to claim 8, characterized in that, The first default value is 2. The second default value is 1.

10. The method according to any one of claims 1 to 9, characterized in that, The resampling is performed through nearest neighbor upsampling or nearest neighbor downsampling.

11. The method according to claim 10, characterized in that, The size of the tensor input of the nearest neighbor upsampling is The size of the tensor output of the nearest neighbor upsampling is , where s represents the integer factor, and the nearest neighbor upsampling does not involve interpolation.

12. The method according to claim 10, characterized in that, The size of the tensor input of the nearest neighbor downsampling is The size of the tensor output of the nearest neighbor downsampling is , where s represents the integer factor, and the nearest neighbor downsampling does not involve interpolation.

13. The method according to any one of claims 1 to 12, characterized in that, The primary component of the image is the luminance component, and the at least one secondary component of the image is the chrominance component.

14. The method according to any one of claims 1 to 13, characterized in that, The second tensor represents two minor components, one of which is a chromaticity component and the other is another chromaticity component.

15. The method according to any one of claims 1 to 14, characterized in that, The processing (S234) of the first prior tensor includes transforming the first prior tensor into the first tensor.

16. The method according to claim 15, characterized in that, The step of obtaining a second tensor representing at least one minor component of the image includes: The second prior tensor is concatenated with the resampled first prior tensor to obtain a concatenated tensor, and the concatenated tensor is transformed into the second tensor.

17. The method according to any one of claims 1 to 16, characterized in that, The first bitstream is parsed by a first neural network, and the second bitstream is parsed by a second neural network different from the first neural network.

18. The method according to claim 16, characterized in that, The first prior tensor is transformed by a third neural network, and the cascaded prior tensor is transformed by a fourth neural network different from the third neural network.

19. The method according to any one of claims 1 to 18, characterized in that, The first tensor represents the principal residual component of the residual of the principal component of the image; The second tensor uses information from the first prior tensor to represent at least one minor residual component of the residual of at least one minor component of the image.

20. The method according to any one of claims 3 to 6 and claims 8 and 9, characterized in that, The first scaling factor includes a first horizontal scaling factor and a first vertical scaling factor.

21. The method according to any one of claims 3 to 6, 8 and 9, and 20, characterized in that, The second scaling factor includes a second horizontal scaling factor and a second vertical scaling factor.

22. A computer program stored on a non-transient medium, characterized in that, The computer program includes code that, when executed on one or more processors, performs the steps of the method according to any one of claims 1 to 21.

23. A processing apparatus (40, 250, 2000, 2100) for reconstructing at least a portion of an image, characterized in that, The processing device (40, 250, 2000, 2100) includes: One or more processors (43, 255, 2030, 2102); A non-transitory computer-readable storage medium coupled to the one or more processors and storing a program executable by the one or more processors, wherein, when executed by the one or more processors, the program causes the apparatus to perform the method according to any one of claims 1 to 21.

24. A processing apparatus (250) for reconstructing at least a portion of an image, characterized in that, The processing apparatus (40, 250, 2000, 2100) includes a processing circuit (255) for performing the method according to any one of claims 1 to 21.

Images