Method for chroma sub-sampled format handling in machine learning-based picture coding

Abstract

To provide video coding, an encoder, and a decoder that further improve efficiency on the basis of a trained network.SOLUTION: In order to handle different sized luma-chroma channels, a chroma component is up-sampled such that an obtained up-sampled chroma component has a resolution matching one of a luma component. The luma and the up-sampled chroma component are then encoded into a bitstream. To reconstruct the picture portion, the luma component and an intermediate chroma component matching the resolution of the luma component are decoded from the bitstream, and subsequently, the intermediate chroma component is down-sampled. A sub-sampled chroma format is handled by an auto-encoder / auto-decoder framework, while preserving the luma channel.SELECTED DRAWING: Figure 13

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of picture or video encoding and decoding, and more particularly to handling color formats with subsampled chroma in trained network-based video coding. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital television, video transmission over the Internet and mobile networks, real-time conversation applications such as video chat, video conferencing, DVDs and Blu-ray Discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] The amount of video data required to depict even a relatively short video can be substantial, which can pose difficulties when the data is streamed or otherwise communicated over communication networks that have limited bandwidth capacity. Thus, video data is typically compressed before being communicated over today's telecommunications networks. The size of the video can also be an issue when the video is stored on a storage device, as memory resources can be limited. Video compression devices often use software and / or hardware to code the video data at the source before transmission or storage, thereby reducing the amount of data required to represent a digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. With limited network resources and a constantly increasing demand for higher video quality, improved compression and decompression techniques that improve compression ratios with little or no sacrifice in picture quality are desirable.

[0004] Recently, machine learning (ML) has been applied to image and video coding. In general, machine learning can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding schemes have been discussed. More broadly, machine learning is a subset of artificial intelligence, where a model, typically a network model, is built on example data, often called training data. Once trained, such ML-based techniques then make predictions or decisions based on input data that may not have been used as training data. This means that after training, the ML network is trained and is called a trained network.

[0005] There are many network models that are based on ML techniques, including support vector machines (SVM), neural networks (NN), artificial neural networks (ANN), fully connected neural networks (FCN), convolutional neural networks (CNN), to name a few. These types of networks often have in common that their network structures involve multiple layers (i.e., at least one layer) of network nodes, whereby some or all of the nodes in one layer may be connected to some of the nodes in all of the nodes in the layer(s) before or after said one layer.

[0006] It is desirable to further improve the efficiency of such video coding (video coding or still picture coding) based on trained networks. Summary of the Invention [Means for solving the problem]

[0007] Some embodiments of the present disclosure provide methods and apparatus for encoding and decoding of picture portions having luma and downsampled chroma components as input. Correspondingly, the chroma components are upsampled to match the luma components, and encoding by trained networks is applied to components with matching sizes. This may provide more efficient encoding and decoding, since trained networks with similar designs (input dimensions, resolution) may be applied. Spatial correlation between luma and chroma may also be preserved in this way.

[0008] These and other objects are achieved by the subject matter of the independent claims. Further implementations become evident from the dependent claims, the description and the figures.

[0009] According to one aspect of the present disclosure, a method for encoding a picture portion is provided, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the method including: upsampling the chroma components to obtain upsampled chroma components that match the resolution of the luma component; and encoding the luma component and the upsampled chroma components into a bitstream.

[0010] This provides as output an equal-sized luma-chroma output, which can then be handled by a trained network (e.g., in a machine learning framework) with a similar structure, in particular the same input resolution. If the resolution of all color components is the same, the machine learning framework can exploit the cross-color correlation in the same way at all sample positions. In the process, the luma channel can remain unaffected, in which case there is no quality loss.

[0011] In some example implementations, the upsampling of the chroma components is based on samples of the luma component, which may allow upsampling to be performed based on the content of the luma component.

[0012] According to an exemplary implementation of the method, the upsampling step further comprises processing the chroma components by a trained network. Coding with a trained network may provide improved performance thanks to the possibility of training such a network on desired data.

[0013] In one implementation, the processing with the trained network includes inputting the luma and chroma components to the trained network and outputting the upsampled chroma component from the trained network. The use of the luma component as an input to the trained network may preserve inter-component information between the luma and chroma channels.

[0014] According to one example, the trained network comprises at least two processing layers, a first of which performs upsampling based on input chroma components and a second of which performs processing of the output of the first layer together with the luma component. The combined processing may allow improved exploitation of correlation between luma and chroma and may allow for a common network design for multiple components.

[0015] In one implementation of the method, processing the output of the first layer together with the luma component includes element-wise summing of the luma component and the upsampled chroma component. Thus, chroma upsampling using the luma as side information can be performed efficiently since the luma contains more information, for example about the shape of an object. Moreover, details that may be present in the luma, which originally has a higher resolution, can be restored in the originally subsampled chroma.

[0016] In one example, the upsampling step includes inserting a line of samples after every n lines of samples of a chroma component, where the lines are rows and / or columns and n is an integer equal to or greater than 1. This example implementation allows for low-complexity and efficient upsampling insertion of every two rows or columns having a certain predefined value.

[0017] According to one example, the upsampling step includes inserting a line of samples having the value 0 after every n lines of samples of the chroma components, where the lines are rows and / or columns and n is an integer greater than or equal to 1. This is a simple scheme for upsampling by simply setting each line sample to 0.

[0018] According to a further implementation of the method, the upsampling step includes inserting, after every n lines of samples of the chroma components, a line of samples having a value equal to the respective nearest neighbor sample in the chroma components, where the lines are rows and / or columns and n is an integer equal to or greater than 1. This allows further adaptation of the chroma upsampling by taking into account the content of the neighboring samples in a particular row and / or column. Furthermore, using the values ​​of the inserted line samples with reference to the nearest neighbor samples of chroma, the accuracy of the chroma upsampling may be increased because neighboring information (reflected by the neighboring chroma sample values) with reference to the original chroma is used.

[0019] According to one example, the encoding is performed by a variational autoencoder (VAE), which can be advantageous to perform the tasks related to the encoding and decoding of images (pictures) very efficiently, with a good compromise between the required bit rate and the quality of the reconstructed image / picture.

[0020] According to one aspect of the present disclosure, a method is provided for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the method including: decoding the luma component and an intermediate chroma component that matches the resolution of the chroma component from the bitstream; and downsampling the intermediate chroma component to obtain the chroma component of the picture portion. Thus, the original chroma component (i.e., the subsampled chroma) can be reconstructed from the intermediate chroma component. In particular, the picture portion having the original luma-chroma format can be reconstructed with high quality. Thus, the decoder can provide as output the reconstructed picture portion in a chroma format, the chroma having a lower resolution (i.e., size) than the luma. Furthermore, the luma information and spatial information of Y are mostly preserved since the luma is not downsampled. In general, the advantages provided on the decoding side may be similar to those provided on the encoder side by the corresponding additional features of the respective encoding method.

[0021] According to one exemplary implementation, the downsampling further includes processing the intermediate chroma components by a trained network. By providing a bitstream based on the upsampled chroma, the decoding side may reconstruct a picture (still or video picture) at a higher quality and / or a lower rate. The trained network may provide adaptive and efficient coding, thus providing improved quality at the same rate or improved rate at the same quality.

[0022] For example, the step of processing with the trained network includes inputting intermediate chroma components to the trained network and outputting the chroma components from the trained network. Thus, the trained network may be designed with similar resolution at the luma and chroma inputs, thus reducing complexity and providing low processing overhead. For example, in some embodiments, it may not be necessary to use luma as side information to recover chroma.

[0023] According to one implementation of the method, the trained network comprises at least two processing layers, a first of which processes the input intermediate chroma components and a second of which downsamples the output of the first layer. This exemplary network architecture may enable obtaining the original resolution based on a bitstream generated using the upsampled chroma.

[0024] In another example, the downsampling of the intermediate chroma components is based on samples of the luma component, so the downsampling of the chroma channels may take into account the content of the luma channel.

[0025] According to one implementation of the method, the step of processing with the trained network includes inputting the luma component to the trained network.

[0026] For example, the first layer performs a step of processing intermediate chroma components together with the luma component, so that inter-component information between the luma channel and the chroma channel(s) is preserved.

[0027] In a further example of the method, the downsampling step includes removing a line of samples after every n lines of samples of the intermediate chroma component, where the lines are rows and / or columns and n is an integer greater than or equal to 1. Thus, downsampling of the intermediate chroma can be performed quickly by simply removing the respective line samples.

[0028] For example, the decoding may be performed by a variational autoencoder (VAE), so the decoding may be performed efficiently and with a good compromise between the required bit rate and / or the quality of the reconstructed pictures.

[0029] According to one aspect of the present disclosure, there is provided an encoder for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than the resolution of the luma component, the encoder comprising processing circuitry configured to perform any of the steps of the encoding methods described above or referred to herein.

[0030] According to one aspect of the preset disclosure, there is provided a decoder for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the decoder comprising processing circuitry configured to perform any of the steps of the decoding methods described above and herein.

[0031] According to one aspect of the present disclosure, there is provided a computer program stored on a non-transitory medium comprising code which, when executed on one or more processors, performs the steps of any of the methods described above and herein.

[0032] According to one aspect of the disclosure, there is provided an encoder for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the encoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the encoder to perform any of the encoding methods described above and herein.

[0033] According to one aspect of the present disclosure, there is provided a decoder for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the decoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the decoder to perform any of the decoding methods described above and herein.

[0034] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the encoder comprising: an upsampling module configured to upsample the chroma components to obtain upsampled chroma components that match the resolution of the luma component; and an encoding module configured to encode the luma component and the upsampled chroma components into a bitstream.

[0035] According to one aspect of the present disclosure, a decoder is provided for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the decoder comprising: a decoding module configured to decode the luma component and an intermediate chroma component that matches the resolution of the chroma component from the bitstream; and a downsampling module configured to downsample the intermediate chroma component to obtain a chroma component of the picture portion.

[0036] The embodiments provide for efficient encoding and / or decoding of picture portions having luma and upsampled chroma components, in other words, picture portions to be encoded and / or decoded may have different luma-chroma formats, i.e., luma and chroma channels of different sizes.

[0037] It should be noted that the present disclosure is not limited to any particular framework. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, image generation, and recognition systems.

[0038] The present invention can be implemented in hardware and / or software.

[0039] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0040] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims.

[0041] The following embodiments of the invention are explained in more detail with reference to the accompanying figures and drawings. [Brief description of the drawings]

[0042] [Figure 1] 1 shows examples of luma-chroma YUV formats including (a) YUV444, (b) YUV422, and YUV420. [Figure 2A] FIG. 2 is a schematic diagram illustrating the inputs and outputs of an artificial neural network layer process. [Figure 2B] FIG. 2 is a block diagram illustrating portions of an example autoencoder. [Figure 2C] FIG. 1 is a schematic diagram illustrating a variational autoencoder architecture including a hyper-prior model. [Diagram 3] 1 shows a schematic diagram of a VAE structure including an ultra prior model with downsampling of the luma component. [Figure 4] 1 shows a schematic diagram of the VAE-based encoder-decoder architecture of the present disclosure, which performs upsampling and downsampling of chroma channels using the luma component as side information. [Diagram 5] 13 shows a schematic diagram of a size equalization network for encoder-side chroma upsampling in a VAE architecture, using the luma component as side information. [Figure 6] 13 shows a schematic diagram of a size conversion network for decoder-side (intermediate) chroma downsampling of a VAE architecture using the luma component as side information. [Figure 7] 13 shows a schematic diagram of a size equalization network for encoder-side chroma upsampling in a VAE architecture that does not use the luma component as side information. [Figure 8] 13 shows a schematic diagram of a size conversion network for decoder-side (intermediate) chroma downsampling in a VAE architecture that does not use the luma component as side information. [Figure 9] 1 shows an example of chroma upsampling by inserting zeros in odd rows and odd columns. [Figure 10]1 shows an example of chroma downsampling by removing zeros in odd rows and odd columns. [Figure 11] 13 shows another example of chroma upsampling by inserting zeros in odd rows and columns with values ​​equal to the respective nearest neighbor samples in the chroma components. [Figure 12] 4 illustrates another example of chroma downsampling by removing sample values ​​at odd rows and odd columns corresponding to nearest samples in the chroma components. [Figure 13] 1 shows a flowchart of an encoding method that includes upsampling subsampled chroma to an intermediate chroma having the same size as luma. [Figure 14] 1 shows a flowchart of a decoding method including steps of decoding intermediate chroma from a bitstream and downsampling the intermediate chroma to a subsampled chroma having a different size than the luma. [Figure 15] 2 shows a block diagram of an encoder including an upsampling module and an encoding module. [Figure 16] 2 shows a block diagram of a decoder including a decoding module and a downsampling module. [Figure 17] 1 shows a schematic diagram of an encoder that performs luma downsampling without the chroma component(s) (upper branch) and chroma upsampling without the luma component (lower branch) to determine the cost based on rate-distortion optimization (RDO) and its signaling in the bitstream. [Figure 18] 1 shows a schematic diagram of a decoder that performs either luma upsampling without using the chroma component(s) (top branch) or chroma downsampling without using the luma component (bottom branch) depending on an indicator parsed from the encoded bitstream. [Figure 19]1 shows a schematic diagram of a size equalization network for encoder-side luma downsampling in a VAE architecture that does not use chroma component(s) as side information. [Figure 20] 13 shows a schematic diagram of a size equalization network for encoder-side chroma upsampling in a VAE architecture that does not use the luma component as side information. [Figure 21] 1 shows a schematic diagram of a size conversion network for decoder-side (intermediate) luma upsampling in a VAE architecture that does not use chroma component(s) as side information. [Figure 22] 1 shows a schematic diagram of a size conversion network for decoder-side (intermediate) chroma downsampling (one or more) in a VAE architecture, without using the luma component as side information. [Diagram 23] 1 shows a schematic diagram of an encoder that performs luma downsampling using chroma component(s) (upper branch) and chroma upsampling using luma component (lower branch) to determine the cost based on rate-distortion optimization (RDO) and its signaling in the bitstream. [Figure 24] 1 shows a schematic diagram of a decoder that performs either luma upsampling using chroma component(s) (top branch) or chroma downsampling using luma component (bottom branch) depending on an indicator parsed from the encoded bitstream. [Diagram 25] 1 shows a schematic diagram of a size equalization network for encoder-side luma downsampling of a VAE architecture using chroma component(s) as side information. [Figure 26] 13 shows a schematic diagram of a size equalization network for encoder-side chroma upsampling in a VAE architecture, using the luma component as side information. [Figure 27]1 shows a schematic diagram of a size conversion network for (intermediate) luma upsampling on the decoder side of a VAE architecture, using chroma component(s) as side information input to the network. [Figure 28] 1 shows a schematic diagram of a size conversion network for (intermediate) chroma downsampling on the decoder side of a VAE architecture, using the luma component as side information input to the network. [Figure 29] 1 shows a schematic diagram of an encoder that performs luma downsampling without (one or more) chroma components (upper branch) and chroma upsampling without (lower branch) the luma component to determine the cost based on rate-distortion optimization (RDO) without signaling an indicator in the bitstream. [Diagram 30] 1 shows a schematic diagram of a decoder that performs either luma upsampling without using the chroma component(s) (upper branch) or chroma downsampling without using the luma component (lower branch) depending on the result of a decision module regarding the sampling mode. [Diagram 31] 1 shows a flowchart of an encoding method that includes selecting one of the picture processes (luma downsampling or chroma upsampling) based on the lowest cost and generating a bitstream from the processed picture portions. [Diagram 32] 1 shows a flowchart of a decoding method including steps of decoding luma-chroma from a bitstream and determining one of the picture processing (luma upsampling or chroma downsampling) to reconstruct the original picture portion from the processed picture portion. [Diagram 33] 1 shows a block diagram of an encoder including a selection module, a processing module, and a generation module. The selection module may further include a downsampler (luma) and an upsampler (chroma). [Diagram 34]1 shows a block diagram of a decoder including a decoding module, a decision module, and an acquisition module. The decision module may further include a downsampler (chroma) and an upsampler (luma). [Diagram 35] 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Diagram 36] 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention. [Figure 37] 1 is a block diagram showing an example of an encoding device or a decoding device. [Figure 38] FIG. 2 is a block diagram showing another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0043] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and which show, by way of example, certain aspects of embodiments of the invention or in which embodiments of the invention may be used. It is understood that embodiments of the invention may be used in other ways and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description is not to be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0044] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, where one or more particular method steps are described, the corresponding device may include one or more units, e.g., functional units, for performing the described one or more method steps (e.g., one unit performing one or more steps, or multiple units each performing one or more of the multiple steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, where a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step for performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more functions of multiple units), even if such one or more steps are not explicitly described or illustrated in the figures. Furthermore, it is understood that features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise specified.

[0045] The following provides a brief overview of some of the technical terms used throughout this disclosure.

[0046] Color format of a video sequence: A color format describes a particular organization of colors within a video sequence. Usually, a color format is expressed as a mathematical model that describes how colors can be represented, for example, as a tuple of numbers. For example, the RGB format describes that a frame (picture or image) is combined with red (R), green (G), and blue (B) color components. In contrast, the YUV format describes that a frame is combined with one luma component (Y) and two chroma components (U, V). Luma thereby refers to the luminance component of a sample, pixel, or group of samples / pixels, such as a picture or a picture portion. In general, a color format may have one or more color components.

[0047] RGB: The RGB color format is an additive color model and includes red (R), green (G), and blue (B) components. A picture in the RGB color space typically has components that have the same size in terms of their respective width and height, such that the width and height of the frame correspond to the number of samples or pixels, and thus the resolution. These RGB components can be added in various ways to represent a wide range of colors.

[0048] YUV: The YUV color format contains three components, the luma component Y refers to brightness information (i.e. luminance), and the chroma components U (blue projection) and V (red projection) refer to chrominance or color information. In digital encoding of color information, YUV can be directly converted from RGB by a linear matrix, as exemplified below.

number

[0049] In other words, each of the YUV components may be written by a linear combination of the RGB components, so that after such a transformation, the three components of YUV also have the same size (width and height).

[0050] Chroma Subsampling: Chroma subsampling is a technique for encoding images by implementing a lower resolution for the chroma information (U,V) than for the luma information (Y). The reason for subsampling the chroma components is to take advantage of the human visual system's poorer acuity for color difference than for luma. Thus, as far as the luma visual system is concerned, the luma component can be considered the most important channel when using the YUV (or YCbCr) format.

[0051] Chroma subsampling can significantly reduce the chroma component signals. Commonly used chroma subsampling formats are briefly described below. YUV4:4:4 (or YUV444 for short) refers to no chroma subsampling, i.e., the Y, U, and V components have the same size (width and height), as shown in FIG. 1(a). YUV4:2:2 (or YUV422 for short) refers to subsampling the chroma components (U and V) horizontally by a factor of 2, as shown in Figure 1(b). For example, if the luma component has a size of w x h, where w represents the width of the luma component and h represents the height of the luma component, then the chroma components U and V of YUV4:2:2 each have a size of w / 2 x h (e.g., half the width, i.e., w / 2, and the same height h as the luma component, in terms of number of samples or pixels). In this case, each chroma component has a rectangular shape. YUV4:2:0 (or YUV420 for short) refers to subsampling the chroma components (U and V) by a factor of 2 in both the horizontal and vertical directions, as shown in Figure 1(c). For example, if the luma component has a size of w x h, where w represents the weight of the luma component and h represents the height of the luma component, then the chroma components of YUV4:2:0 each have a size of w / 2 x h / 2 (e.g., half the width, i.e., w / 2, and half the height, i.e., h / 2, of the luma component in terms of number of samples or pixels). In this case, each chroma component has a square shape.

[0052] It should be noted that YUV4:2:0 chroma subsampling is the most commonly used chroma subsampling method in video coding. Subsampling can generally be the result of the configuration of the image sensor that captures the picture. For example, some sensors provide more pixels for capturing a particular color component. However, subsampling can also be the result of further processing of the captured picture.

[0053] The above YUV format is used in this disclosure as an example of luma-chroma components (channels), with each chroma having a different resolution (i.e., different size) as the luma channel. It should also be noted that in general, only one of the two chroma components may be subject to subsampling, while the other maintains its size. Furthermore, instead of subsampling the chroma components only horizontally as shown in FIG. 1(b), the chroma component(s) may instead be subsampled vertically.

[0054] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems loosely inspired by the biological neural networks that make up animal brains. Such systems "learn" to perform tasks by considering examples and are not generally programmed with task-specific rules. For example, in image recognition, such a system may learn to identify images that contain cats by analyzing example images that have been manually labeled as "cat" or "not cat" and using the results to identify cats in other images. Such systems do this without prior knowledge of cats, e.g., that cats have fur, tails, whiskers, and cat-like faces. Instead, such systems automatically generate discriminative characteristics from the examples they process.

[0055] ANNs are based on a collection of connected units or nodes called artificial neurons that loosely model neurons in a biological brain. Each connection can send signals to other neurons, similar to synapses in a biological brain. An artificial neuron that receives a signal can then process the signal and signal to neurons connected to it.

[0056] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layer multiple times.

[0057] The original goal of ANN methods was to solve problems in the same way that the human brain does. Over time, attention shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a variety of tasks including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even activities traditionally considered to be exclusively human, such as painting.

[0058] Convolutional Neural Networks The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution, which is a special kind of linear operation. Convolutional networks are simply neural networks that use convolution instead of the common matrix multiplication in at least one of their layers.

[0059] FIG. 2A shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer and an output layer, as well as multiple hidden layers. The input layer is the layer to which the input (such as a portion of an image as shown in FIG. 2A) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with multiplication or other dot products. The result of the layer is one or more feature maps (f.maps in FIG. 2A), sometimes called channels. There may be subsampling involved in some or all of the layers. As a result, the feature maps may be smaller, as shown in FIG. 2A. The activation function in a CNN is typically a RELU (Regulated Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, whose inputs and outputs are masked by the activation function and the final convolution, and are therefore called hidden layers. Although the layers are colloquially called convolutions, this is only by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index into the matrix in that it affects how the weight is determined at a particular index point.

[0060] When programming a CNN to process images, the input is a tensor with shape (number of images) x (image width) x (image height) x (image depth), as shown in Figure 2A. Then, after passing through a convolutional layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). A convolutional layer in a neural network must have the following attributes: A convolution kernel defined by width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) must be equal to the number of input feature map channels (depth).

[0061] In the past, traditional multi-layer perceptron (MLP) models have been used for image recognition. MLPs are also known as fully connected neural networks (FCNs). However, due to the full connections between nodes, MLP models suffer from high dimensionality and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be conveniently handled efficiently at scale with full connections. Thus, MLPs / FCNs are often prone to overfitting the data. Since each node (perceptron) in a layer takes as input the output of all nodes in the adjacent layer, the network architecture of FCNs does not take into account the spatial structure of the data (e.g., across the nodes of a layer) and treats input pixels that are far apart the same as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Thus, for purposes such as image recognition, where spatially local input patterns predominate, the full connectivity of neurons is wasteful.

[0062] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of CNNs. The parameters of the layer consist of a set of learnable filters (kernels, as shown above) that have small receptive fields but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns filters that activate when it detects some particular type of feature at some spatial location in the input.

[0063] Stacking the activation maps of all filters along the depth dimension forms the full output volume of the convolutional layer. Thus, every entry in the output volume can also be interpreted as the output of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation are synonymous. It is called an activation map in some papers because it is a mapping that corresponds to the activation of different parts of the image, and also a feature map because it is a mapping of where certain types of features can be found in the image. High activation means that a particular feature has been found.

[0064] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0065] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers progressively reduce the spatial size of the representation, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus also helping to control overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0066] A pooling layer operates independently on each depth slice of the input and spatially resizes it. The most common form is a pooling layer with a filter of size 2x2, applied with a step size of 2 downsamples per depth slice of the input by 2 along both width and height, discarding 75% of the activations. In this case, all max operations are over 4 numbers. The depth dimension remains unchanged.

[0067] In addition to max pooling, the pooling unit can also use other functions such as average pooling and L2 norm pooling. Average pooling has been used often historically, but has recently fallen out of favor in comparison to max pooling, which works better in practice. There is a recent trend to use smaller filters or to discard the pooling layer entirely for aggressive reduction in the size of the representation. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling, where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0068] The ReLU in the above is short for rectified linear unit and applies a non-saturating activation function. ReLU effectively removes negative values ​​from the activation map by setting them to zero. ReLU increases the non-linearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions, such as the saturated hyperbolic tangent and sigmoid functions, are also used to increase the non-linearity. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly compromising generalization accuracy.

[0069] After several convolutional and max-pooling layers, higher-level inference in neural networks is performed via fully connected layers, whose neurons are connected to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as affine transformations, matrix multiplications followed by bias offsets (vector addition of a learned or fixed bias term).

[0070] A "loss layer" specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions may be used that are suited to different tasks: softmax loss is used to predict a single class of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0071] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram is shown in Figure 2B. The goal of an autoencoder is to learn a representation (encoding) of a dataset, typically for dimensionality reduction, by training the network to ignore signal "noise". Along with the reduction side, a reconstruction side is trained, and the autoencoder tries to generate a representation from the reduced encoding that is as close as possible to its original input, hence its name. In the simplest case, given one hidden layer, the encoder stage of an autoencoder takes an input x and maps it to h, h = σ(Wx+b) It is.

[0072] This image h is usually called the code, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are usually initialized randomly and then updated iteratively during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x: x'=σ'(W ’ h'+b') where σ', W', and b' of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0073] Variational autoencoder models make strong assumptions about the distribution of the latent variables. They use variational methods for learning latent representations, resulting in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is then fed into a directed graphical model p θ (x|h), and the encoder estimates the posterior distribution p θ Approximation q for (x|h) φ Suppose we are learning (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder. The objective of a VAE has the following form:

number

[0074] In the formula, D KL stands for the Kullback-Leibler divergence. The prior distribution for the latent variables is usually a centrally isotropic multivariate Gaussian distribution p θ We set (h) = N(0,Ι). In general, the shapes of the variational and likelihood distributions are chosen such that they are factored Gaussians: q φ (h|x)=N(ρ(x),ω 2(x)I) p φ (x|h)=N(μ(h),σ 2 (h)I) where ρ(x) and ω 2 (x) is the encoder output, μ(h) and σ 2 (h) is the decoder output.

[0075] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have enabled researchers' interest in applying neural network-based techniques to the task of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed. Correspondingly, data compression is considered a fundamental and well-studied problem in engineering and is generally formulated with the objective of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all real codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors. In this context, known as the lossy compression problem, we must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over channels with limited capacity, demand different rate-distortion tradeoffs. Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space is intractable. For this reason, most existing image compression methods work by linearly transforming a data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a reversible entropy code. This scheme is called transform coding due to the central role of the transform. For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method, transform, quantization, and entropy code, are optimized separately (often by manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use the transformed representation to code the residual signal after prediction.Several transforms are used for that purpose, such as the discrete cosine and discrete sine transforms (DCT, DST) and the low frequency non-separable hand-optimized transform (LFNST).

[0076] Of the above types of networks, CNNs have been shown to be advantageous for video / image (de)coding and are implemented in variational autoencoders (VAEs) as discussed by J. Balle in "Variational image compression with a scale hyperior" (arXiv preprint arXiv:1802.01436 (2018)). In the following, this paper will be referred to as "J. Balle".

[0077] The VAE structure is designed for deep learning based image compression and constitutes a very efficient structure. The color format of the image input to the VAE is usually RGB without subsampling, so the three components / channels have the same size (width and height). Hence, RGB color format images provided as input to the VAE structure can be directly processed and reconstructed.

[0078] J. Balle describes an end-to-end trainable model for image compression based on a variational autoencoder (VAE). The model incorporates a hyper-prior to effectively capture spatial dependencies in the latent representation. This hyper-prior relates to side information that is also transmitted to the decoding side, a concept common to virtually all modern image codecs, but rarely explored in image compression using ANNs. Unlike existing autoencoder compression methods, the model trains a complex prior together with the underlying autoencoder.

[0079] The VAE framework can be regarded as a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in Figure 2C, which shows the VAE framework.

[0080] Figure 2C shows the network architecture including the hyper-prior model. a ,g s ) shows the image autoencoder architecture (VAE), and on the right (h a ,h s ) corresponds to an autoencoder that implements a hyperprior. The factorized prior model is then applied to the analytic and synthetic transformation g a and g s The same architecture is used for both. Q stands for quantization, and AE and AD stand for arithmetic encoder and arithmetic decoder, respectively.

[0081] The encoder computes g for the input image x. a , resulting in responses y (latent representations) with spatially varying standard deviations. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function. a (1) maps an image x to a latent representation y via a function y = f(x). The latent representation is then converted to discrete values

number

[0082] The response is h a to summarize the distribution of standard deviations in z. z is then quantized, compressed, and transmitted as side information. Entropy model or hyperencoder / decoder h a (also known as the hyperprior) (3) is used to obtain the minimum achievable rate for lossless entropy source coding.

number

number

number

number

number

number

number

[0083] The decoder side of the VAE structure shown in FIG. 2C, i.e., g s and h s So, the decoder first extracts the arithmetic decoder from the bitstream.

number

number

number

number

number

number

number

number

number

[0084] Most deep learning based image / video compression systems reduce the dimensionality of a signal before converting it into binary numbers (bits). For example, in the VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has smaller width and height than x. Since y has smaller width and height and thus smaller size, the dimensionality of the signal is reduced. Therefore, it is easier to compress the signal y. The general principle of compression is illustrated in Figure 2C. The encoder g a Output of and decoder g s The input of the latent space

number

[0085] The reduction in the size of the input signal is illustrated in FIG. 2C, which depicts a deep learning based encoder and decoder. As mentioned before, the input image x corresponds to the input data, which is the input of the encoder shown in FIG. 2C. The transformed signal y is represented in the latent space

number

[0086] As is evident from FIG. 2C, the encoding operation corresponds to a reduction in the size of the input signal, while the decoding operation corresponds to the reconstruction of the original size of the image.

[0087] At the input of the VAE structure, one convolutional network layer (conv N×5×5 / 2↓) is used. Note that "N×5×5 / 2↓" here refers to the notation that specifies the layers of a CNN. In this example, a CNN with a 5×5 convolutional layer is defined as having N channels as output. The output is downsampled by a factor of 2 in both dimensions, as indicated by the down arrow. For this convolutional layer, the size of all three components of the input image is the same (e.g., RGB or YUV444). If this were not the case, the convolutional network layer in this example would not take as input images (or picture parts) with different sizes, as is the case for example with YUV422 and YUV420 formats.

[0088] Similarly, as Fig. 2C shows, due to the last deconvolutional network layer (conv 3 × 5 × 5 / 2↑) at the end of the VAE structure, the sizes of the three components of the output image (reconstructed image) are the same.

[0089] Therefore, there are problems when compressing frames in a video sequence using existing VAE structures. For example, the VAE structures may not directly handle video frames in chroma subsampled color formats (e.g., YUV422, YUV420) because the sizes of the three components are not identical. As mentioned above, in some cases, the input image may not be processed by the input convolutional network layer, and as a result, the output image may not have the same size as the input image.

[0090] The MPEG proposal (Ankitesh K. Singh et al. "M55267: [DNNVC] A study of handling YUV420 input format for DNN-based video coding") discusses ways to address some of the above issues for the specific case of YUV420 color format frames in video sequences.

[0091] Figure 3 shows one method proposed by MPEG. At the encoder side, the luma component is downsampled by a factor of 2 in both horizontal and vertical directions using a convolutional network layer conv N×5×5 / 2↓ (i.e., the convolution kernel is 5×5, the output channels are N, and the output is downsampled by 2 in both dimensions). The chroma components are processed by a convolutional network layer conv N×5×5 / 1↓ without downsampling (in terms of CNN notation, this is indicated by " / 1", i.e., division by 1). Thus, the outputs Y, U, and V have the same size and can be processed by a VAE structure. At the decoder side, the luma output Y component is upsampled by a factor of 2 in both horizontal and vertical directions using conv 1×5×5 / 2↑. In contrast, the chroma components are processed by conv 2×5×5 / 1↑ without upsampling. Thus, the final reconstructed image has the same size as the original input YUV420 image.

[0092] As is evident from the above processing, the method of FIG. 3 provides luma and chroma components having the same size by downsampling the luma component. In other words, the components that are most important to the human visual system and carry the most detailed information of the video frame are processed. This can result in important luma information being discarded, which can lead to quality loss.

[0093] However, when the components and / or channels of a frame have different resolutions (i.e., sizes), the VAE structure can no longer compress this frame of the video sequence. In particular, the VAE structure cannot directly process video frames in chroma subsampling color formats (e.g., YUV422, YUV420) because the sizes of the three components are not the same. As mentioned above, the input image (including the picture portion of the input image) cannot be processed by the input convolutional network layer, and the output image does not have the same size as the input image.

[0094] First embodiment Some embodiments of the present disclosure may provide a solution to the above problem of dealing with subsampled chroma components that is suitable for existing ANN or VAE framework structures.

[0095] According to one embodiment of the present disclosure, a method for encoding a picture portion is provided, the picture portion including a luma component and a chroma component having a resolution lower than the resolution of the luma component.

[0096] A picture portion refers to a number of samples of a picture, said number (quantity) being smaller than the total number of samples of said picture. It should be noted that the terms picture and image in this specification have the same meaning. A picture portion may include some (one or more) or all samples of a picture. The terms sample and pixel in this specification have the same meaning. A picture may be a still image or a video picture, for example corresponding to a frame of a video sequence (moving image).

[0097] A picture portion may include one or more color components, e.g., a luma component and one or more chroma components. For example, as described above, a picture portion in luma-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the luma channel for the entire picture. Alternatively, the chroma resolution may be lower than the resolution of the luma channel for a portion of the image. In other words, a portion of the scene captured in the picture may be represented by more luma samples than chroma samples (of the same chrominance). The resolution may be given by the size (dimension) of the two-dimensional (2D) luma or by the size of the 2D chroma channels. This is typically given by the number of samples along the xy direction of the luma samples or chroma samples in the 2D image, or by the total number of samples. The relative resolution may be specified as a ratio between luma samples and chrominance samples, as described above, e.g., 4:2:0, 4:2:2, etc. It should be noted that although some example embodiments are described herein with respect to a luma channel and / or chroma channel(s), the present disclosure is not limited thereto. In general, other channels, such as depth, may be processed as discussed in the present disclosure in addition to or in place of one or more color components.

[0098] The method is shown in Figure 13 and includes a step 1310 of upsampling the chroma components to obtain upsampled chroma components that match the resolution of the luma component. The upsampled chroma components are also called intermediate chroma components. The matching of the resolution may correspond to matching of the dimensions of the luma components. The method further includes a step 1320 of encoding the luma component and the upsampled chroma components into a bitstream.

[0099] Prior to the upsampling step, the resolution of the luma component may be obtained. This may be done by determining the resolution by reading it from a memory or from the bitstream of the input picture, etc. The resolution of the chroma components may be obtained similarly. The resolution of each picture part including the luma and chroma channels may be indicated with the respective channels in a memory, or in general in any kind of storage medium. The resolution of each channel may be provided as a number of samples. Alternatively or additionally, the channel resolution may be provided as a ratio to the size of the picture (which may be indicated in the bitstream or obtained in another manner), i.e., all samples of said picture.

[0100] The encoding above may be performed by a variational autoencoder VAE. VAEs are known in the art for efficiently performing tasks related to picture encoding and decoding, and have a good compromise between the required bit rate and the quality of the reconstructed image / picture. The original subsampled YUV color format may be subject to encoding by the VAE by upsampling the subsampled chroma components (i.e., chroma having a lower resolution than the luma channel) such that the resized chroma has the same resolution as the luma. The encoder portion of the VAE shown in FIG. 2C may then process the respective equal-sized three-channel inputs in the manner as previously described with reference to FIG. 2C. In the process, the luma channel remains unaffected, and thus potentially important image information such as that provided by the luma may be preserved.

[0101] However, it should be noted that encoding by a VAE is only one of the possible advantageous implementations. This is not a limitation of the present disclosure, which may also work efficiently with other types of autoencoders, such as those having a structure as shown in FIG. 2B. Furthermore, the present disclosure is not limited to autoencoders, but may be applied to any ANN-based or generally any trained network. In principle, the present disclosure is applicable to any type of encoder or decoder.

[0102] In one exemplary implementation of the method, the upsampling of the chroma components is based on the samples of the luma components, which means that the upsampling is based on the values ​​of the luma samples and is therefore content-adaptive. A luma sample may refer to one or more luma samples. This includes using all luma samples for upsampling.

[0103] The chroma upsampling may further include processing the chroma components through a trained network. The size equalization may include inputting the luma and chroma components to a trained network and outputting the upsampled chroma components from the trained network. The use of the luma component as an input to the trained network may preserve inter-component information between the luma and chroma channels.

[0104] The trained network on the encoding side may be any kind of network suitable for upsampling. This includes, for example, artificial neural networks (ANNs), such as fully connected networks, sparse neural networks, and convolutional neural networks (CNNs). In particular, CNNs have been successfully applied to image / picture processing tasks for picture encoding and decoding. The trained network used for chroma upsampling may also be called a size equalization network, since it provides as output resized chroma components (i.e., intermediate chroma components) with a resolution that matches the resolution of the luma channel. In other words, after processing by the size equalization network, all output luma-chroma channels have the same resolution (i.e., size). However, the present disclosure is not limited to such upsampling. As will be shown later, there are several possible advantageous implementations of upsampling. The term trained refers to a type of network that is pre-trained by machine learning. Such a network may also be a trainable network with inputs (supervised or unsupervised) for machine learning. However, the trained network may also be provided by configuring weights without training the same network, for example by retrieving the weights from a storage medium. The weights may be pre-obtained and stored by training a similar network. Alternatively, the weights (also called coefficients) may be derived by use of a mathematical filter such as a Wiener filter or a Kalman filter. Another option is to derive the weights by use of a transform layer, for example a discrete cosine transform (DCT) or a discrete sine transform (DST). Although a transform layer may be included in the trained network, each weight may be a fixed coefficient and may be considered a parameter.It should be noted that DCT and DST are just examples and in general any suitable known transform may be applied, such as, for example, the Hadamard transform, the Fast Fourier Transform (FFT), the Discrete Fourier Transform (DFT), or a transform obtained by modifying the above transforms to comply with some implementation constraints (such as number representation or bit depth), or an orthogonal unitary transform, such as the Karhunen-Loeve transform.

[0105] As discussed further below, upsampling, which may be performed in the first layer (or elsewhere) of the trained network, may use a bilinear filter or another type of linear or nonlinear interpolation filter instead of a nearest-neighbor based filtering method. Nearest-neighbor filtering may be considered a simplified version of an interpolation filter. Additionally or alternatively, other super-resolution methods may be used.

[0106] As an example, on the encoder side, the chroma components of a video frame in a sub-sampled color format, such as YUV420, YUV422, etc., are equalized to the size of the luma component using a size equalization network layer.

[0107] In one exemplary implementation, the trained network comprises at least two processing layers, including a first layer for upsampling based on input chroma components and a second layer for processing with the luma component of the output of the first layer. The terms "first" and "second" here are merely labels and do not necessarily correspond to the first and second layers of the network in the layer order. There may be some other layer or layers between the first and second layers.

[0108] In one implementation example, processing the output of the first layer together with the luma component includes element-wise addition of the luma component and the upsampled chroma components. Thus, chroma upsampling using luma as side information can be performed efficiently since the "addition operation" has low complexity. Processing based on the "combination" of the luma component and the chroma component may not be limited to addition operations. In this example, inter-component information is preserved. Here, the term preserved means that the luma information, including high-frequency information and details regarding sharp edges, is used in the chroma upsampling operation. Compared to the prior art, the chroma components are thus deeply enhanced.

[0109] Alternatively, the chroma may be concatenated with the luma. This typically results in a concatenated luma-chroma composite that has a size larger than each of the luma and chroma components. Other types of "combination" type operations may be used instead and / or combined with addition or concatenation, whereby the luma information is utilized for upsampling and / or processing of the subsampled chroma by a size equalization network.

[0110] The above process can be illustrated by Figs. 4 to 8, which show several embodiments of the present disclosure, covering both the encoder side and the decoder side.

[0111] The encoder side shown at the top of FIG. 4 includes inputs of luma component Y and chroma components U and V. In particular, the chroma components U and V are input to a size equalization network layer 410 that upconverts the chroma component(s) to the size of the luma component (compared to the subsampled, luma component Y). The term "upconversion" may also be referred to as upsampling. As mentioned above, the upconversion or upsampling may also be performed using the luma component as input information (i.e., side information) for the size equalization network, which is illustrated in FIG. 4 by a third input to the NN size equalization module 410. The output of the NN size equalization module 410 is intermediate chroma components U and V that are dimension-matched with the luma component Y. Both the luma component and the intermediate chroma components are input to a VAE encoder 420. The VAE encoder may be an encoder such as that shown in FIG. 2B or FIG. 2C. The VAE encoder 420 outputs, for example, a compressed bitstream in which the intermediate chroma components U, V and the luma component Y are included.

[0112] Correspondingly, the bottom part of FIG. 4 shows an exemplary decoder side. In particular, an encoded bitstream (such as the bitstream encoded by the VAE 420) is input to the VAE decoder 470. The VAE decoder 470 may be a decoder such as that shown in FIG. 2B or FIG. 2C. The VAE decoder 470 outputs a luma component Y and intermediate chroma components U and V. These decoded luma component Y and intermediate chroma components U and V are not necessarily the same as the luma and intermediate chroma components on the encoder side. This is because, in some embodiments, the VAE encoder 420 may compress the input in a lossy manner. The decoded components Y, U, V, which have the same resolution, are then input to the downsampling network 480, which performs a NN size conversion back to the input size (resolution). The intermediate chroma components U and V are downsampled to the dimensions of the chroma components. In the downsampling process, the luma component Y may be used.

[0113] 5 shows an example of the implementation of the present disclosure for the encoder-side size equalization network 410. In this example, each layer belongs to a convolutional neural network (CNN). Referring to FIG. 5, the convolutional network layers are defined as follows: Conv N×K×K: represents a convolutional network layer with K×K convolution kernels, where K is a positive integer. A typical number for K is 3 or 5. However, K is not limited to these numbers and may be any other positive integer. In particular, larger values ​​of K are possible. N is a positive integer and represents the number of output channels of the convolutional neural network. A typical number for N is 128 or 192. However, any other number may be used depending on the output needs. In particular, N may be 1 or 2, i.e. conv 1×K×K or conv 2×K×K, corresponding to a CNN layer with one or two output channels. Act.: Refers to the activation layer, which can be a nonlinear activation function, for example, the rectified linear unit (ReLU) or GDN mentioned above. The function is used to activate the convolution layer. Without activation, the convolution layer becomes just a linear combination, which is the same as without the convolution operation. Conv N×K×K / 2↑: represents the same as Conv N×K×K, but (in this example) additionally upsamples (upconverts) the output by a factor of 2 both horizontally and vertically. This is indicated by the "2" and the up arrow. Conv. N×K×K, Act: Represents a convolutional layer + activation layer.

[0114] Figure 5 shows an example on the encoder side for the color format YUV420, which means that three channels Y, U and V are involved, so N=3.

[0115] For example, if the luma size is w×h, the chroma size is w / 2×h / 2, and the color format is YUV420, the size equalization network layer upsamples the chroma components U and V by a factor of 2 in both horizontal and vertical directions, where w represents the width and h represents the height of the luma component, e.g., counted in (integer) samples. The conv N×K×K / 2↑ layer upsamples the U and V components by a factor of 2 each in both horizontal and vertical directions. The output of this layer has N channels.

[0116] As is clear from Figure 5, the Y component is processed in a conv N×K×K layer without upsampling, thus preserving the resolution (dimension) of the luma component. The output of this layer also has N channels as a size equalization layer conv N×K×K / 2↑. The output signals of both layers conv N×K×K / 2↑ and conv N×K×K are processed by respective activation layers (i.e. nonlinear activation functions, e.g. ReLU or GDN) that output feature maps, which are then added (summed). In this example, a simple sum of signals is used. This means that the respective outputs are added element-wise (the elements of the feature maps are summed). This is possible because the size (dimension) of the outputs from the conv (convolution) layers is the same in terms of width and height.

[0117] Other types of operations based on merging chroma and luma (combination) are possible. For example, concatenation of luma and chroma outputs is suitable. Addition can be one of the simplest solutions with only N channels in the resulting frame. This means that the sum operation preserves the number of channels. In contrast, concatenation introduces a latent representation with 2N channels. Thus, the sum operation provides low complexity for upsampling chroma together with luma without further increase in the number of channels.

[0118] The combined signal is then processed by one or more convolutional and activation layers to provide two chroma channels as output (resized U, V corresponding to the upsampled chroma components). The original Y signal is also provided to the next step, which is encoding 420 into a bitstream. The upsampled chroma is also called the intermediate chroma component / channel. These equal-sized outputs (signals) are then used as inputs to the VAE structure.

[0119] In the example of Fig. 5, the summation operation of Y and the upsampled chroma U and V is placed after the first layer. However, said operation (or, for that matter, any other suitable operation for concatenating and / or merging signals) may be placed after any other layer. The advantage of this design is that the Y features are mostly extracted by the convolution layer placed after the summation operation.

[0120] In another example, if the luma size is w×h, the chroma size is w / 2×h, and the color format is YUV422, the size equalization network layer upsamples the chroma components U and V horizontally by a factor of 2. In this case, the upsampled chroma components have a rectangular shape. In either case, the upsampled chroma components U and V output from the size equalization network layer have the same size as the original Y component. Thus, the output of the size equalization network layer can be used as the input of a VAE structure.

[0121] In one example implementation, the upsampling involves inserting an additional line of samples after every n lines of samples of the chroma components, where line refers to row and / or column, and n is an integer equal to or greater than 1. This means, for example, that every odd row is an added row. Additionally or alternatively, for example, every odd column of the resulting upsampled components is an added column. Instead of odd, every even row and / or column may be added. These examples are for n=1, where an additional row is inserted after (or before) every other row. Such upsampling is a 2x upsampling. However, the present disclosure is not limited to this particular example. Rather, n may be larger, which means that an additional row of samples is inserted after an integer number of rows. A similar approach is applicable to columns. n may be different for rows and columns. Furthermore, this allows further adaptation of chroma upsampling by inserting line samples at specific rows and / or columns.

[0122] Furthermore, upsampling may be performed by inserting a line of samples with value 0 after every n lines of samples of the chroma components. This may be called zero-filling. This is a simple scheme to perform upsampling by simply setting each line sample to 0. Thus, no further signaling or processing overhead is required. Such an approach is illustrated in FIG. 9, which shows 4×4 samples of upsampled chroma. The solid circles refer to the original chroma samples. The open circles refer to samples with 0 values. In this example, a line of zero-value samples is inserted after each row and each column of the chroma components. Furthermore, in this example, the size equalization network is simplified to a one-layer network (M=1) with no further layers after the “add” operation of FIG. 5. In FIG. 9 (encoder side), the upsampling of U and V is simplified as filling each row and each column with 0 for the case of YUV420 color format. In the case of YUV422 format, the filling of zeros is for each column). In this case, the network is not a convolutional neural network, but can be understood as one layer of a fully connected neural network (FCNN). Therefore, this network is not necessarily trainable by itself. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network.

[0123] In another implementation, chroma upsampling may be performed by inserting a line of samples having a value equal to the respective nearest neighbor sample in the chroma component after every n lines of samples of the chroma component. The value insertion may be performed, for example, by copying the chroma value to the respective nearest neighbor sample. As in the case of zero filling above, this allows further adaptation of the chroma upsampling by inserting line samples at specific rows and / or columns. Furthermore, using the value of the inserted line sample with reference to the nearest neighbor sample of the chroma, the accuracy of the chroma upsampling is more accurate because the neighbor information (reflected by the neighbor chroma sample value) with reference to the original chroma is used.

[0124] This is illustrated in FIG. 11, which shows 4×4 samples of upsampled chroma. Again, the black circles refer to samples of the original chroma. Here, the white circles refer to samples whose values ​​depend on their respective chroma neighbors. In FIG. 11, the insertion of chroma values ​​into the respective neighbors is indicated by arrows. In this example, a sample value is inserted after every one row and every one column of the chroma components. Furthermore, in this example, the size equalization network is simplified to a one-layer network (M=1) with no further layers after the “add” operation in FIG. 5, and the inter-component information from luma to chroma is preserved by the “add” operation. For example, if luma contains more information (because it has a higher original resolution), the additional information is added to the chroma samples in the correct position when luma and chroma have the same dimensions. In FIG. 11 (encoder side), the upsampling of U and V is simplified as the upsampling of the nearest neighbor samples in each row and each column, as shown for the YUV420 format in FIG. 11. In the case of YUV422 format, the upsampling of the nearest sample is for each column. In the above example of chroma upsampling, the network is not a convolutional neural network, but can be understood as one layer of a fully connected network. Therefore, the network itself is not trainable, but can be trained together with a VAE structure for better coding performance. Thus, each FCNN may also refer to a trained network. Nearest neighbor upsampling here means that the value of the nearest neighbor sample (in a predefined direction) is copied to the newly added sample. In FIG. 11, the nearest neighbors from the left and above are taken. This is a rule that does not limit the present disclosure. The nearest neighbor may also be copied from the right and below. Interpolation (e.g., the average of adjacent sample values) or other types of upsampling may be involved.

[0125] The above upsampling network is just an example of a trained network for size equalization with one layer. It is clear to those skilled in the art that the size equalization layer may have two or more layers, as shown in Figures 5 and 7. In this case, the size equalization network may be trained to perform chroma upsampling.

[0126] In the preceding discussion, the luma component was used as side information and input to the size equalization layer to perform upsampling of the subsampled chroma channels. In a further implementation of the size equalization network as shown in FIG. 7, the upsampling may be performed without further use of luma. In other words, the luma channel is not input to the size equalization network on the encoder side. Otherwise, the processing of the subsampled chroma by the encoder is the same as already discussed for the case where luma is used as side information for chroma upsampling. This allows for a simplified design of the size equalization network, since it takes only the chroma component(s) as input.

[0127] According to an embodiment of the present disclosure, a method is provided for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component. The picture portion refers to a number of samples of an image and / or picture, said number being smaller than the total number of samples of said image / picture. The picture portion may also include all samples of an image or picture. The term sample may also refer to pixels. An image may be a still picture or a video picture. The picture portion may include one or more chroma components. For example, a picture portion of a luma-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the luma channel for the entire picture / image. Alternatively, the chroma resolution may be lower than the resolution of the luma channel for a part of the image. In this "partial" luma resolution case, the luma channel and the chroma channel(s) are associated with the same part of the picture. The same portion of a picture may also be referred to as a region of the picture.

[0128] The method is illustrated in FIG. 14 and includes a step 1410 of decoding a luma component and an intermediate chroma component that matches the resolution of the chroma components from the bitstream. The intermediate chroma components are also called upsampled chroma components, reflecting that said intermediate components are generated / obtained by upsampling the chroma components. The resolution is given in terms of the size (i.e., dimension) of a two-dimensional (2D) luma or chroma channel. This is typically given by the number of samples along the xy direction of the 2D luma samples or the 2D chroma samples, or by the total number of samples. The method further includes a step 1420 of downsampling the intermediate chroma components to obtain the chroma components of said picture portion. The above decoding may be performed by a variational autoencoder (VAE). VAEs are known in the art for performing tasks related to image / picture encoding / decoding very efficiently, with a good compromise between the required bit rate and the quality of the reconstructed picture. The encoder portion of the VAE shown in Figure 2C may then process the respective equal-sized channel inputs, including luma and intermediate chroma, in the manner described above with reference to Figure 2C. In the process, the luma channel remains unaffected, thus preserving the most important image information provided by the luma when decoding the luma component from the bitstream.

[0129] By downsampling the intermediate chroma components (i.e., chroma having the same resolution as the luma channel), the original chroma components (i.e., subsampled chroma) may be reconstructed. In particular, picture parts having the original luma-chroma format may be reconstructed with high quality. Thus, the decoder may provide as output picture parts in a chroma format, where the chroma has a lower resolution (i.e., size) than the luma.

[0130] In one exemplary implementation of the method, the downsampling further includes processing the intermediate chroma components by a trained network, which may be referred to as a size conversion network, reflecting its purpose of converting the input intermediate chroma (having the same size as the luma) to the size of the original chroma, which has a lower resolution (i.e., smaller size) than the luma.

[0131] The trained network on the decoding side may be any kind of network suitable for downsampling. This includes, for example, artificial neural networks (ANNs), such as neural networks, convolutional neural networks (CNNs), etc. In particular, CNNs have been successfully applied to image / picture processing tasks for picture encoding and decoding. The trained network used for chroma downsampling may also be called a size conversion network, since it provides as output resized chroma components (i.e., the original subsampled chroma components) that have a lower resolution than the luma channel. In other words, after processing by the size conversion network, all output luma-chroma channels have different resolutions (i.e., sizes).

[0132] In a first exemplary implementation of the method, the step of processing by the trained network includes inputting the intermediate chroma components to the trained network and outputting the chroma components from the trained network. This means that the trained network uses only the intermediate chroma as an input for downsampling the intermediate chroma. Thus, in this example, no side information regarding luma is used by the resizing network layer. Thus, the trained network may be designed with low complexity and provides low processing overhead because luma is not used as side information.

[0133] This is illustrated in FIG. 8, which shows that the luma component may not be used as an input of the size conversion network at the decoder. In this example, the two chroma components U and V refer to the color format YUV444, where U and V have the same size as luma Y. Otherwise, the processing of intermediate chroma by the decoder is the same as that already discussed with reference to FIG. 4. In an exemplary implementation of the trained network, the trained network comprises at least two processing layers. Of these at least two layers, a first layer performs processing of the input intermediate chroma components, and a second layer of the at least two processing layers performs downsampling of the output of said first layer.

[0134] In the example of Figure 8, the size conversion network on the decoder side consists of multiple convolutional and activation layers Conv. N×K×K, Act. The first Conv. N×K×K, Act on the left takes the intermediate chroma components U and V as input. The last layer of the network (Conv. 2×K×K / 2↓) downsamples the signal by a factor of 2 in both horizontal and vertical directions and outputs two channels. These two channels are used as the reconstructed U and V signals in YUV420 format. When all intermediate Conv. N×K×K, Act. layers are removed, the last Conv. 2×K×K / 2↓ points to the second layer.

[0135] In this example, the Y information is not downsampled compared to the approach of J. Balle. Thus, the luma and spatial information of Y is mostly preserved. As mentioned above, the chroma downsampling can be done without utilizing the luma information as side information.

[0136] In another example implementation, the downsampling of the intermediate chroma components is based on the samples of the luma components. This means that the downsampling is not based on the size, but rather on the content of the luma channel, i.e., the respective values ​​of the luma samples. A luma sample may refer to one or more luma samples. This includes using all luma samples for upsampling.

[0137] When chroma is processed by the trained network, the step of processing by the trained network includes inputting the luma component to the trained network. In one example implementation, the first layer of the trained network performs a step of processing the intermediate chroma components together with the luma component. In other words, the luma is used as side information for the chroma downsampling process. This preserves the inter-component information between the luma channel and the chroma channel. In particular, the luma component may contain more detailed information than the subsampled (original) chroma components. By using the luma as side information, some details of the chroma may be restored based on the luma component. The upsampling network may be specifically trained for such upsampling.

[0138] Note that on the decoder side, the first processing layer of the size conversion network may not need to perform an "addition operation" in contrast to the encoder side, and such a type of operation may be performed as shown in FIG. 5 (at least some kind of explicit combining operation may be used for the size equalization network in the encoder). This is because the input to the first layer of the size conversion network of the decoder, including the luma Y and the intermediate chroma(s) U, V, already has the same size as shown in FIG. 6. Rather, the three equal-sized channels may directly input to said first CNN layer without requiring a separate addition operation. This is not the case for the input to the first layer of the size equalization network of the encoder shown in FIG. 5. In this case, the chroma is first upsampled, followed by adding the upsampled chroma and luma. This is then fed to the CNN layer.

[0139] FIG. 6 shows an example implementation of the decoding side of the VAE structure, which takes as its input the output of the encoding side of the VAE structure described above. As mentioned above, here each input to the size conversion network has three components of the same size (i.e., YUV444). The first channel is used directly as the reconstructed luma component and has the same size as the original luma component in the YUV420 format. Conv. 2×K×K / 2↓ represents the same as Conv N×K×K, but further downsamples (downconverts) the output by a factor of 2 in both the horizontal and vertical directions. This is indicated by the “2” and the down arrow. As mentioned above, such an arrangement can be advantageous to preserve inter-component information, for example to enhance chroma with additional details from luma, which originally had a higher resolution (dimension) than chroma.

[0140] In general, a trained network using luma at the decoder side of the VAE structure may be constructed in the same manner as without luma, except for considering an additional luma as an input. This is illustrated in Figure 6, where the size conversion network is again designed with multiple convolutional and activation layers Conv. N×K×K, Act. Again, the last layer of the network (Conv. 2×K×K / 2↓) downsamples the signal by a factor of 2 in both horizontal and vertical directions and outputs two channels. These two channels are used as the reconstructed U and V signals in YUV420 format.

[0141] For example, if the color format is YUV420, the resizing network layer downsamples the intermediate chroma components U and V by a factor of 2 in both the horizontal and vertical directions. In another example, if the color format is YUV422, the resizing network layer downsamples the intermediate chroma components U and V by a factor of 2 in the horizontal direction. In either case, the final output has the same size as the original components of the input frame with color format YUV420 or YUV422, respectively.

[0142] In one exemplary implementation of the method, downsampling includes removing a line of samples after every n lines of samples of the intermediate chroma components, where a line is a row and / or column and n is an integer equal to or greater than 1. This means, for example, that after every one (n=1) or two (n=2) rows and / or columns, said line sample may be removed or deleted. Thus, downsampling of the intermediate chroma may be performed quickly by simply removing the respective line sample. Alternatively, downsampling may be performed, for example, by averaging (possibly weighted) the sample values ​​nearest to each chroma sample.

[0143] Examples of line sample removal are shown in Figures 10 and 12. Note that the sample configurations in Figures 10 and 12 correspond to the sample configurations in Figures 9 and 11, respectively, as a result of the upsampling process of the chroma components at the encoder side.

[0144] Specifically, Figures 10 and 12 show 4x4 samples of intermediate chroma. The black circles refer to samples of the original chroma. The open circles with crosses refer to samples that may be non-zero as a result of the chroma upsampling process by the size equalization network on the encoding side (with and without using luma as side information) and / or the chroma processing through the first layer(s) of the size conversion network on the decoding side.

[0145] In the examples shown in Fig. 10 and Fig. 12, the size conversion network is simplified as a one-layer network (M=1). In Fig. 10 and Fig. 12 (decoder side), the downsampling of U and V is simplified to discard odd row and column samples in the case of YUV420 color format. In YUV422, odd column samples are removed. In either of these cases shown in Fig. 10 and Fig. 12, the respective networks are not convolutional neural networks, but can be understood as one layer of a fully connected neural network (FCNN). Therefore, this network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network.

[0146] However, this is just an example of a trained network for size conversion with one layer. It is clear to those skilled in the art that the size conversion layer may have two or more layers, as shown throughout Figures 6 and 8. In this case, the size conversion network may be trained to perform chroma upsampling.

[0147] It should be noted that the encoder-side size equalization network shown in Fig. 5 and the decoder-side size conversion network shown in Fig. 6 can be trained together using a VAE structure. This can further improve the encoding-decoding process, especially for subsampled chroma components, within the VAE framework, including, for example, the encoding-decoding performance and / or the quality of the reconstructed picture / image.

[0148] Alternatively, they may be trained separately and used as pre-trained VAE networks. In either case, the respective networks refer to the trained networks. Note that (sampling factor) up / down sampling by a factor of 2 is a non-limiting example, but is one of the typical values ​​used. In general, the sampling factor may be any integer equal to or greater than 1. The sampling factor may alternatively be a fractional sampling factor given as a ratio, e.g., p / q, of two positive integers p and q.

[0149] To summarize this part of the embodiment, a size equalization network (for upsampling) and a size conversion network (for downsampling) are used to adjust the size of the subsampled chroma channels so that the three components of YUV420 have the same size. These can then be handled / processed by the existing VAE structure.

[0150] Moreover, as previously explained, the present disclosure also provides devices configured to perform the steps of the above methods.

[0151] 15 shows an encoder 1500 for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component. The encoder comprises an upsampling module 1510 configured to upsample the chroma components to obtain upsampled chroma components that match the resolution of the luma component, and an encoding module 1520 configured to encode the luma component and the upsampled chroma components into a bitstream.

[0152] In one implementation, the encoder may include one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the encoder to perform the above method.

[0153] 16 shows a decoder 1600 for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component. The decoder comprises a decoding module 1610 configured to decode the luma component and an intermediate chroma component that matches the resolution of the chroma component from the bitstream, and a downsampling module 1620 configured to downsample the intermediate chroma components to obtain the chroma components of the picture portion.

[0154] In particular, the decoder may comprise one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the decoder to perform the above-described methods.

[0155] A computer program stored on a non-transitory medium comprising code that, when executed on one or more processors, performs the steps of any of the above methods, according to one embodiment of the present disclosure.

[0156] Some embodiments of the present disclosure may provide the following advantages, which are summarized as follows: Luma sample information is fully preserved. Luma spatial information is fully preserved. The networks (i.e., trained networks) used for size equalization and size conversion (both trained networks may be CNNs) may be trained together with the VAE network structure. This may provide the advantage of further improving the results. Flexibility in handling various other chroma subsampling formats through the VAE network structure, such as YUV422. Inter-component information can be preserved.

[0157] Second embodiment The second embodiment may be combined with any of the example implementations of the first embodiment. As mentioned above, the first embodiment is directed to handling subsampled chroma components compared to the luma components. Upsampling of the chroma components makes such resized chroma components along with the luma components suitable for further processing within a trainable network-based framework such as an ANN or a VAE. Thereby, the luma components may remain largely unaffected, and thus the sample information and spatial information of the luma may be preserved. This may be desirable for many applications where high quality of the reconstructed picture (or parts thereof) is required. This means that a large resolution needs to be maintained, and therefore the encoding-decoding process must not tamper with the luma.

[0158] However, in other applications, high resolution is not necessarily the most important goal, but rather making the encoding-decoding process faster. Furthermore, the desired goal may be a low bit rate resulting in a lower resolution, such as that provided by luma downsampling. Thus, it may be desirable to choose between encoding-decoding processes that provide either low-resolution or high-resolution pictures.

[0159] This problem may be solved by a second embodiment, which is directed to adaptive switching between luma downsampling and chroma upsampling when adapting the resolution of two or more input channels to match. Luma downsampling and chroma upsampling may also be referred to as sampling modes. Which of the two sampling modes may be performed is determined at the encoder side, signaled to the decoder side, and / or determined by the decoder side based on the luma and chroma channels.

[0160] In particular, as further discussed below with reference to various example implementations and aspects of the present disclosure, luma downsampling that matches the size (resolution) of the chroma and chroma upsampling that matches the size (resolution) of the luma are adaptively selected by the encoder. Which of the sampling modes is selected by the encoder may be indicated in various ways, as further detailed below, and may include an indication by and / or a determination from the received bitstream. Depending on how one of the sampling modes (i.e., luma downsampling or chroma upsampling) is indicated, the decoder side then performs either luma upsampling or chroma downsampling to generate a reconstructed picture portion having the same resolution as the picture portion input to the encoder (i.e., the same subsampling format as the input frame).

[0161] According to an embodiment of the present disclosure, a method for encoding a picture portion is provided. The picture portion includes a luma component and a chroma component having a resolution lower than that of the luma component. The picture portion refers to a number of samples of an image and / or picture, said number being smaller than the total number of samples of said image / picture. The picture portion may also include all samples of a picture. The term sample may also refer to a pixel. The picture portion may be a still picture or a video picture. The terms picture, image, and picture portion may be used synonymously. The picture portion may include one or more chroma components. For example, a picture portion of a luma-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the luma channel for the entire picture or image. Alternatively, the chroma resolution may be lower than the resolution of the luma channel for a portion of an image. In this "partial" luma resolution, the luma channel and the chroma channel(s) are associated with the same portion of the picture, which may also be referred to as a region of the picture.

[0162] Downsampling luma is beneficial for saving bitrate at low resolutions, and upsampling chroma is beneficial for preserving luma information at large resolutions. Thus, adaptive switching allows the selection of a sampling mode depending on the requirements and / or limitations on bitrate or picture or image quality. In any case of selected picture processing of a picture part, the processed luma-chroma components have a matched resolution, so that the output can be handled within a VAE framework or another framework that allows efficient encoding.

[0163] This method is illustrated in FIG. 31 as a picture processing of the picture part. downsampling the luma component to obtain a downsampled luma component that matches the resolution of the chroma components; upsampling the chroma components to obtain upsampled chroma components that match the resolution of the luma component The method includes a step 3110 of selecting one of the above.

[0164] Resolution is given in terms of the size (i.e., dimensionality) of the two-dimensional (2D) luma channel. However, it should be noted that resolution may also be given by the dimensionality of channels other than luma, such as chroma channels, or other types of channels, such as depth channels. Resolution is typically given by the number of samples along the xy directions of the 2D luma and / or chroma channels, or by the total number of samples in the channels at one instant (total number of samples in the picture).

[0165] Prior to luma downsampling and / or chroma upsampling, the resolution of the luma and / or chroma components may be obtained, for example, from a memory or storage, and the resolution may be stored together with the picture portion to be encoded. A picture portion may be a part of a picture or may be the whole picture. Thus, a picture portion may include one or more samples of luma and one or more samples of chroma. In some embodiments, a picture portion may be one of multiple picture portions forming a picture. These may be obtained by subdividing a picture into picture portions. In general, the picture portions may have the same size or different sizes (dimensions).

[0166] A picture may be a part of a video. The resolution of each channel may be given as a number of samples. Alternatively, a channel resolution may be given as a ratio to the size of a picture, i.e. all samples of said picture.

[0167] The method of Fig. 31 further comprises a step 3120 of processing the picture portion according to the selected picture processing. In other words, the selected picture processing may be one of luma downsampling or chroma upsampling. Furthermore, the method comprises a step 3130 of generating a bitstream according to the selected picture processing, the generating comprising encoding the processed picture portion. The encoding of the picture portion may be performed by an autoencoder, for example by a variational autoencoder VAE, or by another type of autoencoder. VAEs are known in the art for efficiently performing tasks related to encoding and decoding pictures, and have a good compromise between the required bit rate and the quality of the reconstructed picture.

[0168] Picture processing that selects one of luma downsampling and chroma upsampling in either case provides resized luma and chroma components with equal resolution (e.g., equal dimensions). Thus, regardless of the selected picture processing, the equal-sized luma-chroma channels can be handled within a common framework, such as an ANN framework, e.g., a VAE.

[0169] This includes in particular subsampled chroma formats such as YUV422, YUV420, etc. The color formats may also include any other luma-chroma formats in which any of the channels may have different sizes. The encoder portion of the VAE shown in FIG. 2 may then process each equal-sized three-channel input in the manner as previously described with reference to FIG. 2. Furthermore, the selection of picture processing allows adaptively switching between luma downsampling and chroma upsampling. Thus, a low-resolution or high-resolution picture processing may be selected.

[0170] FIG. 17 illustrates the encoder side with an example of YUV420 color format, where the chroma inputs U, V have a lower resolution than the luma component Y. As shown in FIG. 15, on the encoding side, the YUV input undergoes a size equalization process. This includes, on the one hand, a downsampling 1710 of the luma component as shown in the upper branch of FIG. 17. The output of this example is three YUV components with the same size. Here, the same size refers to the size (dimension) of the chroma components U and V. On the other hand, the chroma components are upsampled 1720 via a size equalization process performed in the lower branch of FIG. 17. Again, the output is three YUV components with the same size. Here, the same size refers to the luma component. After encoding the size-equalized YUV of the upper and lower branches, for example by VAE, a cost in terms of a cost function is determined. This may be based, for example, on a rate-distortion optimization (RDO) 1730. Which sampling mode (i.e., chroma upsampling or luma downsampling) is ultimately used depends on whether the upper or lower branch process offers the lowest cost. In the example of Figure 17, the lowest cost sampling mode is indicated by a flag (indicator) and is encoded in the bitstream or signaled separately to the decoder. Various means of how to select a sampling mode based on cost are discussed in detail below.

[0171] In RDO, a weighted sum of rate and distortion is optimized. However, the present disclosure is not limited to cost minimization based on rate and distortion. For example, in addition to or instead of rate and / or distortion, complexity may be considered in the cost function. Other cost parameters may be added or used instead of the above ones.

[0172] In one implementation of the method, the upsampling and / or downsampling is performed by a trained network.

[0173] As already mentioned in embodiment 1, the trained network on the encoding side may be any kind of network suitable for downsampling or upsampling. This includes, for example, artificial neural networks (ANNs), such as neural networks, convolutional neural networks (CNNs), etc. The trained network used for luma downsampling and / or chroma upsampling may also be called a size equalization network, since both networks provide as output resized color (luma, chroma) components with a resolution that matches the resolution of the chroma channel (luma downsampling 1710) or that matches the resolution of the luma channel (chroma upsampling 1720). In other words, after processing by the size equalization network, the output Y, U, and V channels have the same resolution (i.e., size).

[0174] Figures 19 and 20 show examples of implementations of the size equalization network on the encoder side for the YUV420 color format. Figure 19 shows a size equalization network (upper branch) with luma downsampling, and Figure 20 shows a size equalization network with upsampling of the two chromas U and V. In this example, each layer belongs to a convolutional neural network (CNN).

[0175] The convolutional network layers shown in Figures 19 and 20 are defined in the same manner as in embodiment 1 (for example, Figures 5 to 8). In summary, they are as follows. Conv N×K×K: represents a convolutional network layer with K×K convolution kernels, where K is a positive integer. Typical values ​​of K are 3 or 5. However, K is not limited to these numbers and may be set to another positive integer. In particular, values ​​of K equal to or greater than 2 are possible. N is a positive integer and represents the number of output channels of the convolutional neural network. Act.: Refers to the activation layer, which can be a nonlinear activation function, e.g., ReLU or GDN as mentioned above. Conv N×K×K / 2↑: Represents the same as Conv N×K×K, but this layer additionally upsamples (upconverts) the output by a factor of 2 both horizontally and vertically. This is indicated by the "2" and the up arrow. Conv. 2×K×K / 2↓: Represents the same as Conv N×K×K, but further downsamples (downconverts) the output by a factor of 2 both horizontally and vertically. This is indicated by the "2" and the down arrow. Conv. N×K×K, Act: Represents a convolutional layer + activation layer.

[0176] As shown in Figures 19 and 20, the color format is YUV420 for the picture portion to be encoded. This means that the luma size is wxh and the chroma size is w / 2xh / 2. However, as will be clear to those skilled in the art, other formats (types of chroma subsampling), such as YUV422, are also feasible.

[0177] Luma Processing - The size equalization network for luma processing in Figure 19 comprises a first layer Conv. NxKxK / 2↓ with KxK kernels and N channels as output. This layer downsamples the luma component by a factor of 2 in both horizontal and vertical directions. This is followed by a sequence of activation layers Act. and a convolutional layer Conv. NxKxK, Act. with activations. The size equalization network is terminated by a last layer which is a convolutional layer Conv. 1xKxK, thus having one channel as output, namely the downsampled luma component Y. Here, the luma component has been resized to have the same size (resolution) as the chroma components U and V. This processing is done in the upper branch (1710) of Figure 17.

[0178] Chroma Processing - The size equalization network for chroma component processing in Fig. 20 comprises a first layer Conv N×K×K / 2↑ with K×K kernels and N channels as output. This layer upsamples the two chroma components U and V by a factor of 2 in both horizontal and vertical directions. This is followed by a sequence of activation layers Act. and a convolutional layer Conv. N×K×K, Act. with activations. The size equalization network is terminated by a last layer which is a convolutional layer Conv. 2×K×K, thus having two channels as output, namely the upsampled chroma components U and V. Here, the chroma components U and V have been resized to have the same size (resolution) as the luma component Y. This processing is performed in the lower branch (1720) of Fig. 17. It should be noted that in this case the luma component is preserved with all the advantages discussed in the first embodiment.

[0179] As is evident from FIG. 19 and FIG. 20, in either case, the output of each convolutional layer Conv. N×K×K / 2↓ and Conv N×K×K / 2↑ has N channels and is further processed in an activation layer (i.e., a nonlinear activation function, e.g., ReLU or GDN). The resized luma and chroma(s) may be referred to as intermediate luma and intermediate chroma(s), respectively. The respective equal-sized output luma-chroma(s) are then used as inputs to the respective VAEs of the upper and lower branches. The VAEs of the upper and lower branches result in a coded bitstream, which may then be reconstructed to obtain the quality (loss). The quality (inversely proportional to the distortion) and rate (e.g., measured by the amount of bits in the bitstream) may then be used in the RDO 1730 to select the appropriate processing that may be performed by the VAE following the RDO.

[0180] The above size equalization process is not limited to the YUV420 color format, but may also be applied to other subsampled chroma formats, such as YUV422.

[0181] In the first embodiment, it was described that the encoder-side chroma upsampling may utilize the luma component as side information. This may be combined with the above processing of the chroma component(s) in the second embodiment. Furthermore, the same procedure may also be applied to the encoder-side luma downsampling in that chroma is used as side information.

[0182] Figure 23 further illustrates this for an exemplary case where, for an encoder side showing a similar encoder structure as Figure 17, the selection of picture processing for a picture portion may be signaled by a flag (indicator) encoded in the encoded bitstream. As Figure 23 shows, the difference with respect to the schematic diagram of Figure 17 is that the luma downsampling (upper branch) utilizes information about chroma components U and V. Similarly, the chroma(s) upsampling (lower branch) utilizes information about luma Y. Otherwise, the respective sampling processing of luma and chroma(s) is performed in the same manner as discussed above with reference to Figure 17.

[0183] Thus, regardless of whether luma downsampling and / or chroma upsampling is performed, in either case cross information is used for the respective sampling, which means that the correlation between luma and chroma(s) is preserved.

[0184] FIG. 25 and FIG. 26 show example implementations of size equalization networks for luma downsampling and chroma upsampling on the encoder side, respectively. As shown in FIG. 25 and FIG. 26, each size equalization network has the same design in terms of the order of how the convolutional layers and activation layers are arranged as shown in FIG. 19 and FIG. 20. The difference is that an additional layer is added to provide the chroma(s) of FIG. 25 or the luma of FIG. 26 to each size equalization network. In the case of FIG. 25, the added layer is a convolutional layer Conv N×K×K / 1↑ for chroma components with K×K kernels and N channels as output, which is input to the activation layer Act. In the case of FIG. 26, the added layer is a convolutional layer Conv. N×K×K / 1↓ for luma with K×K kernels and N channels as output, which is input to the activation layer Act. In the examples shown in Figures 25 and 26, the respective output signals are added to the output of the activation layer Act. of the size equalization network after the first convolutional layer Conv. N×K×K / 2↓ and Conv N×K×K / 2↑, respectively.

[0185] For example, each addition may be an element-wise addition of the chroma component(s) and the downsampled luma component(s) and the upsampled chroma component(s), respectively. Thus, chroma upsampling using luma as side information and luma downsampling using chroma as side information may be efficiently performed. Moreover, details that may be present in the luma originally having a higher resolution may be restored in the originally subsampled chroma. Similarly, details that may be present in the chroma originally having a lower resolution may be restored in the original luma.

[0186] The combining (combining) process of the luma and chroma components (in both the luma downsampling process and / or the chroma upsampling process) may not be limited to an addition operation. Alternatively, the chroma and luma may be concatenated. This typically results in a concatenated luma-chroma composite that has a size larger than each of the luma and chroma components. Other types of combining operations may be used instead and / or combined with addition or concatenation. Thereby, the luma and chroma information is utilized for chroma upsampling and / or luma downsampling by respective size equalization networks.

[0187] As mentioned above, which picture processing is performed on the encoding side can be determined in various ways, which are discussed next.

[0188] In an exemplary implementation of the method, the selecting includes evaluating a rate and distortion cost function. According to one implementation of the (encoding) method, the selecting further includes determining a cost of the intermediate downsampled luma and chroma components and a cost of the luma component and the intermediate upsampled chroma component, respectively, according to the cost function. Furthermore, in the selection, the picture processing of the picture portion whose determined cost is lower is selected. Thus, by using the sampling mode with the lowest cost (i.e., luma downsampling or chroma upsampling), the overall encoding of the luma component and the subsampled chroma components is efficiently performed.

[0189] For example, the cost may be a loss specified in terms of distortion and bit rate between the reconstructed picture portion and the original picture portion. In this case, the loss may be expressed as: Loss = Rate + λ Distortion or Loss = λ rate + distortion.

[0190] where λ represents the Lagrange multiplier, which is a number that may be predefined. This number may be empirically determined based on the desired application and / or outcome. As noted above, alternative or additional parameters of the cost function may be used. Furthermore, it should be noted that the present disclosure does not necessarily include RDO. For example, the selection may be made by a trained network (e.g., an ANN) that is pre-trained for such selection. Various other approaches may also be possible.

[0191] The selecting may further include downsampling the luma component, thereby obtaining an intermediate downsampled luma component, and upsampling the chroma component, thereby obtaining an intermediate upsampled chroma component. In FIG. 17, after RDO 1730, a VAE is applied to the color components of matching size after either luma downsampling or chroma upsampling. However, this does not have to be the case. Rather, instead of using the VAE again, the bitstream generated by the VAE of either the selected upper (1710) or lower (1720) branch may be taken. In other words, the coded portion may be reused, thus reducing some complexity.

[0192] FIG. 17 shows three VAEs. Applying VAEs three times can be complex and time-consuming. Thus, in some example implementations, to reduce complexity, the VAEs in the upper and lower branches are not full networks. Rather, they may be replaced by models that do not actually generate the bitstream, but rather estimate the rate and distortion. After it is estimated / selected which branch is better, the actual VAE is used after RDO 1730 to generate the bitstream that is sent (conveyed) to the decoder. This may be advantageous if the rate and distortion estimation by the model VAE in the branches is faster than the VAE following the RDO. On the encoder side, both cases are possible (two VAEs vs. three VAEs).

[0193] In one of the example implementations, the cost determination includes encoding the intermediate downsampled luma and chroma components into a first bitstream, and encoding the luma and intermediate upsampled chroma components into a second bitstream. This is followed by determining the cost of the first and second bitstreams according to the cost function. Thus, the actual cost can be accurately determined since it is based on the encoded first and second bitstreams. However, as mentioned above, the present disclosure is not limited to such an approach. Rather, instead of encoding into a bitstream, the cost may be estimated by an estimation function. Furthermore, the encoding to obtain the rate does not have to be performed by actually generating a bitstream. For example, the cost may be estimated in terms of the number of quantization data and / or their distribution, or by applying a trained network trained for such estimation, or by any conceivable manner.

[0194] According to one implementation, generating a bitstream for carrying encoded picture or video data includes encoding a selection indicator into the bitstream indicating a selected picture processing. This provides low-complexity signaling of the picture processing performed on the bitstream. For example, the selection indicator may have a length of one bit and may be able to distinguish between two values, one representing luma downsampling and the other representing chroma upsampling.

[0195] This is illustrated in FIG. 17. As mentioned above, the luma downsampling (upper branch) and chroma upsampling (lower branch) sampling processes are performed on a picture portion having a luma component and a subsampled chroma component. In both cases, equal-sized luma-chroma components are provided and input, for example, to the VAEs of the upper and lower branches, respectively. Both VAEs calculate a cost, which may be based on bitrate and distortion (RDO), for each luma-chroma input. Depending on which cost is lower, the respective picture process is selected. Based on the selected picture process, a bitstream is generated, but without encoding an indicator in the bitstream. The selection of said picture process (i.e., luma downsampling or chroma upsampling at the encoder side) may be indicated by a selection indicator. The indicator may be, for example, a flag. By inserting (which may include encoding) a flag in the bitstream, the selected picture process performed at the encoding side may be signaled in the bitstream to the decoder. The decoder then parses the indicator from the bitstream and determines which picture process to perform based on the indicator. In particular, such a selection indicator can take a first value and a second value. A first value of such a selection indicator may indicate that luma has been downsampled (1710) at the encoder and should therefore be upsampled at the decoder. A second value of such a selection indicator may indicate that chroma has been upsampled (1720) at the encoder and should therefore be downsampled at the decoder. In Figure 17, the functional block "up or down" 1740 represents the insertion of the selection indicator into the bitstream.

[0196] Alternatively, the selected picture processing may be indicated without an explicit indicator (eg, a one-bit flag) encoded in the bitstream.

[0197] According to another implementation example, generating the bitstream includes encoding an indication of the width W and / or height H of the luma component into the bitstream. Thus, the width and height of the original picture portion may be signaled to the encoder. Such an indication may specify the resolution of the luma component. If the ratio between the resolution of the luma and the chroma components is known, the chroma resolution may be determined accordingly based on the signaled W and H. This signaling may be part of the picture or video coding. Typically, an encoded picture or video bitstream includes an indication of the picture or video dimensions.

[0198] An example implementation is illustrated in FIG. 29. Similar to the example illustrated in FIG. 17, for a picture portion having a luma component and a chroma component with a lower resolution than the luma component (hence "subsampled" with respect to the luma component), a luma downsampling (upper branch) 2910 and a chroma upsampling (lower branch) 2920 sampling process is performed. The equal-sized luma-chroma components of the upper and lower branches are input to a VAE, respectively. Both VAEs calculate a cost, which may be based on bitrate and distortion (RDO) 2930, for each luma-chroma input. Depending on which cost is lower, a respective picture process is selected. Based on the selected picture process, a bitstream is generated. In addition to FIG. 17, this example implementation does not encode a selection indicator (e.g., a flag) into the bitstream. The decoder may then determine which picture process to perform based on the bitstream, e.g., based on the luma width and height encoded in the bitstream. Alternatively or additionally, further parameters, e.g., signaled in the bitstream for coding purposes, may be used in the decoder to derive which processing (luma upsampling or chroma downsampling) should be applied. Such parameters may be predefined and known in the encoder and decoder so that the encoder and decoder use them consistently for the selection of the processing.

[0199] According to an embodiment of the present disclosure, a method for reconstructing a picture portion from a bitstream is provided. Such a method may be applied at the decoder side. As described above with reference to embodiment 1, a picture portion includes a luma component and a chroma component having a resolution lower than that of the luma component. A picture portion refers to a number of samples of an image and / or picture, said number being smaller than the total number of samples of said image / picture. A picture portion may also include all samples of an image or picture. A picture may be a still picture or a video (a sequence of pictures). A picture portion may include one or more chroma components. For example, a picture portion of luma-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution may be lower than the resolution of the luma channel (in the picture portion or in the whole picture). The luma and chroma components of a picture portion are typically spatially correlated since they refer to the same picture region.

[0200] The method is illustrated in FIG. 32 and includes a step S3210 of decoding luma and chroma components having the same resolution from a bitstream. As mentioned above, the resolution may be given by the size (i.e., dimension) of the two-dimensional (2D) luma channel. The decoding of the processed picture portion from the bitstream may be performed by an autoencoder, such as a variational autoencoder (VAE). VAEs are known in the art for performing tasks related to encoding and decoding of images or pictures very efficiently, with a good compromise between the required bit rate and the quality of the reconstructed picture. However, the present disclosure is not limited to the application of autoencoders. In general, any encoder may be applied, including, for example, an encoder that uses a network (e.g., an ANN or any ML / DL based network) trained only on a part of the decoder operation or functionality.

[0201] The method is as follows: upsampling the decoded luma component to obtain a reconstructed picture portion comprising the decoded chroma components and the upsampled luma component; and downsampling the decoded chroma components to obtain a reconstructed picture portion comprising the decoded luma component and the downsampled chroma components. The method further includes a step S3220 of determining one of the above based on the bitstream.

[0202] The determination may be made by parsing a selection indicator from the bitstream to indicate the processing to be applied. However, as noted above, in some embodiments, the selection indication need not be explicitly signaled. Rather, it may be derived from other parameters in the bitstream associated with the encoded picture or video data.

[0203] Furthermore, the method includes a step S3230 of obtaining said picture portion according to the determined picture processing. Thus, the picture portion encoded in the bitstream can be processed by the decoder portion of the same type of trained network, such as the VAE shown in FIG. 2, since the encoded luma and chroma components have the same size. Furthermore, a picture processing decision of one of luma upsampling and chroma downsampling provides in each case resized luma or chroma components such that they have different sizes corresponding to the components input for encoding (original picture components). This means that the picture portion is reconstructed to its original color format, with chroma having a lower resolution than chroma. Thus, regardless of the determined picture processing, the encoded equal-sized luma-chroma channels can be handled within a common VAE framework. Furthermore, the picture processing decision allows adaptively switching between luma upsampling and chroma downsampling. This ensures that in each case the picture portion with its original color format is reconstructed. Thus, adaptive switching allows the selection of low or high resolution picture processing based on some optimization criterion, such as for example RDO.

[0204] FIG. 18 illustrates the decoder side and its operation. Referring to the example of FIG. 17, where the sampling mode is indicated by a flag (indicator) encoded in the bitstream, the flag is parsed from the bitstream at the decoder side. Depending on the value of the flag in the functional block 1840 (i.e., whether it indicates downconversion (luma) or upconversion (chroma) at the encoder side), in the respective upper and lower branches, luma is upconverted 1810 or chroma is downconverted 1820. This means that the luma-chroma components of the input bitstream have either the size of the chroma(s) or the size of the luma of the originally coded picture, depending on the indication by the flag. In either case, the respective upsampling and / or downsampling operations may be performed by a size conversion operation, providing as output a reconstructed frame in its original format. However, as shown in FIG. 18, the upper branch 1810 performs luma upsampling and the lower branch performs chroma component (U and / or V) downsampling 1820. Each size conversion may be performed by a corresponding size conversion network. As in the case of FIG. 17, the indication of the sampling mode by the encoded flag is one example that allows the decoder to determine which sampling mode to perform for the reconstruction of the original picture portion. Various means of determining which sampling mode to perform at the decoder side may be performed by other methods as discussed below. Furthermore, it should be noted that in these examples, the resized color components (channels) match the size of the originally coded picture. However, this is not intended to limit the present disclosure. In some applications, it may be beneficial to provide color components with dimensions different from the dimensions of the originally coded picture.

[0205] In one implementation, the upsampling and / or downsampling is performed by a trained network. The trained network on the encoding side may be any kind of network suitable for downsampling or upsampling. This includes, for example, artificial neural networks (ANNs), such as neural networks, convolutional neural networks (CNNs), etc. The trained network in the decoder may be any resizing network that adapts the size of the channels input to it depending on the desired resolution. The result may be channels with different dimensions (sizes), such as the dimensions of the originally encoded picture or other desired dimensions.

[0206] Figures 21 and 22 show an example implementation of a decoder-side size conversion network for the YUV444 color format, where the luma and chroma components U and V have the same size. Figure 21 shows a size conversion network with luma upsampling (top branch corresponding to 1810). Figure 22 (corresponding to 1820) shows a size conversion network with two chroma U and V downsampling. In this example, each layer can be seen as a layer of a convolutional neural network.

[0207] Referring to Figure 5, the convolutional network layers shown in Figures 21 and 22 are similarly defined. Correspondingly, the same terminology and / or notation applies here as well. As shown in Figures 21 and 22, the input color format is YUV444 for the picture portion to be decoded from the bitstream. This means that the luma and chroma components decoded from the bitstream have the same size (i.e., resolution).

[0208] Luma Processing (top branch 1810 in Fig. 18 and Fig. 21): The size conversion network for luma processing consists of a sequence of convolutional layer + activation layer pairs Conv. N×K×K, Act. The size conversion network is terminated by a last layer, which is a convolutional layer Conv 1×K×K / 2↑, which upsamples the luma component by a factor of 2 in both horizontal and vertical directions. The output is one channel, i.e., a reconstructed luma component with the same size as the original luma.

[0209] Chroma Processing (lower branch 1820 in FIG. 18 and FIG. 22): The size conversion network for chroma component processing similarly consists of a convolution layer + activation layer Conv. N×K×K. Act. The size conversion network is terminated by a last layer which is a convolution layer Conv. 2×K×K / 2↓, which downsamples the two chroma components by a factor of 2 in both horizontal and vertical directions. The output is two channels, i.e., two reconstructed chroma components with the same size as the original chroma components. Note that in this case the luma component can be preserved with all the advantages discussed in the first embodiment.

[0210] Since the input luma-chroma components to the decoding-side size conversion network(s) have the same resolution (i.e., size), they can be efficiently processed within the same existing framework, such as an ANN architecture with the same input size. Thus, picture parts with any color format with subsampled chroma(s) can be reconstructed using the same network architecture as applied to luma. Thus, it also allows preserving inter-component correlation information in the processing of different components (channels).

[0211] In the first embodiment, it was described that the decoder-side chroma downsampling may utilize the luma component as side information. This may be combined with the above processing of the chroma component(s) in the second embodiment. Furthermore, the same procedure may also be applied to the decoder-side luma upsampling in that chroma is used as side information.

[0212] Figure 24 further illustrates this for the decoder side showing a decoder structure similar to Figure 18, for an exemplary case where the selection of picture processing for a picture portion may be signaled by a flag (indicator) encoded in the encoded bitstream. As Figure 24 shows, the difference with the schematic of Figure 18 is that the luma upsampling (upper branch) utilizes information about chroma components U and V. Similarly, the chroma(s) downsampling (lower branch) utilizes information about luma Y. Otherwise, the respective sampling processing of luma and chroma(s) is performed in the same manner as discussed above with reference to Figure 18.

[0213] Thus, regardless of whether luma upsampling and / or chroma downsampling is performed, in both cases cross information is used for the respective sampling. This means that the correlation between luma and chroma(s) can be exploited to reconstruct the channel. The same can be applied to channels of other color spaces, or in general to any channel, not limited to color channels, such as depth, motion or other additional information related to the picture.

[0214] Figures 27 and 28 show example implementations of size conversion networks for decoder-side luma upsampling and chroma downsampling, respectively. As shown in Figures 27 and 28, the size conversion networks have the same design in terms of the order of how the convolutional layers and activation layers are arranged as shown in Figures 21 and 22. The difference is that in the case of Figure 27, the chroma components U and V are directly input to the first convolutional layer Conv. N×K×K, Act. together with the downsampled luma. Similarly, in the case of Figure 28, the luma is directly input to the first convolutional layer Conv. N×K×K, Act together with the upsampled chroma components U and V.

[0215] This means that on the decoder side, the first processing layer of the size conversion network may not need to perform addition operations in contrast to the encoder side, and such kinds of operations can be done as shown in Figures 25 and 26. The combining (combination or joint processing) in the size equalization network of the encoder helped to take advantage of inter-component correlation. There is no need to invert such processing on the decoder side, because the input to the first layer of the size conversion network of the decoder, including the luma and chroma components, already has the same size obtained at the encoder, as illustrated in Figure 24.

[0216] As mentioned above, which picture processing is performed on the decoding side can be determined in various ways, which are discussed next.

[0217] In one example implementation, determining the processing to be applied further includes decoding a selection indicator from the bitstream indicating a picture processing and determining the processing indicated by the selection indicator as said picture processing. This allows for efficient signaling of picture processing. Correspondingly, the decoding side may quickly determine which picture processing was used at the encoder side (i.e., luma downsampling or chroma upsampling) based on a simple indicator included in the bitstream. The selection indicator is described above with reference to the encoder side. As described above, it may be an indicator that can take at least a first value indicating luma upsampling (at the decoder side, this corresponds to luma downsampling at the encoder side) and a second value indicating chroma(s) downsampling (at the decoder side, this corresponds to chroma(s) upsampling at the encoder side). It should be noted that the selection indicator may generally take one of three or more values. For example, the selection indicator may take a third value indicating that no resizing is performed and / or a fourth value indicating that two or more components are resized, or further specify how the resizing should be performed (only which allocations, or to which desired resolution). This is true for both the encoder and the decoder side, since the encoder and the decoder should have the same understanding (interpretation or semantics) of the selection indicator. On the other hand, providing a one-bit flag (indicator) that can take only one of the two values ​​(first value and second value) may be efficient, since it does not introduce high additional signaling overhead.

[0218] The decoder-side operation is illustrated in Figure 18. As mentioned above, the decoder receives a bitstream with luma and chroma components of the same size. This is the result of picture processing at the encoder side. If a selection indicator (flag, e.g., RDO) is encoded in the bitstream, the decoder parses the bitstream and determines which picture processing to perform based on the selection indicator. The decoder then upsamples the luma component if the flag indicates that the luma has been downsampled at the encoder side. Conversely, the decoder downsamples the chroma component if the flag indicates that the chroma component has been upsampled at the encoder side. In either case, the original size of the picture portion is reconstructed.

[0219] The determining may in some embodiments include (in addition to or as an alternative to other example implementations) decoding the width W and / or height H of the luma component of the picture portion from the bitstream. This allows determining the picture processing based on the width and height of the luma channel of the picture portion. In other words, the channel carrying the important information of the picture portion is used as a reference for said determination.

[0220] Moreover, according to one example implementation, determining further comprises determining whether the width W and height H are the respective widths of the decoded luma components.

number

number

number

number

number

number

number

number

[0221] It should be noted that this picture processing of the picture parts on the encoding side and on the decoding side, i.e. chroma upsampling (encoder) and chroma downsampling (decoder), corresponds to the processing of the first embodiment as far as the sampling process (up / down) is concerned.

[0222] In the above example, W and H of the original picture are respectively

number

number

[0223] However, this disclosure is not limited in this respect. The default functionality may instead include upsampling the luma at the decoder side, assuming that the encoder downsampled the luma.

[0224] Correspondingly, as a result of the decision, either W or H

number

number

number

number

number

number

[0225] So the picture processing is done by dividing the luma W and H and the respective widths of the decoded luma.

number

number

[0226] It should be noted that the above chroma downsampling or luma upsampling may both be applicable at the decoder. For example, the decoder and encoder may have a rule according to which of the two processing types is used. The rule may be a decision based on some conditions that depend on some syntax elements (encoding parameters), such as width and height, and / or quality parameters or other parameters.

[0227] In another example implementation, determining the processing to be applied includes determining a size S of the bitstream. The size S may be in any unit, such as bits, bytes, words, etc. The determining includes determining from the bitstream a latent representation of the picture portion.

number

number

number

number

number

number

number

[0228] If the determination of the processing to be applied results in a ratio less than a predefined threshold, then in one example implementation the decoded luma component is upsampled, whereas in this case the luma was downsampled on the encoding side (lower resolution result).

[0229] Therefore, picture processing depends on the bitstream size and the size of the latent space.

number

[0230] This is shown in Figure 30. In this case, the decoder determines from the bitstream which picture processing to perform. Said determination may be made, for example, by a decision module that parses the bitstream. For example, if the width W and height H of the luma component of a picture part are encoded in the bitstream, the decoder parses W and H from the bitstream. Furthermore, the luma and chroma components are decoded from the bitstream. Thus, the width of the decoded luma component

number

number

number

number

number

number

[0231] Alternatively or additionally, the decision module may determine the size S of the bitstream and the latent space representation

number

number

number

number

[0232] If so, i.e., if R>=T, then the decoded chroma components are downsampled. In this case, chroma was upsampled at the encoding side (high resolution result). Otherwise (i.e., if R>T), then the decoded luma components are upsampled. In this case, luma was downsampled at the encoding side (low resolution result). According to one implementation, the determining comprises:

number

[0233] Such a neural network-based decision may also be made by a decision module, which decides the picture processing based on the classification result. The classification may be, for example, a classification of a reconstructed latent space as input.

number

[0234] For example, chroma upsampling (encoding side) may involve inserting a line of samples after every n lines of samples of a chroma component, where line refers to row and / or column, and n is an integer equal to or greater than 1. This means that, for example, after every one (n=1) or two (n=2) rows and / or columns, said line samples may be inserted. This allows further adaptation of chroma upsampling by inserting line samples at specific rows and / or columns. A similar process may be applied for luma at the decoder side.

[0235] Alternatively, chroma upsampling (encoding side) may involve inserting a line of samples with value 0 after every n lines of samples of the chroma components. This may be called zero-filling. This is a simple scheme for upsampling by simply setting each line sample to 0. Thus, no further signaling and / or processing overhead is required. A similar process may be applied to luma at the decoder side.

[0236] This is illustrated in FIG. 9, which shows 4×4 samples of upsampled chroma. The black circles refer to samples of the original chroma. The white circles refer to samples with zero values. In this example, zero-value samples are inserted after each of the first rows and columns of the chroma components. Furthermore, in this example, the size equalization network is simplified to a one-layer network (M=1) with no further layers after the “add” operation in FIG. 5. In FIG. 9 (encoder side), the upsampling of U and V is simplified as filling each row and each column with zeros for the case of YUV420 color format. In the case of YUV422 format, the filling of zeros is for each column). In this case, the network is not a convolutional neural network, but can be understood as one layer of a fully connected neural network (FCNN). Therefore, this network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network.

[0237] In another implementation, chroma upsampling may be performed by inserting a line of samples having a value equal to the respective nearest neighbor sample in the chroma component after every n lines of samples of the chroma component. The value insertion may be performed, for example, by copying the chroma value to the respective nearest neighbor sample. As in the case of zero filling above, this allows further adaptation of the chroma upsampling by inserting line samples at specific rows and / or columns. Furthermore, using the value of the inserted line sample with reference to the nearest neighbor sample of the chroma, the accuracy of the chroma upsampling is more accurate because the neighbor information (reflected by the neighbor chroma sample value) with reference to the original chroma is used.

[0238] This is illustrated in FIG. 11, which shows 4×4 samples of upsampled chroma. Again, the black circles refer to the original chroma samples. Here, the white circles refer to samples whose values ​​depend on their respective chroma neighbors. In FIG. 11, the insertion of chroma values ​​into the respective neighbors is indicated by arrows. In this example, the sample values ​​are inserted after each of the first rows and columns of chroma components. Furthermore, in this example, the size equalization network is simplified to a one-layer network (M=1) with no further layers after the “add” operation of FIG. 5. In FIG. 11 (encoder side), the upsampling of U and V is simplified as the upsampling of the nearest neighbor samples in each row and each column, as shown for the YUV420 format in FIG. 11. In the case of the YUV422 format, the upsampling of the nearest neighbor samples is for each column. In the above example of chroma upsampling, the network is not a convolutional neural network, but can be understood as one layer of a fully connected network. Therefore, this network is not trainable by itself, but can be trained together with the VAE structure for better coding performance. Thus, each FCNN can also be referred to as a trained network.

[0239] However, this is just an example of a trained network for size equalization with one layer. It is clear to those skilled in the art that the size equalization layer may have two or more layers, as shown in Figures 5 and 7. In this case, the size equalization network may be trained to perform chroma upsampling.

[0240] A similar picture process of upsampling luma on the decoding side may be performed, at least with respect to filling every nth line with some additional samples. However, it may be advantageous to perform luma upsampling by using an interpolation filter, i.e., by determining the value of the summed luma sample as a function of neighboring samples, such as an average or weighted average. It may be advantageous to perform luma upsampling by using a trained (pre-trained) network, such as an ML / CL-based network.

[0241] For example, chroma downsampling (decoder side) may be performed by removing a line of samples after every n lines of samples of intermediate chroma components, where a line is a row and / or column and n is an integer equal to or greater than 1. This means that, for example, after every one (n=1) or two (n=2) rows and / or columns, said line (row and / or column) may be removed (deleted). Thus, intermediate chroma downsampling can be performed quickly by simply removing the respective line samples.

[0242] Examples of line sample removal are shown in Figures 10 and 12. Note that the sample configurations in Figures 10 and 12 correspond to those in Figures 9 and 11, respectively, as a result of the upsampling process of the chroma components at the encoder side. Specifically, Figures 10 and 12 show 4x4 samples of intermediate chroma. The black circles refer to samples of the original chroma. The open circles with crosses refer to samples that may be non-zero as a result of the chroma upsampling process by the size equalization network at the encoding side (with and without using luma as side information) and / or the chroma processing through the first layer(s) of the size conversion network at the decoding side.

[0243] In the examples shown in Fig. 10 and Fig. 12, the size conversion network is simplified as a one-layer network (M=1). In Fig. 10 and Fig. 12 (decoder side), the downsampling of U and V is simplified to discard odd row and column samples in the case of YUV420 color format. In YUV422, odd column samples are removed. In either of these cases shown in Fig. 10 and Fig. 12, the respective networks are not convolutional neural networks, but can be understood as one layer of a fully connected neural network (FCNN). Therefore, in this example, the network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, the respective FCNNs can also refer to trained networks.

[0244] However, this is just an example of a trained network for size conversion with one layer. It is clear to those skilled in the art that the size conversion layer may have two or more layers, as shown throughout Figures 6 and 8. In this case, the size conversion network may be trained to perform chroma upsampling.

[0245] It should be noted that the encoder-side size equalization network shown in Fig. 5 and the decoder-side size conversion network shown in Fig. 6 can be trained together using a VAE structure. This can further improve the encoding-decoding process, especially for subsampled chroma components, within the VAE framework, including, for example, the encoding-decoding performance and / or the quality of the reconstructed picture / image.

[0246] Alternatively, they may be trained separately and used as pre-trained VAE networks. In either case, the respective networks are referred to as trained networks or pre-trained networks. Similar picture processing of downsampling luma on the encoding side may be performed. In general, training may be done in advance, i.e., before the network is used in the inference phase for encoding, upsampling, decoding, etc.

[0247] Moreover, as previously explained, the present disclosure also provides devices configured to perform the steps of the above methods.

[0248] FIG. 33 illustrates an encoder 3300 for encoding a picture portion. The picture portion includes a luma component and a chroma component having a resolution lower than that of the luma component. The encoder comprises a selection module 3310 configured to select, as a picture processing for the picture portion, one of downsampling the luma component to obtain a downsampled luma component that matches the resolution of the chroma component and upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the luma component. Furthermore, the encoder comprises a processing module 3320 configured to process the picture portion according to the selected picture processing. The encoder further comprises a generation module 3330 configured to generate a bitstream according to the selected picture processing, where the generating comprises encoding the processed picture portion.

[0249] In one implementation, the encoder may include one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the encoder to perform the above method.

[0250] FIG. 34 illustrates a decoder 3400 for reconstructing a picture portion from a bitstream. The picture portion includes a luma component and a chroma component having a resolution lower than that of the luma component. The decoder comprises a decoding module 3410 configured to decode the luma component and the chroma component having the same resolution from the bitstream. Furthermore, the decoder comprises a determination module 3420 configured to determine, as a picture operation from the bitstream, one of upsampling the decoded luma component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled luma component, and downsampling the decoded chroma component to obtain a reconstructed picture portion including the decoded luma component and the downsampled chroma component. The decoder further comprises an retrieval module 3430 configured to retrieve the picture portion according to the determined picture operation.

[0251] In one implementation, the decoder may include one or more processors and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the decoder to perform the above method.

[0252] A computer program stored on a non-transitory medium comprising code that, when executed on one or more processors, performs the steps of any of the above methods, according to one embodiment of the present disclosure.

[0253] Some of the embodiments of the present disclosure may provide advantages that may be summarized as follows. Luma sample information is fully preserved. Luma spatial information is fully preserved. The size equalization CNN and the size conversion CNN can be trained together using a VAE network structure. The joint training may further improve the results. Alternatively, the size equalization CNN and the size conversion CNN may also be trained separately from the VAE. Other kinds of chroma subsampling formats (e.g. YUV422) can be handled by the VAE network structure. Inter-component luma-chroma information can be preserved. · Adaptive switching between luma downsampling for small resolutions that save bitrate, and chroma upsampling for large resolutions that preserve luma information.

[0254] Some exemplary implementations in hardware and software A corresponding system in which the above encoder-decoder processing chain may be deployed is shown in FIG. 35. FIG. 35 is a schematic block diagram illustrating an example coding system, e.g., a video, image, audio, and / or other coding system (or coding system for short) that may utilize the techniques of the present application. The video encoder 20 (or encoder for short) and the video decoder 30 (or decoder for short) of the video coding system 10 represent examples of devices that may be configured to perform techniques according to various examples described in the present application. For example, the video coding and video decoding may use neural networks such as those shown in FIG. 2A to FIG. 2C, which may be distributed and may include the above encoded bitstreams in advance where chroma is upsampled (embodiment 1) or where it is evaluated whether to upsample chroma or downsample luma (embodiment 2).

[0255] As shown in FIG. 35, coding system 10 includes, for example, a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding the encoded picture data 13.

[0256] The source device 12 comprises an encoder 20, and may additionally, i.e. optionally, comprise a picture source 16, a pre-processor (or pre-processing unit) 18, for example a picture pre-processor 18, and a communication interface or unit 22.

[0257] It should be noted that the first and second embodiments of the present disclosure and their exemplary implementations described above in relation to encoding may be performed as part of the encoder 20. However, alternatively, it is contemplated that these embodiments are performed by the pre-processor 18. Similarly, on the decoder side, the first and second embodiments of the present disclosure and their exemplary implementations described above in relation to decoding may be performed as part of the decoder 30. However, alternatively, it is contemplated that these embodiments are performed by the post-processor 32.

[0258] Picture source 16 comprises or may be any kind of picture capture device, such as a camera for capturing real-world pictures, and / or any kind of picture generation device, such as a computer graphics processor for generating computer-animated pictures, or any kind of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any kind of memory or storage that stores any of the aforementioned pictures.

[0259] In distinction to the pre-processor 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 may also be referred to as a raw picture or raw picture data 17 .

[0260] The pre-processor 18 is configured to receive the (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It can be understood that the pre-processing unit 18 may be an optional component. It is noted that the pre-processing may also use a neural network (e.g., as shown in any of Figures 1 to 7). As mentioned above, the pre-processing may include processing by a neural network (trained network) to upsample chroma (embodiment 1 and / or embodiment 2) or downsample luma (embodiment 2).

[0261] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0262] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0263] The destination device 14 comprises a decoder 30 (e.g., a video decoder 30) and may additionally, i.e. optionally, comprise a communications interface or communications unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0264] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), e.g. directly from the source device 12 or from any other source, e.g. a storage device, e.g. a storage device for encoded picture data, and to provide the encoded picture data 21 to a decoder 30.

[0265] The communication interface 22 and the communication interface 28 may be configured to transmit and receive the encoded picture data 21 or the encoded data 13 via a direct communication link between the source device 12 and the destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired or wireless network or any combination thereof, any type of private and public network, or any type of combination thereof.

[0266] The communications interface 22 may be configured, for example, to package the encoded picture data 21 in a suitable format, e.g., packets, and / or process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0267] Communications interface 28 forms the counterpart to communications interface 22 and may, for example, be configured to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or unpackaging to obtain encoded picture data 21.

[0268] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces as indicated by the arrows of communication channel 13 in Figure 35 pointing from source device 12 to destination device 14, or as bidirectional communication interfaces, and may be configured, for example, to send and receive messages, for example, to set up connections, and to verify and exchange communication links and / or any other information related to data transmission, e.g., encoded picture data transmission. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded pictures 31 (e.g., using a neural network based on one or more of Figures 1 to 7).

[0269] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., a post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing, e.g., to prepare the decoded picture data 31 for display, e.g., by a display device 34.

[0270] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33 for displaying the picture, e.g., to a user or viewer. Display device 34 may be or include any type of display for presenting the reconstructed picture, e.g., an integrated or external display or monitor. The display may include, e.g., a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0271] 35 depicts source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both or both functionality, i.e., source device 12 or corresponding functionality and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0272] As will be clear to a person skilled in the art based on the description, the presence and (exact) division of functions or functions of different units within the source device 12 and / or destination device 14 shown in FIG. 35 may vary depending on the actual device and application.

[0273] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including neural networks, such as those shown in any of or portions of FIGS. 2A-2C. The decoder 30 may be implemented via processing circuitry 46 to embody various modules discussed with respect to FIGS. 2A-2C and / or any other decoder system or subsystem described herein. The processing circuitry may be configured to perform various operations, as described below. If the techniques are implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and may execute those instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) in a single device, for example, as shown in FIG.

[0274] Source device 12 and destination device 14 may comprise any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiving device, a broadcast transmitting device, and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0275] In some cases, the video coding system 10 illustrated in FIG. 35 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video coding and video decoding) that do not necessarily include data communication between encoding and decoding devices. In other examples, data is retrieved from local memory or streamed over a network. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that simply encode data to memory and / or retrieve data from memory and decode it without communicating with each other.

[0276] 37 is a schematic diagram of a video coding device 1000 according to one embodiment of the present disclosure. The video coding device 1000 is suitable for implementing embodiments of the disclosure described herein. In one embodiment, the video coding device 1000 may be a decoder, such as the video decoder 30 of FIG. 35, or an encoder, such as the video encoder 20 of FIG. 35.

[0277] The video coding device 1000 comprises an ingress port 1010 (or input port 1010) and a receiver unit (Rx) 1020 for receiving data, a processor, logic unit, or central processing unit (CPU) 1030 for processing the data, a transmitter unit (Tx) 1040 and an egress port 1050 (or output port 1050) for transmitting the data, and a memory 1060 for storing the data. The video coding device 1000 may also comprise optical-electrical (OE) and electrical-optical (EO) components coupled to the ingress port 1010, the receiver unit 1020, the transmitter unit 1040, and the egress port 1050 for the entry and exit of optical or electrical signals.

[0278] The processor 1030 is implemented by hardware and software. The processor 1030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGA, ASIC, and DSP. The processor 1030 communicates with the ingress port 1010, the receiver unit 1020, the transmitter unit 1040, the egress port 1050, and the memory 1060. The processor 1030 comprises a coding module 1070. The coding module 1070 implements the embodiments of the disclosure described above. For example, the coding module 1070 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 1070 provides a significant improvement to the functionality of the video coding device 1000, resulting in the transformation of the video coding device 1000 into different states. Alternatively, the coding module 1070 is implemented as instructions stored in the memory 1060 and executed by the processor 1030.

[0279] The memory 1060 may comprise one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device for storing programs when such programs are selected for execution and for storing instructions and data read during the execution of the programs. The memory 1060 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0280] FIG. 38 is a simplified block diagram of an apparatus 800 that may be used as either or both of the source device 12 and destination device 14 of FIG. 35, according to one exemplary embodiment.

[0281] The processor 1102 in the device 1100 may be a central processing unit. Alternatively, the processor 1102 may be any other type of device, or multiple devices, now existing or later developed, capable of manipulating or processing information. Although implementations of the disclosure may be implemented with a single processor as shown, e.g., processor 1102, advantages in speed and efficiency may be achieved using two or more processors.

[0282] The memory 1104 in the apparatus 1100 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 1104. The memory 1104 may include code and data 1106 that is accessed by the processor 1102 using a bus 1112. The memory 1104 may further include an operating system 1108 and application programs 1110, which include at least one program that enables the processor 1102 to perform the methods described herein. For example, the application programs 1110 may include Application 1 through Application N, which further include a video coding application that performs the methods described herein.

[0283] The apparatus 1100 may also include one or more output devices, such as a display 1118. The display 1118, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 1118 may be coupled to the processor 1102 via the bus 1112.

[0284] Although depicted here as a single bus, bus 1112 of device 1100 may be comprised of multiple buses. Additionally, secondary storage may be directly coupled to other components of device 1100 or may be accessed over a network, and may comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 1100 may be implemented in a wide variety of configurations.

[0285] Overview of Some Exemplary Embodiments Some embodiments of the present disclosure provide methods and apparatus for encoding and decoding of picture portions having luma and subsampled chroma components as input. The luma component may be processed by downsampling so that the encoding-decoding process is performed more quickly at a lower bit rate, resulting in a lower resolution. The chroma components are processed by chroma upsampling to provide a high-resolution result. The spatial correlation between luma and chroma may also be preserved in this way. In either case, the resized luma-chroma components have a matched resolution, and encoding by a trained network is applied to the matched components. The encoder may adaptively select between luma downsampling and chroma upsampling, and the decoder performs the inverse picture processing of luma upsampling and chroma downsampling. Thus, the encoding-decoding process may provide either a low-resolution or a high-resolution picture / image depending on the selected picture processing.

[0286] According to one aspect of the disclosure, a method for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the method comprising: selecting, as a processing of the picture portion, one of downsampling the luma component to obtain a downsampled luma component that matches the resolution of the chroma component, and upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the luma component; processing the picture portion according to the selected picture processing; and generating a bitstream according to the selected processing, the generating comprising encoding the processed picture portion.

[0287] This provides an equal-sized luma-chroma output as output, regardless of whether the luma component is downsampled or the chroma component is upsampled. Thus, the output can then be handled by a trained network (e.g., in a machine learning framework) with a similar structure, in particular the same input resolution. If the resolution of all color components is the same, the machine learning framework can exploit cross-color correlation in the same way at all sample positions. In one of the picture processes, the luma channel can remain unaffected, in which case there is no quality loss.

[0288] Downsampling luma is beneficial for saving bitrate at low resolutions, and upsampling chroma is beneficial for preserving luma information at large resolutions. Thus, adaptive switching allows the selection of a sampling mode depending on requirements and / or limitations on bitrate or picture quality. For any selected picture processing of a picture portion, the processed luma-chroma components have a matched resolution, so that the output can be handled within a VAE framework or another framework that allows efficient encoding.

[0289] In some example implementations, the selecting step includes evaluating a rate and distortion cost function, and thus the selection of the sampling mode (i.e., luma downsampling and chroma upsampling) is cost-based.

[0290] According to one exemplary implementation of the method, the selecting step includes downsampling the luma component, thereby obtaining an intermediate downsampled luma component, and upsampling the chroma component, thereby obtaining an intermediate upsampled chroma component. The selection of a sampling mode including luma downsampling and chroma upsampling in each case provides intermediate luma and intermediate chroma components, each having a matching resolution (i.e., size). The equal-sized luma-chroma components may then be subjected to further processing, for example by a VAE.

[0291] In a further implementation of the method, the selecting step includes determining the cost of the intermediate downsampled luma and chroma components and the cost of the luma component and the intermediate upsampled chroma component, respectively, according to the cost function, and in the selection, the picture processing of the picture portion whose determined cost is lower is selected. Thus, by using the sampling mode with the lowest cost (i.e., luma downsampling or chroma upsampling), the overall encoding of the luma component and the subsampled chroma components is efficiently performed.

[0292] According to one implementation example of the method, the determining step includes: encoding the intermediate downsampled luma and chroma components into a first bitstream, encoding the luma and intermediate upsampled chroma components into a second bitstream, and determining costs of the first and second bitstreams according to the cost function. Thus, the actual cost can be accurately determined because it is based on the actual encoded first and second bitstreams.

[0293] In one example method, generating the bitstream includes encoding a selection indicator into the bitstream indicating the selected picture processing, which provides low complexity signaling of the picture processing performed on the bitstream.

[0294] In another example, generating the bitstream includes encoding into the bitstream an indication of the width W and / or height H of the luma component. Thus, the width and height of the original picture portion may be signaled to the encoder.

[0295] According to an exemplary implementation of the method, the upsampling and / or downsampling is performed by a trained network. Coding with a trained network may provide improved performance thanks to the possibility of training such a network on the desired data.

[0296] According to one example, the encoding of the processed picture parts is performed by a variational autoencoder (VAE), which may be advantageous for performing tasks related to the encoding and / or decoding of images and / or pictures very efficiently, with a good compromise between the required bit rate and the quality of the reconstructed image or picture.

[0297] According to one aspect of the present disclosure, a method for reconstructing a picture portion from a bitstream is provided, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the method including: decoding luma and chroma components having the same resolution from the bitstream; determining from the bitstream, as a picture process, one of: upsampling the decoded luma component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled luma component; and downsampling the decoded chroma component to obtain a reconstructed picture portion including the decoded luma component and the downsampled chroma component; and obtaining the picture portion according to the determined picture process.

[0298] Thus, the determined picture processing of one of luma upsampling and chroma downsampling in each case provides resized luma or chroma components having different sizes corresponding to the components input for encoding (original picture components). Thus, the picture part is reconstructed to its original color format, with chroma having a lower resolution than chroma. As a result, the encoded equal-sized luma-chroma channels can be handled within a common VAE framework regardless of the determined picture processing. In general, the advantages provided on the decoding side can be similar to those provided on the encoder side by the corresponding additional features of the respective encoding methods.

[0299] According to an implementation example of the method, the determining step further includes: decoding a selection indicator indicating a picture processing from the bitstream; and determining the processing indicated by the selection indicator as the picture processing. This enables efficient signaling of picture processing. Correspondingly, the decoding side can quickly determine which picture processing was used at the encoder side (i.e., luma downsampling or chroma upsampling) based on a simple indicator included in the bitstream.

[0300] According to another exemplary implementation, the determining step further includes decoding the width W and / or height H of the luma component of the picture part from the bitstream. This allows determining the picture processing based on the width and height of the luma channel of the picture part. In other words, the channel carrying the important information of the picture part is used as a reference for said determination.

[0301] In one example, the determining step includes determining whether the width W and height H are the respective widths of the decoded luma components.

number

number

number

number

number

number

number

number

[0302] According to a further implementation example, the determining step may include determining a size S of the bitstream, and extracting from the bitstream a latent representation of the picture portion.

number

number

number

number

number

number

[0303] In one exemplary implementation, the determining step comprises determining a size S and

number

number

number

[0304] According to another implementation, the determining step comprises:

number

[0305] According to one example implementation, the upsampling and / or downsampling is performed by a trained network. By providing a bitstream based on downsampled luma or upsampled chroma, the decoding side can reconstruct a picture (still or video) at a higher quality and / or lower rate. The trained network can provide adaptive and efficient coding, thus providing improved quality at the same rate or improved rate at the same quality.

[0306] For example, the decoding of the processed picture parts is performed by a variational autoencoder (VAE), so that the decoding can be performed efficiently and with a good compromise between the required bit rate and / or the quality of the reconstructed picture or image.

[0307] According to one aspect of the present disclosure, there is provided a computer program stored on a non-transitory medium comprising code that, when executed on one or more processors, performs the steps of any of the methods described above.

[0308] According to one aspect of the present disclosure, there is provided an encoder for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the encoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the encoder to perform the encoding method described above.

[0309] According to one aspect of the present disclosure, there is provided a decoder for reconstructing a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the decoder comprising: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the decoder to perform the above-mentioned decoding method.

[0310] According to one aspect of the present disclosure, there is provided an encoder for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than the resolution of the luma component, the encoder comprising processing circuitry configured to perform the steps of the encoding method described above.

[0311] According to one aspect of the preset disclosure, there is provided a decoder for reconstructing a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than the resolution of the luma component, the decoder comprising processing circuitry configured to perform the steps of the above-described decoding method.

[0312] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the encoder comprising: a selection module configured to select one of downsampling the luma component to obtain a downsampled luma component that matches the resolution of the chroma component and upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the luma component as a picture processing for the picture portion; a processing module configured to process the picture portion according to the selected picture processing; and a generation module configured to generate a bitstream according to the selected picture processing, where generating includes encoding the processed picture portion.

[0313] According to one aspect of the present disclosure, a decoder for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the decoder comprises: a decoding module configured to decode luma and chroma components having the same resolution from the bitstream; a determination module configured to determine from the bitstream, as a picture processing, one of upsampling the decoded luma component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled luma component, and downsampling the decoded chroma component to obtain a reconstructed picture portion including the decoded luma component and the downsampled chroma component; and an acquisition module configured to acquire the picture portion according to the determined picture processing.

[0314] The embodiments allow adaptively switching between chroma upsampling and luma downsampling of input picture parts depending on the desired application. For example, some applications aim at reducing complexity or bit rate, where luma downsampling may be more advantageous, while other applications aim at higher quality, where chroma upsampling may be more advantageous. Correspondingly, picture parts having luma and subsampled chroma components may be efficiently encoded and / or decoded in any of the sampling modes of picture processing including luma downsampling and chroma upsampling.

[0315] It should be noted that the present disclosure is not limited to any particular framework. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, image generation, and recognition systems.

[0316] The present invention can be implemented in hardware and / or software.

[0317] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0318] In summary, the present disclosure relates to video encoding and decoding, and in particular to handling chroma subsample formats in deep learning-based video (de)coding. The corresponding apparatus and method enable a process for encoding and decoding of a respective picture portion including a luma component and a chroma component having a lower resolution than the luma component. To handle such different sizes of luma-chroma channels, the chroma components are upsampled such that the resulting upsampled chroma components have a resolution matching one of the luma components. The luma and upsampled chroma components are then encoded into a bitstream. To reconstruct the picture portion, the luma component and an intermediate chroma component matching the resolution of the luma component are decoded from the bitstream, followed by downsampling the intermediate chroma components. Thus, the subsampled chroma format can be handled by an autoencoder / autodecoder framework while preserving the luma channel.

[0319] Further, the present disclosure relates to encoding and decoding of pictures or videos. A picture portion includes a luma component and a chroma component having a resolution lower than that of the luma component. A selection is made between a luma component downsampling process and a chroma component upsampling process to obtain a matched resolution of the luma component and the chroma component. The selected process is then applied to the picture portion. A bitstream is then generated based on the selected process, including encoding the processed picture portion. The selection between chroma upsampling and luma downsampling allows for adaptation of the input picture portion for a desired application. For example, some applications aim at reducing complexity or bitrate when luma downsampling may be more advantageous, while other applications aim at higher quality when chroma upsampling may be more advantageous. [Explanation of symbols]

[0320] 410 NN size equalization module 420 VAE Encoder 470 VAE Decoder 480 Downsampling Network 800 equipment 1500 Encoder 1510 Upsampling Module 1520 Encoding Module 1600 Decoder 1610 Decoding Module 1620 Downsampling Module 1710 Luma Downsampling 1720 Chroma Upsampling 1730 Rate Distortion Optimization (RDO) 1740 Function Block "Up or Down" Branching luma upsampling on 1810 1820 Lower Branch Chroma Downsampling 1840 Function Blocks 2910 Luma Downsampling 2920 Chroma Upsampling 2930 Bitrate and Distortion (RDO) 3300 Encoder 3310 Selection Module 3320 Processing Module 3330 Generation Module 3400 Decoder 3410 Decoding Module 3420 Decision Module 3430 Acquisition Module 10. Video Coding System 12 Source Device 13 Communication Channels 14 Destination Device 16 Picture Source 17 Picture Data 18 Preprocessors 19 Preprocessed Picture Data 20 Video Encoder 21 Encoded picture data 22 Communication Interface 28 Communication Interface 30 Video Decoder 31 Decoded picture data 32 Post Processor 33 Post-Processed Picture Data 34 Display Devices 46 Processing Circuit 1000 Video Coding Devices 1010 Inlet port 1020 Receiver Unit 1030 Processor 1040 Transmitter Unit 1050 Exit Port 1060 Memory 1070 Coding Module 1100 equipment 1102 Processor 1104 Memory 1106 Code and Data 1108 Operating Systems 1110 Application Program 1112 Bus 1118 Display

Claims

1. 1. A method for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the method comprising: upsampling the chroma components to obtain upsampled chroma components that match a resolution of the luma component; encoding the luma component and the upsampled chroma components into a bitstream; Including, The method, wherein the encoding step is performed by a variational autoencoder (VAE).

2. The method of claim 1 , wherein the upsampling of the chroma components is based on samples of the luma component.

3. The method of claim 1 or 2, wherein the step of upsampling further comprises processing the chroma components by a trained network.

4. 4. The method of claim 3, wherein the step of processing with a trained network comprises inputting the luma and chroma components to the trained network and outputting the upsampled chroma components from the trained network.

5. The trained network comprises at least two processing layers; A first layer of the at least two processing layers performs upsampling based on the input chroma components; 5. The method of claim 3 or 4, wherein a second layer of the at least two processing layers performs processing of the output of the first layer together with the luma component.

6. 6. The method of claim 5, wherein processing the first layer output together with the luma component comprises element-wise adding the luma component and the upsampled chroma components.

7. 2. The method of claim 1, wherein the upsampling step comprises inserting a line of samples after every n lines of samples of the chroma components, where lines are rows and / or columns and n is an integer greater than or equal to 1.

8. 8. The method of claim 1 or 7, wherein the upsampling step comprises inserting a line of samples having value 0 after every n lines of samples of the chroma components, where the lines are rows and / or columns and n is an integer greater than or equal to 1.

9. 8. The method of claim 1 or 7, wherein the upsampling step comprises inserting, after every n lines of samples of the chroma components, a line of samples having a value equal to the respective nearest neighbor sample in the chroma components, where the lines are rows and / or columns and n is an integer greater than or equal to 1.

10. 1. A method for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the method comprising: decoding, from the bitstream, the luma component and intermediate chroma components that match the resolution of the luma component; downsampling the intermediate chroma components to obtain the chroma components of the picture portion; Including, The method, wherein the decoding step is performed by a variational autoencoder (VAE).

11. The method of claim 10 , wherein the step of downsampling further comprises processing the intermediate chroma components by a trained network.

12. The method of claim 11 , wherein the step of processing with the trained network includes inputting the intermediate chroma components to the trained network and outputting the chroma components from the trained network.

13. The trained network comprises at least two processing layers; a first layer of the at least two processing layers for processing the input intermediate chroma components; 13. The method of claim 11 or 12, wherein a second layer of the at least two processing layers downsamples the output of the first layer.

14. The method of claim 10 , wherein the downsampling of the intermediate chroma components is based on samples of the luma component.

15. 15. The method of claim 11, wherein the step of processing with the trained network comprises inputting the luma component to the trained network.

16. 16. The method of claim 13, wherein the first layer performs processing of the intermediate chroma components together with the luma component.

17. 11. The method of claim 10, wherein the downsampling step comprises removing a line of samples after every n lines of samples of the intermediate chroma components, where the lines are rows and / or columns and n is an integer greater than or equal to 1.

18. A computer program stored on a non-transitory medium comprising code which, when executed on one or more processors, performs the method according to any one of claims 1 to 17.

19. 1. An encoder for encoding a picture portion, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the encoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the encoder to perform the method of any one of claims 1 to 9; and An encoder comprising:

20. 1. A decoder for decoding a picture portion from a bitstream, the picture portion including a luma component and a chroma component having a resolution lower than a resolution of the luma component, the decoder comprising: one or more processors; a non-transitory computer-readable storage medium coupled to the one or more processors and storing programming for execution by the one or more processors, the programming, when executed by the one or more processors, configuring the decoder to perform the method of any one of claims 10 to 17; and A decoder comprising:

Images