A method for handling chroma subsampling formats in machine learning-based picture coding.

Abstract

To provide video coding, an encoder, and a decoder that further improve efficiency on the basis of a trained network.SOLUTION: In order to handle different sized luma-chroma channels, a chroma component is up-sampled such that an obtained up-sampled chroma component has a resolution matching one of a luma component. The luma and the up-sampled chroma component are then encoded into a bitstream. To reconstruct the picture portion, the luma component and an intermediate chroma component matching the resolution of the luma component are decoded from the bitstream, and subsequently, the intermediate chroma component is down-sampled. A sub-sampled chroma format is handled by an auto-encoder / auto-decoder framework, while preserving the luma channel.SELECTED DRAWING: Figure 13

Description

[Technical Field]

[0001] Embodiments of this disclosure generally relate to the field of encoding and decoding pictures or videos, and more particularly to the handling of color formats having subsampled chroma in video coding based on a trained network. [Background technology]

[0002] Video coding (video encoding and decoding) is used in a wide range of digital video applications, such as broadcast digital television, video transmission over the internet and mobile networks, real-time communication applications like video chat, video conferencing, DVD and Blu-ray discs, video content acquisition and editing systems, and camcorders for security applications.

[0003] Even relatively short videos can have a considerable amount of video data required to depict them, which can cause difficulties when streaming over communication networks with limited bandwidth or when the data is transmitted in other ways. Therefore, video data is generally compressed before being transmitted over today's telecommunications networks. Video size can also be a problem when the video is stored on a storage device, as memory resources can be limited. Video compression devices often use software and / or hardware at the source to encode the video data before transmission or storage, thereby reducing the amount of data required to represent the digital video image. The compressed data is then received at the destination by a video decompression device that decodes the video data. Given limited network resources and the constantly increasing demand for higher video quality, improved compression and decompression techniques that improve the compression ratio with little to no sacrifice of picture quality are desirable.

[0004] In recent years, machine learning (ML) has been applied to image and video coding. Generally, machine learning can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding schemes have been discussed. More broadly, machine learning is a subset of artificial intelligence, where a model, generally a network model, is built, often on sample data called training data. Once trained, such ML-based techniques then make predictions or decisions based on input data that might not have been used as training data. This means that after training, the ML network is trained and is called a trained network.

[0005] Numerous network models exist that are based on ML techniques, some examples of which include support vector machines (SVMs), neural networks (NNs), artificial neural networks (ANNs), fully connected neural networks (FCNs), and convolutional neural networks (CNNs). These types of networks often share the common characteristic that their network structure involves multiple layers of network nodes (i.e., at least one layer). This allows some or all of the nodes in one layer to be connected to some of all the nodes in one or more layers preceding or following that layer.

[0006] It is desirable to further improve the efficiency of such video coding (video coding or still picture coding) based on trained networks. [Overview of the project] [Means for solving the problem]

[0007] Some embodiments of this disclosure provide methods and apparatus for encoding and decoding picture portions having a lumen component and a downsampled chroma component as input. Correspondingly, the chroma component is upsampled to match the lumen component, and encoding by a trained network is applied to the component having a matching size. This can provide more efficient encoding and decoding since a trained network with a similar design (input dimension, resolution) can be applied. Spatial correlation between the lumen and chroma can also be preserved in this manner.

[0008] The aforementioned and other objectives are achieved by the subject matter of the independent claims. Further implementations will become apparent from the dependent claims, this specification, and the figures.

[0009] According to one aspect of the present disclosure, a method is provided for encoding a picture portion, the picture portion comprising a rumar component and a chroma component having a resolution lower than that of the rumar component, the method comprising the steps of: upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the rumar component; and encoding the rumar component and the upsampled chroma component into a bitstream.

[0010] This provides a lumar-chroma output of equal size as output. Thus, the output can then be processed by a trained network (e.g., within a machine learning framework) with a similar structure, particularly one with the same input resolution. If all color components have the same resolution, the machine learning framework can utilize cross-color correlation in the same way at all sample locations. In this process, the lumar channel may remain unaffected, in which case there is no loss of quality.

[0011] In some exemplary implementations, chroma component upsampling is based on ruma component sampling. This can enable upsampling based on the content of the ruma component.

[0012] According to an exemplary implementation of the method, the upsampling step further includes processing the chroma components with a trained network. Coding with a trained network can offer improved performance thanks to the possibility of training such a network with the desired data.

[0013] In one implementation example, the step of processing by the trained network includes inputting the lumern and chroma components into the trained network and outputting the upsampled chroma component from the trained network. Using the lumern component as input to this trained network allows for the preservation of inter-component information between the lumern and chroma channels.

[0014] For example, the trained network comprises at least two processing layers, the first of which performs upsampling based on the input chroma component, and the second of which processes the output of the first layer together with the luma component. The combined processing can enable improved utilization of the correlation between luma and chroma and allow for a common network design for multiple components.

[0015] In one implementation example of the method, processing the output of the first layer together with the lumern component involves adding the lumern component and the upsampled chroma component element by element. Therefore, since the lumern contains more information, for example, about the shape of an object, chroma upsampling using the lumern as side information can be performed efficiently. Moreover, details that may have been present in the lumern, which originally had a higher resolution, can be restored in the originally subsampled chroma.

[0016] In one example, the step of upsampling includes the step of inserting a line of samples after every n lines of samples of the chroma component, where the line is a row and / or a column and n is an integer greater than or equal to 1. This exemplary implementation enables low-complexity and effective upsampling insertion every two rows or every two columns with a given value.

[0017] According to one example, the step of upsampling includes the step of inserting a line of samples having a value of 0 after every n lines of samples of the chroma component, where the line is a row and / or a column and n is an integer greater than or equal to 1. This is a simple way of performing upsampling by simply setting each line sample to 0.

[0018] According to a further implementation example of the method, the step of upsampling includes the step of inserting a line of samples having a value equal to each nearest neighbor sample within the chroma component after every n lines of samples of the chroma component, where the line is a row and / or a column and n is an integer greater than or equal to 1. This allows for further adaptation of chroma upsampling by considering the content of adjacent samples in a particular row and / or column. Further, by using the value of the inserted line sample that references the nearest neighbor sample of the chroma, the accuracy of chroma upsampling can be enhanced because the adjacent information (reflected by the neighboring chroma sample values) referencing the original chroma is used.

[0019] According to one example, the encoding is performed by a variational autoencoder (VAE). VAE can be advantageous for efficiently performing tasks related to encoding and decoding of images (pictures), and has a good compromise between the required bitrate and the quality of the reconstructed image / picture.

[0020] According to one aspect of the present disclosure, a method for decoding a picture portion from a bitstream is provided, the picture portion including a luma component and a chroma component having a resolution lower than that of the luma component, the method comprising the steps of decoding the luma component and an intermediate chroma component that matches the resolution of the chroma component from the bitstream, and downsampling the intermediate chroma component to obtain the chroma component of the picture portion. Thus, the original chroma component (i.e., subsampled chroma) can be reconstructed from the intermediate chroma component. In particular, a picture portion having the original luma-chroma format can be reconstructed with high quality. Thus, the decoder can provide the reconstructed picture portion of the chroma format as an output, where the chroma has a resolution (i.e., size) lower than that of the luma. Further, the luma information and spatial information of Y are mostly preserved since the luma is not downsampled. Generally, the advantages provided on the decoding side can be similar to the advantages provided on the encoder side by the corresponding additional features of each encoding method.

[0021] According to one exemplary implementation, the downsampling further comprises the step of processing the intermediate chroma component by a trained network. By providing a bitstream based on the upsampled chroma, the decoding side can reconstruct a picture (still or video picture) with higher quality and / or at a lower rate. The trained network provides adaptive and efficient coding, and thus can provide improved quality at the same rate or an improved rate at the same quality.

[0022] [[ID=,7]] For example, the steps performed by the trained network include inputting intermediate chroma components into the trained network and outputting chroma components from the trained network. Therefore, the trained network may be designed with similar resolution in the input of luma and chroma, thus reducing complexity and providing low processing overhead. For example, in some embodiments, it may not be necessary to use luma as side information for reconstructing chroma.

[0023] According to one implementation example of the method, the trained network comprises at least two processing layers, the first of which processes the input intermediate chroma components, and the second of which downsamples the output of the first layer. This exemplary network architecture may allow the original resolution to be obtained based on a bitstream generated using the upsampled chroma.

[0024] In another example, downsampling of the intermediate chroma component is based on the lumen component sample. Therefore, downsampling of the chroma channel may take into account the contents of the lumen channel.

[0025] According to one implementation example of the method, the step of processing by a trained network includes the step of inputting the luma components into the trained network.

[0026] For example, the first layer performs a step of processing the intermediate chromatic components together with the chroma components. Thus, inter-component information between the chroma channel and (one or more) chromatic channels is preserved.

[0027] In a further example of the method, the downsampling step includes removing the sample lines after every n lines of the intermediate chroma component, where the lines are rows and / or columns and n is an integer greater than or equal to 1. Thus, downsampling of the intermediate chroma can be performed quickly by simply removing each line sample.

[0028] For example, decoding is performed by a variational autoencoder (VAE). Therefore, decoding can be performed efficiently and has a good compromise between the required bitrate and / or the quality of the reconstructed picture.

[0029] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion comprising a rumor component and a chroma component having a resolution lower than that of the rumor component, and the encoder comprises a processing circuit configured to perform any step of the encoding method described above or herein.

[0030] According to one aspect of a pre-configured disclosure, a decoder is provided for decoding a picture portion from a bitstream, wherein the picture portion comprises a rumor component and a chroma component having a resolution lower than that of the rumor component, and the decoder comprises a processing circuit configured to perform any step of the decoding method described above and herein.

[0031] According to one aspect of this disclosure, a computer program stored on a non-temporary medium is provided which, when executed on one or more processors, includes code that performs any of the steps described above and herein.

[0032] According to one aspect of the present disclosure, an encoder is provided for encoding a picture portion, the picture portion comprising a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, the encoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program is configured to execute any of the encoding methods described above and herein when executed by the one or more processors.

[0033] According to one aspect of the present disclosure, a decoder is provided for decoding a picture portion from a bitstream, wherein the picture portion comprises a rumor component and a chroma component having a resolution lower than that of the rumor component, and the decoder comprises one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the program is configured to cause the decoder to perform any of the decoding methods described above and herein when executed by the one or more processors.

[0034] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion comprising a rumar component and a chroma component having a resolution lower than that of the rumar component, the encoder comprising an upsampling module configured to upsample the chroma component to obtain an upsampled chroma component that matches the resolution of the rumar component, and an encoding module configured to encode the rumar component and the upsampled chroma component into a bitstream.

[0035] According to one aspect of the present disclosure, a decoder is provided for decoding a picture portion from a bitstream, wherein the picture portion comprises a rumor component and a chroma component having a resolution lower than that of the rumor component, and the decoder comprises a decoding module configured to decode the rumor component and an intermediate chroma component matching the resolution of the chroma component from the bitstream, and a downsampling module configured to downsample the intermediate chroma component to obtain the chroma component of the picture portion.

[0036] The embodiment provides efficient encoding and / or decoding of a picture portion having a rumor component and an upsampled chroma component. In other words, the picture portion to be encoded and / or decoded may have different rumor-chroma formats, i.e., rumor channels and chroma channels of different sizes.

[0037] Please note that this disclosure is not limited to any particular framework. Furthermore, this disclosure is not limited to image or video compression, but may also apply to object detection, image generation, and recognition systems.

[0038] The present invention can be implemented in hardware and / or software.

[0039] For clarity, any one of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of this disclosure.

[0040] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims.

[0041] The following embodiments of the present invention will be described in more detail with reference to the accompanying figures and drawings. [Brief explanation of the drawing]

[0042] [Figure 1] (a) Examples of rumor-chroma YUV formats including YUV444, (b) YUV422, and YUV420 are shown. [Figure 2A] This is a schematic diagram illustrating the input and output of artificial neural network layer processing. [Figure 2B] This is a block diagram illustrating an example of an autoencoder. [Figure 2C] This is a schematic diagram illustrating a variational autoencoder architecture that includes a super-prior model. [Figure 3] This diagram shows a schematic representation of the VAE structure, including an ultra-prior model in which the luma component is downsampled. [Figure 4] A schematic diagram of an encoder-decoder architecture based on the VAE of this disclosure, which uses the lumen component as side information to perform upsampling and downsampling of the chroma channel, is shown. [Figure 5] This diagram shows a schematic of a size equalization network for chroma upsampling on the encoder side of a VAE architecture, using the lumen component as side information. [Figure 6] This diagram shows a schematic of the size transformation network for (intermediate) chroma downsampling on the decoder side of a VAE architecture, using the chroma component as side information. [Figure 7] This diagram shows a schematic of a size equalization network for chroma upsampling on the encoder side of a VAE architecture, where the lumen component is not used as side information. [Figure 8] This diagram shows a schematic of the size transformation network for (intermediate) chroma downsampling on the decoder side of a VAE architecture, where the chroma component is not used as side information. [Figure 9] This shows an example of chroma upsampling by inserting zeros in odd-numbered rows and columns. [Figure 10]This shows an example of chroma downsampling by removing zeros in odd-numbered rows and columns. [Figure 11] This shows another example of chroma upsampling by inserting zeros in odd rows and odd columns that have equal values ​​to the nearest neighbor sample in each chroma component. [Figure 12] This shows another example of chroma downsampling by removing sample values ​​in odd rows and odd columns corresponding to the nearest neighbor sample in the chroma component. [Figure 13] The flowchart shows an encoding method that includes the step of upsampling a subsampled chroma to an intermediate chroma having the same size as the rumor. [Figure 14] A flowchart of a decoding method is shown, which includes the steps of decoding intermediate chroma from a bitstream and downsampling the intermediate chroma to subsampled chroma having a different size from the chroma. [Figure 15] This shows a block diagram of an encoder that includes an upsampling module and an encoding module. [Figure 16] This shows a block diagram of a decoder, including a decoding module and a downsampling module. [Figure 17] A schematic diagram shows an encoder that performs lumen downsampling (upper branch) and lumen upsampling (lower branch) without using (one or more) chroma components to determine the cost based on rate distortion optimization (RDO) and its signaling in the bitstream. [Figure 18] This diagram shows a schematic of a decoder that, depending on the parsed indicators from the encoded bitstream, performs either lumern upsampling without using (one or more) chroma components (upper branch) or lumern downsampling without using lumern components (lower branch). [Figure 19]This diagram shows a schematic of a size equalization network for encoder-side lumens downsampling in a VAE architecture, where (one or more) chroma components are not used as side information. [Figure 20] This diagram shows a schematic of a size equalization network for chroma upsampling on the encoder side of a VAE architecture, where the lumen component is not used as side information. [Figure 21] This diagram shows a schematic of the size transformation network for (intermediate) lumen upsampling on the decoder side of a VAE architecture, where (one or more) chroma components are not used as side information. [Figure 22] This diagram shows a schematic of a size transformation network for (one or more) (intermediate) chroma downsampling on the decoder side of a VAE architecture, where the chroma component is not used as side information. [Figure 23] A schematic diagram shows an encoder that performs lumen downsampling (upper branch) using (one or more) chroma components and lumen upsampling (lower branch) using lumen components to determine the cost based on rate distortion optimization (RDO) and its signaling in the bitstream. [Figure 24] This diagram shows a schematic of a decoder that, depending on the parsed indicators from the encoded bitstream, performs either lumar upsampling using (one or more) lumar components (upper branch) or lumar downsampling using lumar components (lower branch). [Figure 25] This diagram shows a schematic of a size equalization network for encoder-side lumens downsampling in a VAE architecture, using (one or more) chroma components as side information. [Figure 26] This diagram shows a schematic of a size equalization network for chroma upsampling on the encoder side of a VAE architecture, using the lumen component as side information. [Figure 27]This diagram shows a schematic representation of a size transformation network for (intermediate) lumens upsampling on the decoder side of a VAE architecture, using (one or more) chroma components as side information inputs to the network. [Figure 28] This diagram shows a schematic representation of a size transformation network for (one or more) (intermediate) chroma downsampling on the decoder side of a VAE architecture, using the chroma component as a side information input to the network. [Figure 29] A schematic diagram shows an encoder that performs lumen downsampling (upper branch) and lumen upsampling (lower branch) without using (one or more) chroma components to determine cost based on rate distortion optimization (RDO) without indicator signaling in the bitstream. [Figure 30] This diagram shows a schematic of a decoder that, depending on the result of a decision module regarding the sampling mode, performs either lumern upsampling without using (one or more) chroma components (upper branch) or lumern downsampling without using lumern components (lower branch). [Figure 31] The flowchart shows an encoding method that includes the steps of selecting one of the picture processing methods (luma downsampling or chroma upsampling) based on the lowest cost, and generating a bitstream from the processed picture portion. [Figure 32] The flowchart shows a decoding method that includes the steps of decoding the rumor-chroma from the bitstream and determining one of the picture processing steps (rumor upsampling or chroma downsampling) to reconstruct the original picture portion from the processed picture portion. [Figure 33] The diagram shows a block diagram of an encoder including a selection module, a processing module, and a generation module. The selection module may further include a downsampler (lumer) and an upsampler (chromer). [Figure 34]The diagram shows a block diagram of a decoder including a decoding module, a decision module, and an acquisition module. The decision module may further include a downsampler (chroma) and an upsampler (luma). [Figure 35] This is a block diagram showing an example of a video coding system configured to implement embodiments of the present invention. [Figure 36] This is a block diagram showing another example of a video coding system configured to implement embodiments of the present invention. [Figure 37] This is a block diagram showing an example of an encoding or decoding device. [Figure 38] This is a block diagram showing another example of an encoding or decoding device. [Modes for carrying out the invention]

[0043] The following description refers to the accompanying drawings, which form part of this disclosure and, as an example, illustrate specific embodiments of the present invention or specific ways in which embodiments of the present invention may be used. It is understood that embodiments of the present invention may be used in other ways, and may include structural or logical modifications not depicted in the drawings. Therefore, the following detailed description should not be construed as restrictive, and the scope of the invention is defined by the appended claims.

[0044] For example, disclosure relating to a described method may also apply to a corresponding device or system configured to perform that method, and vice versa. For example, if one or more steps of a particular method are described, a corresponding device may include one or more units, e.g., functional units (e.g., one unit performing one or more steps, or multiple units each performing one or more of the steps), even if such one or more units are not explicitly shown or illustrated in the drawings. On the other hand, for example, if a particular device is described based on one or more units, e.g., functional units, a corresponding method may include one step for performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more functions of the units), even if such one or more steps are not explicitly shown or illustrated in the drawings. Furthermore, unless otherwise specified, the various exemplary embodiments and / or features of the aspects described herein may be combined with each other.

[0045] The following provides a brief overview of some of the technical terms used throughout this disclosure.

[0046] Video Sequence Color Formats: A color format describes a specific arrangement of colors within a video sequence. Typically, a color format is expressed as a mathematical model describing how colors can be represented, for example, as a tuple of numbers. For instance, the RGB format describes a frame (picture or image) being combined with red (R), green (G), and blue (B) color components. In contrast, the YUV format describes a frame being combined with one lumen component (Y) and two chromen components (U,V). Thereafter, the lumen refers to the luminance component of a sample, pixel, or group of samples / pixels, such as a picture or a portion of a picture. Generally, a color format can have one or more color components.

[0047] RGB: The RGB color format is an additive color model that includes red (R), green (G), and blue (B) components. A picture in the RGB color space typically has components that are the same size in terms of width and height, such as the width and height of the frame, which correspond to the resolution, depending on the number of samples or pixels. These RGB components can be added together in various ways to represent a wide range of colors.

[0048] YUV: The YUV color format contains three components: the lumen component Y represents brightness information (i.e., luminance), and the chromen components U (blue projection) and V (red projection) represent chrominance or color information. In digital encoding of color information, YUV can be directly converted from RGB using a linear matrix, as illustrated below.

number

[0049] In other words, each YUV component may be written as a linear combination of the RGB components. Therefore, after such a transformation, the three components of YUV will also have the same size (width and height).

[0050] Chroma subsampling: Chroma subsampling is an image encoding technique that implements a lower resolution for chroma information (U,V) than for lumen information (Y). The reason for subsampling the chroma component is to take advantage of the human visual system's lower visual acuity for color difference than for luminance. Therefore, as far as the lumen visual system is concerned, the lumen component can be considered the most important channel when using the YUV (or YCbCr) format.

[0051] Chroma subsampling can significantly reduce the chroma component signal. Commonly used chroma subsampling formats are briefly described below. YUV4:4:4 (or abbreviated as YUV444) refers to the absence of chromatic subsampling, meaning that the Y, U, and V components have the same size (width and height), as shown in Figure 1(a). YUV4:2:2 (or abbreviated as YUV422) refers to subsampling the chroma components (U and V) horizontally by a factor of 2, as shown in Figure 1(b). For example, if the lumen component has size w × h, where w represents the width of the lumen component and h represents the height of the lumen component, then the chroma components U and V in YUV4:2:2 each have size w / 2 × h (e.g., half the width, i.e., w / 2, with respect to the number of samples or pixels, and the same height h as the lumen component). In this case, each chromen component has a rectangular shape. YUV4:2:0 (or abbreviated as YUV420) refers to subsampling the chroma components (U and V) by a factor of 2 in both the horizontal and vertical directions, as shown in Figure 1(c). For example, if the lumen component has size w × h, where w represents the weight of the lumen component and h represents the height of the lumen component, then the chroma components in YUV4:2:0 each have size w / 2 × h / 2 (for example, half the width of the lumen component, i.e., w / 2, and half the height, i.e., h / 2, with respect to the number of samples or pixels). In this case, each chroma component has a square shape.

[0052] It should be noted that YUV4:2:0 chroma subsampling is the most commonly used chroma subsampling method in video coding. Subsampling can generally be a result of the configuration of the image sensor capturing the picture. For example, some sensors provide more pixels to capture certain color components. However, subsampling can also be a result of further processing of the captured picture.

[0053] The YUV format described above is used in this disclosure as an example of lumar-chroma components (channels), where each chroma has a different resolution (i.e., a different size) as a lumar channel. It should also be noted that, generally, only one of the two chroma components may be subsampled, while the other maintains its size. Furthermore, instead of subsampling the chroma components only horizontally, as shown in Figure 1(b), the (one or more) chroma components may instead be subsampled vertically.

[0054] Artificial neural networks Artificial neural networks (ANNs), or connectionist systems, are computing systems that draw loose inspiration from the biological neural networks that make up animal brains. Such systems "learn" to perform tasks by considering examples and are generally not programmed with task-specific rules. For example, in image recognition, such a system may learn to identify images containing cats by analyzing exemplary images that have been manually labeled as "cat" or "not a cat," and using the results to identify cats in other images. Such a system does this without prior knowledge of cats, for example, that cats have fur, tails, whiskers, and cat-like faces. Instead, such systems automatically generate discriminative characteristics from the examples they process.

[0055] ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in the brain of living organisms. Each connection can transmit signals to other neurons, just like synapses in the brain of living organisms. The artificial neuron that receives the signal can then process the signal and signal to the neurons connected to it.

[0056] In ANN implementations, the "signals" in a connection are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have thresholds such that they only transmit a signal if the aggregated signal exceeds that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. The signal travels from the first layer (input layer) to the last layer (output layer), sometimes traversing the layers multiple times.

[0057] The initial goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention shifted to performing specific tasks, leading to deviations from biology. ANNs have been used for a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, board games and video games, medical diagnosis, and even activities traditionally considered uniquely human, such as drawing.

[0058] Convolutional Neural Network The name "Convolutional Neural Network" (CNN) indicates that this network uses a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is simply a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0059] Figure 2A schematically illustrates the general concept of processing by neural networks such as CNNs. A convolutional neural network consists of an input layer, an output layer, and several hidden layers. The input layer is the layer that receives the input for processing (such as a portion of an image, as shown in Figure 2A). The hidden layers of a CNN typically consist of a series of convolutional layers that convolve by multiplication or other dot products. The result of the layers is one or more feature maps (f.maps in Figure 2A), sometimes called channels. Subsampling may exist that involves some or all of the layers. As a result, the feature maps may be small, as shown in Figure 2A. The activation function in a CNN is usually a RELU (Normalized Linear Unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, the inputs and outputs of which are called hidden layers because they are masked by the activation function and the final convolution. The layers are colloquially called convolutions, but this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for indices in a matrix in that it affects how weights are determined at specific index points.

[0060] When programming a CNN for image processing, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth), as shown in Figure 2A. After passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network must have the following attributes: a convolution kernel (hyperparameter) defined by width and height; the number of input and output channels (hyperparameters); and the depth of the convolutional filter (input channels) must be equal to the number of channels (depth) in the input feature map.

[0061] In the past, conventional multilayer perceptron (MLP) models have been used for image recognition. MLPs are also known as fully connected neural networks (FCNs). However, due to the fully connected nature of the nodes, MLP models suffer from high dimensionality and do not handle higher resolution images well. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently and conveniently processed on a large scale with fully connected networks. Therefore, MLPs / FCNs are often prone to overfitting data. Because each node (perceptron) in a layer takes the output of all nodes in adjacent layers as input, the network architecture of an FCN does not take into account the spatial structure of the data (e.g., across layer nodes), treating distant input pixels as if they were adjacent pixels. This ignores locality of reference in image data, both computationally and semantically. Therefore, the fully connected nature of neurons is redundant for purposes such as image recognition where spatially local input patterns are dominant.

[0062] Convolutional neural networks (CNNs) are a biologically inspired variation of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by leveraging the strong spatially local correlations present in natural images. Convolutional layers are the core building blocks of a CNN. The layer parameters consist of a set of learnable filters (kernels, as described above), which have small receptive fields but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, calculating the dot product between the filter's entry and the input to generate a two-dimensional activation map of that filter. As a result, the network learns which filters are activated when it detects a particular type of feature at a given spatial location in the input.

[0063] Stacking the activation maps of all filters along the depth dimension forms the entire output volume of the convolutional layer. Thus, all entries in the output volume can also be interpreted as the outputs of neurons that look at a small region in the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation are synonymous. In some papers, it is called an activation map because it is a mapping that corresponds to the activation of different parts of an image, and also a feature map because it is a mapping of where certain types of features are found in the image. High activation means that a particular feature has been found.

[0064] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions for implementing pooling, of which maximum pooling is the most common. Maximum pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0065] Intuitively, the precise location of a feature is less important than its approximate location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers help progressively reduce the spatial size of representations, lowering the number of parameters in the network, memory footprint, and computational complexity, and thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0066] A pooling layer operates independently for each depth slice of the input, spatially resizing it. The most common form is a pooling layer with a 2x2 filter, applied with a step size of 2 downsamples for each depth slice of the input, both along the width and height, discarding 75% of the activations. In this case, all maximization operations span four numbers. The depth dimension remains invariant.

[0067] In addition to maximum pooling, pooling units can also use other functions such as mean pooling and L2 norm pooling. Mean pooling has historically been widely used, but recently it has become less preferred compared to maximum pooling, which actually works better. There is a recent trend to use smaller filters or to discard the pooling layer altogether for aggressive reduction of the size of representation. Region of interest pooling (also known as ROI pooling) is a variation of maximum pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on fast R-CNN architectures.

[0068] The above ReLU stands for Normalized Linear Unit, which applies a non-saturated activation function. ReLU effectively removes negative values ​​from the activation map by setting negative values ​​to 0. ReLU increases the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions, such as the saturated hyperbolic tangent and sigmoid function, are also used to increase nonlinearity. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly impairing generalized accuracy.

[0069] After several convolutional and maximum pooling layers, high-level inference in the neural network is performed via fully connected layers. The neurons in the fully connected layers are connected to all activations of the previous layer, as seen in typical (non-convolutional) artificial neural networks. Thus, their activations can be computed as affine transformations, followed by matrix multiplication and then bias offsets (vector addition of learned or fixed bias terms).

[0070] The "loss layer" specifies how training penalizes the deviation between the predicted (output) label and the true label, and is usually the final layer of a neural network. Various loss functions can be used that are suitable for different tasks. Softmax loss is used to predict a single class from K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. Euclidean loss is used to regress to real-valued labels.

[0071] Autoencoders and unsupervised learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram of it is shown in Figure 2B. The purpose of an autoencoder is to learn a representation (encoding) of a dataset, typically for dimensionality reduction, by training the network to ignore signal "noise". Along with the reduction side, the reconstruction side is learned, and the autoencoder attempts to generate a representation from the reduced encoding that is as close as possible to its original input, and thus its name. In its simplest case, given one hidden layer, the encoder stage of the autoencoder takes an input x and maps it to h, h = σ(Wx + b) That is the case.

[0072] This image h is typically called the code, latent variable, or latent representation. Here, σ is an element-wise activation function such as a sigmoid function or a normalized linear unit. W is the weight matrix, and b is the bias vector. The weights and biases are typically initialized randomly and then iteratively updated during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstructed x' with the same shape as x: x'=σ'(W ’ h'+b') In the equation, σ', W', and b' of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0073] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use variational methods for latent representation learning, resulting in additional loss components and specific estimators for training algorithms called stochastic gradient variational Bayes (SGVB) estimators. The data is applied to a directed graphical model p θ (x|h) is generated, and the encoder is the posterior distribution p θ Approximation q for (x|h) φ Assuming that (h|x) is being learned, φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vector of a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder. The objective of a VAE has the following form:

number

[0074] In the formula, D KL This represents the Kullback-Leibler divergence. The prior distribution for the latent variables is usually a central isotropic multivariate Gaussian distribution p θ (h) is set to N(0,I). In general, the shapes of the variational and likelihood distributions are chosen so that they are factored Gaussian distributions: q φ (h|x)=N(ρ(x),ω 2(x)Ι) p φ (x|h)=N(μ(h),σ 2 (h)Ι) In the formula, ρ(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0075] Recent advances in the field of artificial neural networks, particularly in convolutional neural networks, have enabled researchers to apply neural network-based techniques to the tasks of image and video compression. For example, end-to-end optimized image compression using a network based on variational autoencoders has been proposed. Correspondingly, data compression is considered a fundamental and well-studied problem in engineering, generally formulated with the objective of designing a code for a given discrete data ensemble with minimum entropy. This solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all actual codes must have a finite entropy, continuous-value data (such as a vector of image pixel intensity) must be quantized into a finite set of discrete values, which introduces errors. In this context, a trade-off must be made between two competing costs: the entropy (rate) of the discretized representation and the errors (distortion) resulting from quantization, known as the lossy compression problem. Different compression applications, such as data storage and transmission over channels with limited capacity, require different rate-distortion trade-offs. Simultaneous optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is unwieldy. For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-value representation, independently quantizing its elements, and then encoding the resulting discrete representation using a reversible entropy code. This method is called transform coding due to the central role of the transformation. For example, JPEG uses a discrete cosine transform for blocks of pixels, and JPEG2000 uses multiscale orthogonal wavelet decomposition. Typically, the three components of a transform coding method—transformation, quantization, and entropy code—are optimized separately (often by manual parameter adjustment). Modern video compression standards such as HEVC, VVC, and EVC also use the transformed representation to code the residual signal after prediction.Several transformations are used for this purpose, including the discrete cosine transform and discrete sine transform (DCT, DST), as well as the low-frequency inseparable manual optimization transform (LFNST).

[0076] Of the types of networks mentioned above, CNNs have been shown to be advantageous for video / image (de)coding and are implemented in variational autoencoders (VAEs), as discussed by J. Balle in "Variational image compression with a scale hyperior" (arXiv preprint arXiv:1802.01436(2018)). Hereafter, this document will be referred to as "J. Balle".

[0077] The VAE structure is designed for deep learning-based image compression and constitutes a highly efficient structure. Since the color format of images input to a VAE is typically RGB without subsampling, the three components / channels have the same size (width and height). Therefore, RGB color format images provided as input to a VAE structure can be processed and reconstructed directly.

[0078] J. Balle describes an end-to-end trainable model for image compression based on variational autoencoders (VAEs). This model incorporates a superprior distribution to effectively capture spatial dependencies in the latent representation. This superprior distribution relates to side information also sent to the decoding side and is a concept common to virtually all modern image codecs, but has been rarely considered in image compression using ANNs. Unlike existing autoencoder compression methods, this model trains a complex prior distribution together with the underlying autoencoder.

[0079] The VAE framework can be considered a nonlinear transformation coding model. The transformation process can be divided into four main parts, which are illustrated in Figure 2C illustrating the VAE framework.

[0080] Figure 2C shows the network architecture including the ultra-prior model. Left side (g a ,g s ) shows the Image Autoencoder Architecture (VAE), and on the right (h a ,h s ) corresponds to an autoencoder that implements a superprior distribution. The factored prior model is analyzed and composited g a and g s The same architecture is used. Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively.

[0081] The encoder converts the input image x to g a This process induces a response y (latent representation) with a spatially varying standard deviation. Encoding g a It includes multiple convolutional layers with subsampling and a generalized partition normalization (GDN) as the activation function. Specifically, the encoder g a (1) maps the image x to the latent representation y via the function y=f(x). The latent representation is discrete values

number

[0082] The response is h a It is supplied to summarize the distribution of the standard deviation at z. z is then quantized, compressed, and transmitted as side information. Entropy model or hyperencoder / decoder h a (3) (also known as the superprior distribution) is used to obtain the minimum rate achievable with reversible entropy source coding.

number

number

number

number

number

number

number

[0083] The decoder side of the VAE structure shown in Figure 2C, i.e., g s and h s First, the decoder uses an arithmetic decoder to process the bitstream.

number

number

number

number

number

number

number

number

number

[0084] Most deep learning-based image / video compression systems reduce the dimensionality of a signal before converting it to binary (bits). For example, in the VAE framework, the encoder, which is a nonlinear transformation, maps the input image x to y, where y has smaller width and height than x. Since y has smaller width and height, and therefore smaller size, the dimensionality of the signal is reduced. Thus, it is easier to compress the signal y. The general principle of compression is illustrated in Figure 2C. Encoder g a Output and decoder g s The latent space is the input.

number

[0085] The reduction in the size of the input signal is illustrated in Figure 2C, which represents a deep learning-based encoder and decoder. As mentioned above, the input image x corresponds to the input data, which is the input to the encoder shown in Figure 2C. The transformed signal y is in the latent space.

number

[0086] As is clear from Figure 2C, the encoding operation corresponds to reducing the size of the input signal, and the decoding operation corresponds to reconstructing the image to its original size.

[0087] The VAE structure uses one convolutional network layer (conv N×5×5 / 2↓) as input. Note that "N×5×5 / 2↓" here refers to the notation for specifying the CNN layer. In this example, a CNN with a 5×5 convolutional layer is defined to have N channels as output. The output is downsampled by a coefficient of 2 in both dimensions, as indicated by the down arrow. For this convolutional layer, all three components of the input image are the same size (e.g., RGB or YUV444). Otherwise, the convolutional network layer in this example will not take images (or picture portions) with different sizes as input, such as in the case of YUV422 or YUV420 formats.

[0088] Similarly, as shown in Figure 2C, due to the final deconvolutional network layer (conv 3×5×5 / 2↑) at the end of the VAE structure, the three components of the output image (reconstructed image) are the same size.

[0089] Therefore, there are problems when compressing frames in a video sequence using existing VAE structures. For example, the VAE structure may not directly handle video frames in chroma subsampling color formats (e.g., YUV422, YUV420) because the sizes of the three components are not the same. As mentioned above, in some cases the input image may not be processed by the input convolutional network layer, and as a result the output image may not have the same size as the input image.

[0090] The MPEG proposal (Ankitesh K. Singh et al., "M55267:[DNNVC] A study of handling YUV420 input format for DNN-based video coding") discusses how to address some of the above problems for specific cases of YUV420 color format frames in video sequences.

[0091] Figure 3 illustrates one method of the MPEG proposal. On the encoder side, the lumern component is downsampled by a factor of 2 in both the horizontal and vertical directions using a convolutional network layer conv N×5×5 / 2↓ (i.e., the convolutional kernel is 5×5, the output channels are N, and the output is downsampled by 2 in both dimensions). The chromern component is processed by a convolutional network layer conv N×5×5 / 1↓ without downsampling (in CNN notation, this is indicated by " / 1", i.e., division by 1). Thus, the output Y, U, and V have the same size and can be processed by a VAE structure. On the decoder side, the lumern output Y component is upsampled by a factor of 2 in both the horizontal and vertical directions using conv 1×5×5 / 2↑. In contrast, the chromern component is processed by conv 2×5×5 / 1↑ without upsampling. Thus, the final reconstructed image has the same size as the original input YUV420 image.

[0092] As is evident from the processing described above, the method in Figure 3 provides lumen and chroma components of the same size by downsampling the lumen component. In other words, the component that is most important to the human visual system and carries the most detailed information of the video frame is processed. This results in the discarding of important lumen information, which can lead to a loss of quality.

[0093] However, if the components and / or channels of a frame have different resolutions (i.e., sizes), the VAE structure can no longer compress this frame of the video sequence. In particular, because the three components are not the same size, the VAE structure cannot directly process video frames of chroma subsampling color formats (e.g., YUV422, YUV420). As mentioned above, the input image (including the picture portion of the input image) cannot be processed by the input convolutional network layer, and the output image will not have the same size as the input image.

[0094] First Embodiment Some embodiments of this disclosure may provide solutions to the above-mentioned problems of handling subsampled chromatic components suitable for existing ANN or VAE framework structures.

[0095] According to one embodiment of the present disclosure, a method is provided for encoding a picture portion, the picture portion comprising a rumor component and a chroma component having a resolution lower than that of the rumor component.

[0096] The "picture portion" refers to the number of samples of a picture (image), and this number (quantity) is less than the total number of samples of the picture. Note that the terms "picture" and "image" have the same meaning as used herein. The "picture portion" may include some (one or more) or all of the samples of a picture. The terms "sample" and "pixel" have the same meaning as used herein. A picture may be a still image or a video picture corresponding to, for example, a frame in a video sequence (moving image).

[0097] The picture portion may contain one or more color components, such as a lumen component and one or more chroma components. For example, as described above, the picture portion of the lumen-chroma format YUV420 contains two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the lumen channels for the picture as a whole. Alternatively, the chroma resolution may be lower than the resolution of the lumen channels for a portion of the image. In other words, a portion of the scene captured in a picture may be represented by more lumen samples (of the same chrominance) than chroma samples. Resolution may be given by the size (dimension) of the two-dimensional (2D) lumen or by the size of the 2D chroma channel. This is typically given by the number of samples along the x and y directions of lumen or chroma samples in a 2D image, or by the total number of samples. Relative resolution may be specified as the ratio between luminance samples and chrominance samples, such as 4:2:0 or 4:2:2, as described above. While several exemplary embodiments of lumens and (one or more) chroma channels are described herein, it should be noted that this disclosure is not limited thereto. In general, in addition to, or instead of, one or more color components, other channels such as depth may be processed as discussed in this disclosure.

[0098] This method, shown in Figure 13, includes step 1310 of upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the luma component. The upsampled chroma component is also called the intermediate chroma component. Resolution matching may correspond to dimensional matching of the luma component. The method further includes step 1320 of encoding the luma component and the upsampled chroma component into a bitstream.

[0099] Prior to the upsampling step, the resolution of the lumar component may be obtained. This may be done by determining the resolution from memory or from the bitstream of the input picture, etc. The resolution of the chroma component may be obtained in a similar manner. The resolution of each portion of the picture, including the lumar and chroma channels, may be given in memory or generally in any type of storage medium, along with the respective channels. The resolution of each channel may be given as the number of samples. Alternatively or additionally, the channel resolution may be given as the size of the picture (which may be given in the bitstream or obtained in another way), i.e., as a ratio to all samples of the picture.

[0100] The above encoding can be performed by a variational autoencoder (VAE). VAEs are well-known in the field for efficiently performing tasks related to picture encoding and decoding, and they offer a good compromise between the required bitrate and the quality of the reconstructed image / picture. The original subsampled YUV color format can be encoded by a VAE by upsampling the subsampled chroma component (i.e., chroma with a lower resolution than the luma channel) so that the resized chroma has the same resolution as the luma. The encoder portion of the VAE shown in Figure 2C can then process each of the three equally sized inputs in the manner described above with reference to Figure 2C. In this process, the luma channel remains unaffected, and thus important image information, such as that provided by the luma, can be preserved.

[0101] However, it should be noted that encoding using VAE is only one of the possible advantageous implementations. This is not limiting to the disclosure, and other types of autoencoders, such as those having the structure shown in Figure 2B, may also work efficiently. Furthermore, the disclosure is not limited to autoencoders and can be applied to any ANN-based or generally any trained network. In principle, the disclosure is applicable to any type of encoder or decoder.

[0102] In an exemplary implementation of the method, the upsampling of the chroma component is based on a sample of the ruma component. This means that the upsampling is based on the value of the ruma sample and is therefore content-adaptive. The ruma sample may refer to one or more ruma samples. This includes using all ruma samples for upsampling.

[0103] Chroma upsampling may further include the step of processing the chroma component with a trained network. Size equalization may include the steps of inputting the lumana and chroma components into the trained network, and outputting the upsampled chroma component from the trained network. Using the lumana component as input to this trained network may preserve inter-component information between the lumana channel and the chroma channel.

[0104] The trained network on the encoding side may be any type of network suitable for upsampling. This includes, for example, fully connected networks, sparse neural networks, and artificial neural networks (ANNs) such as convolutional neural networks (CNNs). In particular, CNNs have been successfully applied to image / picture processing tasks for encoding and decoding pictures. A trained network used for chroma upsampling may also be called a size-equalizing network because, as output, it provides resized chroma components (i.e., intermediate chroma components) with a resolution matching the resolution of the luma channels. In other words, after processing by a size-equalizing network, all output luma-chroma channels have the same resolution (i.e., size). However, this disclosure is not limited to such upsampling. There are several possible advantageous implementations of upsampling, as will be shown later. The term trained refers to a type of network that is pre-trained by machine learning. Such a network may also be a trainable network with inputs (supervised or unsupervised) for machine learning. However, a trained network may be provided by constructing weights without training the same network, for example, by obtaining weights from a storage medium. The weights may be obtained and stored beforehand by training a similar network. Alternatively, the weights (also called coefficients) may be derived by using mathematical filters such as Wiener filters or Kalman filters. Another option is to derive the weights by using transformation layers, such as discrete cosine transform (DCT) or discrete sine transform (DST). Transform layers may be included in the trained network, but each weight may be a fixed coefficient or may be considered a parameter.It should be noted that DCT and DST are merely examples, and in general, any suitable known transformation may be applied, such as the Hadamard transform, Fast Fourier transform (FFT), Discrete Fourier transform (DFT), or any transformation obtained by modifying the above transformations to conform to some implementation constraints (such as numerical representation or bit depth), or any orthogonal unitary transform such as the Carunen-Lobe transform.

[0105] As will be discussed further below, upsampling that may be performed in the first layer (or elsewhere) of the trained network may use a bilinear filter or another type of linear or nonlinear interpolation filter instead of nearest neighbor-based filtering. Nearest neighbor filtering may be considered a simplified version of interpolation filtering. Additional or alternative super-resolution methods may be used.

[0106] For example, on the encoder side, the chroma component of a video frame in a subsampling color format such as YUV420 or YUV422 is equalized to the size of the lumen component using a size equalization network layer.

[0107] In one exemplary implementation, the trained network comprises at least two processing layers. These two processing layers include a first layer for upsampling based on the input chroma component, and a second layer for processing the chroma component of the output of the first layer together with the chroma component. The terms “first” and “second” are merely labels and do not necessarily correspond to the first and second layers of the network in terms of layer order. There may be one or more other layers between the first and second layers.

[0108] In one implementation example, processing the output of the first layer together with the lumern component involves element-wise adding the lumern component and the upsampled chroma component. Therefore, because the “addition operation” has low complexity, chroma upsampling using the lumern component as side information can be performed efficiently. Processing based on the “combination” of the lumern and chroma components is not limited to addition. In this example, inter-component information is preserved. Here, “preserved” means that the lumern information, including high-frequency information and details about sharp edges, is used in the chroma upsampling operation. Compared to prior art, the chroma component is thus deeply enhanced.

[0109] Alternatively, chromas may be concatenated with rumas. This typically results in a concatenated ruma-chroma synthesis having a size larger than each of the ruma and chroma components. Other types of “combination” operations may be used instead and / or combined with addition or concatenation, thereby utilizing the ruma information for upsampling and / or processing of the subsampled chroma by a size equalization network.

[0110] The above process can be illustrated by Figures 4 to 8, which show several embodiments of the present disclosure that cover both the encoder side and the encoder side.

[0111] The encoder side shown at the top of Figure 4 includes inputs of the lumana component Y and chroma components U and V. In particular, the chroma components U and V are input to a size equalization network layer 410, which upconverts (one or more) chroma components (compared to the subsampled lumana component Y) to the size of the lumana component. The term "upconvert" may also be called upsampling. As previously stated, the upconversion or upsampling may also be performed using the lumana component as input information (i.e., side information) for the size equalization network, which is shown in Figure 4 by a third input to the NN size equalization module 410. The output of the NN size equalization module 410 is intermediate chroma components U and V, whose dimensions match those of the lumana component Y. Both the lumana component and the intermediate chroma component are input to the VAE encoder 420. The VAE encoder may be an encoder such as those shown in Figures 2B and 2C, etc. The VAE encoder 420 outputs, for example, a compressed bitstream containing the intermediate chroma components U, V and the lumana component Y.

[0112] Correspondingly, the lower part of Figure 4 shows an exemplary decoder side. In particular, the encoded bitstream (such as the bitstream encoded by VAE 420) is input to VAE decoder 470. VAE decoder 470 may be a decoder as shown in Figures 2B and 2C, etc. VAE decoder 470 outputs a lumen component Y and intermediate chroma components U and V. These decoded lumen component Y and intermediate chroma components U and V are not necessarily the same as the lumen and intermediate chroma components on the encoder side. This is because, in some embodiments, VAE encoder 420 may compress the input in an irreversible manner. The decoded components Y, U and V with matching resolution are then input to a downsampling network 480 that performs an NN size transformation back to the input size (resolution). The intermediate chroma components U and V are downsampled to the dimensions of the chroma components. The lumen component Y may be used in the downsampling process.

[0113] Figure 5 shows an example implementation of the present disclosure for the encoder-side size equalization network 410. In this example, each layer belongs to a convolutional neural network (CNN). Referring to Figure 5, the convolutional network layers are defined as follows: • Conv N×K×K: Represents a convolutional network layer having K×K convolution kernels, where K is a positive integer. Typical values ​​for K are 3 or 5. However, K is not limited to these numbers and can be any other positive integer. In particular, larger values ​​of K are possible. N is a positive integer representing the number of output channels in the convolutional neural network. Typical values ​​for N are 128 or 192. However, any other number may be used depending on the output. In particular, N may be 1 or 2, i.e., conv 1×K×K or conv 2×K×K, which correspond to a CNN layer with one or two output channels. • Act.: Refers to an activation layer that is a nonlinear activation function, such as the normalized linear unit (ReLU) or GDN mentioned above. This function is used to activate the convolutional layer. Without activation, the convolutional layer becomes a simple linear combination, which is equivalent to no convolution operation at all. • Conv N×K×K / 2↑: Represents the same as Conv N×K×K, but (in this example) the output is further upsampled (upconverted) by a coefficient of 2 in both the horizontal and vertical directions. This is indicated by the "2" and the upward arrow. • Conv. N×K×K, Act: Represents the convolutional layer + activation layer.

[0114] Figure 5 shows an example of the encoder side for the color format YUV420. This means that three channels, Y, U, and V, are involved, and therefore N=3.

[0115] For example, if the lumer size is w × h, the chroma size is w / 2 × h / 2, and the color format is YUV420, the size equalization network layer upsamples the chroma components U and V by a factor of 2 in both the horizontal and vertical directions. In the formula, w represents the width, and h represents the height of the lumer component, counted, for example, by an (integer) number of samples. The conv N × K × K / 2↑ layer upsamples the U and V components by a factor of 2 in both the horizontal and vertical directions. The output of this layer has N channels.

[0116] As is clear from Figure 5, the Y component is processed in a conv N×K×K layer without upsampling, thus preserving the resolution (dimensions) of the lumens component. The output of this layer also has N channels as a size equalization layer conv N×K×K / 2↑. The output signals of both the conv N×K×K / 2↑ and the conv N×K×K layer are processed by their respective activation layers (i.e., nonlinear activation functions, e.g., ReLU or GDN) that output feature maps, and the feature maps are then added together (summed). In this example, a simple sum of the signals is used. This means that each output is added element by element (the elements of the feature maps are summed). This is possible because the size (dimensions) of the outputs from the conv (convolutional) layers is the same with respect to width and height.

[0117] Other types of operations are possible for operations (combinations) based on the merging of chroma and rumor. For example, concatenation of the rumor output and the chroma output is suitable. Addition can be one of the simplest solutions, resulting in a frame with only N channels. This means that the union operation preserves the number of channels. In contrast, concatenation introduces a latent representation with 2N channels. Thus, the union operation provides low complexity for upsampling chroma together with rumor without further increasing the number of channels.

[0118] The combined signal is then processed by one or more convolutional and activation layers, providing two chroma channels (resized U, V corresponding to the upsampled chroma components) as outputs. The original Y signal is also provided to the next step, which is encoding 420 into a bitstream. The upsampled chroma is also called the intermediate chroma component / channel. These equally sized outputs (signals) are then used as input to the VAE structure.

[0119] In the example in Figure 5, the addition operation of Y with the upsampled chromomers U and V is placed after the first layer. However, the operation (and any other suitable operation for concatenating and / or merging signals) may be placed after any other layer. The advantage of this design is that the Y features are almost entirely extracted by the convolutional layer placed after the addition operation.

[0120] In another example, if the lumen size is w×h, the chroma size is w / 2×h, and the color format is YUV422, the size equalization network layer upsamples the chroma components U and V horizontally by a factor of 2. In this case, the upsampled chroma components have a rectangular shape. In either case, the upsampled chroma components U and V output from the size equalization network layer have the same size as the original Y component. Therefore, the output of the size equalization network layer can be used as input to a VAE structure.

[0121] In one exemplary implementation, upsampling involves inserting an additional line of sample after every n lines of the sample in the chroma component, where lines refer to rows and / or columns, and n is an integer greater than or equal to 1. This means, for example, that each odd-numbered row is an added row. Additionally or alternatively, for example, each odd-numbered column of the resulting upsampled component is an added column. Instead of odd numbers, each even-numbered row and / or even-numbered column may be added. These examples are for the case where n=1, and an additional row is inserted after (or before) every row. Such upsampling is a 2x upsampling. However, this disclosure is not limited to this particular example. Rather, n may be larger, meaning that an additional line of sample is inserted after an integer number of rows. A similar technique is applicable to columns. n may be different for rows and columns. Furthermore, this allows for further adaptation of chroma upsampling by inserting line samples in specific rows and / or columns.

[0122] Furthermore, upsampling may be performed by inserting a line of sample with a value of 0 after every n lines of sample in the chroma component. This may be called zero-filling. This is a simple method of upsampling by simply setting each line sample to 0. Therefore, no further signaling or processing overhead is required. Such a technique is illustrated in Figure 9, which shows a 4x4 sample of upsampled chroma. Black circles indicate samples of the original chroma. White circles indicate samples with a value of 0. In this example, lines of zero-value samples are inserted after each row and each column of the chroma component. Furthermore, in this example, the size equalization network is simplified to a single-layer network (M=1) with no further layers after the “addition” operation in Figure 5. In Figure 9 (encoder side), upsampling of U and V is simplified as filling each row and each column with 0 for the YUV420 color format (for the YUV422 format, zero-filling is for each column). In this case, the network is not a convolutional neural network, but can be understood as a single layer of a fully connected neural network (FCNN). Therefore, this network itself is not necessarily trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network.

[0123] In another implementation, chroma upsampling can be performed by inserting a line of sample with a value equal to the nearest neighbor sample in the chroma component after every n lines of sample in the chroma component. The value insertion may be performed, for example, by copying the chroma value to each neighbor sample. Similar to the zero-filling described above, this allows for further adaptation of chroma upsampling by inserting line samples in specific rows and / or columns. Furthermore, using the values ​​of the inserted line samples that reference the nearest neighbor samples of the chroma makes the accuracy of chroma upsampling more precise because it uses neighbor information that references the original chroma (reflected by the neighbor chroma sample values).

[0124] This is illustrated in Figure 11, which shows a 4x4 sample of the upsampled chroma. Again, the black circles point to the original chroma samples. Here, the white circles point to the samples that have values ​​corresponding to each chroma neighbor sample. In Figure 11, the insertion of chroma values ​​into each neighbor sample is indicated by arrows. In this example, the sample values ​​are inserted after each row and each column of the chroma components. Furthermore, in this example, the size equalization network is simplified to a single-layer network (M=1) with no further layers after the “addition” operation in Figure 5, and inter-component information from luma to chroma is preserved by the “addition” operation. For example, if the luma contains more information (because it has a higher original resolution), the additional information is added to the chroma sample at the correct position when the luma and chroma have the same dimension. In Figure 11 (encoder side), the upsampling of U and V is simplified as upsampling of the nearest neighbor sample in each row and each column, as shown for the YUV420 format in Figure 11. In the case of the YUV422 format, nearest neighbor upsampling is performed for each column. In the above example of chroma upsampling, the network is not a convolutional neural network, but can be understood as one layer of a fully connected network. Therefore, this network itself is not trainable, but can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network. Nearest neighbor upsampling here means that the values ​​of the nearest neighbor samples (in a predefined direction) are copied to the newly added samples. In Figure 11, the nearest neighbors are taken from the left and top. This is a rule that does not limit this disclosure. Nearest neighbors may also be copied from the right and bottom. Interpolation (e.g., averaging of adjacent sample values) or other types of upsampling may be involved.

[0125] The upsampling network described above is merely an example of a trained network for size equalization having one layer. As will be apparent to those skilled in the art, the size equalization layer may have two or more layers, as shown through Figures 5 and 7. In this case, the size equalization network may be trained to perform chroma upsampling.

[0126] In the previous discussion, the luma component was used as side information to upsample the subsampled chroma channel and input to the size equalization layer. In a further implementation of the size equalization network, as shown in Figure 7, upsampling can be performed without further use of the luma. In other words, the luma channel is not input to the size equalization network on the encoder side. Otherwise, the encoder's processing of the subsampled chroma is the same as that already discussed when the luma was used as side information for chroma upsampling. This allows for a simplified design of the size equalization network, as it takes only (one or more) chroma components as input.

[0127] According to one embodiment of the present disclosure, a method is provided for decoding a picture portion from a bitstream, wherein the picture portion comprises a ruma component and a chroma component having a resolution lower than that of the ruma component. The picture portion refers to the number of samples of an image and / or picture, the number being less than the total number of samples of the image / picture. The picture portion may also contain all of the samples of the image or picture. The term sample may also refer to a pixel. The image may be a still picture or a video picture. The picture portion may contain one or more chroma components. For example, the picture portion of the ruma-chroma format YUV420 contains two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the ruma channel for the entire picture / image. Alternatively, the chroma resolution may be lower than the resolution of the ruma channel for a portion of the image. In the case of this “partial” ruma resolution, the ruma channel and (one or more) chroma channels are related to the same portion of the picture. The same part of a picture can also be called the picture's area.

[0128] The method is illustrated in Figure 14 and includes step 1410 of decoding the lumar component and an intermediate chroma component that matches the resolution of the chroma component from the bitstream. The intermediate chroma component is also called the upsampled chroma component, reflecting that the intermediate component was generated / acquired by upsampling the chroma component. The resolution is given in terms of the size (i.e., dimension) of the two-dimensional (2D) lumar or chroma channel. This is typically given by the number of samples along the x and y directions of the 2D lumar or 2D chroma samples, or by the total number of samples. The method further includes step 1420 of downsampling the intermediate chroma component to acquire the chroma component of the picture portion. The above decoding may be performed by a variational autoencoder (VAE). VAEs are well known in the field for performing tasks related to image / picture encoding / decoding very efficiently and have a good compromise between the required bitrate and the quality of the reconstructed picture. Next, the encoder portion of the VAE shown in Figure 2C can process each equally sized channel input, including the lumar and intermediate chromar, in the manner described above with reference to Figure 2C. In this process, the lumar channel remains unaffected, and thus the most important image information provided by the lumar is preserved when decoding the lumar component from the bitstream.

[0129] The original chroma component (i.e., the subsampled chroma) can be reconstructed by downsampling the intermediate chroma component (i.e., the chroma having the same resolution as the luma channel). In particular, the picture portion with the original luma-chroma format can be reconstructed with high quality. Thus, the decoder can provide the picture portion of the chroma format as output, where the chroma has a lower resolution (i.e., size) than the luma.

[0130] In an exemplary implementation of the method, downsampling further includes the step of processing the intermediate chroma components by a trained network. This network may be called a size-transformation network, reflecting its purpose of transforming the input intermediate chroma (which has the same size as the rumor) to the size of the original chroma, which has a lower resolution (i.e., a smaller size) than the rumor.

[0131] The trained network on the decoding side may be any type of network suitable for downsampling. This includes, for example, artificial neural networks (ANNs) such as neural networks and convolutional neural networks (CNNs). In particular, CNNs have been successfully applied to image / picture processing tasks for picture encoding and decoding. The trained network used for chroma downsampling may also be called a resizing network, as it provides as output resized chroma components (i.e., the original subsampled chroma components) with a lower resolution than the lumar channels. In other words, after processing by the resizing network, all output lumar-chroma channels have different resolutions (i.e., sizes).

[0132] In the first exemplary implementation of the method, the steps processed by the trained network include inputting the intermediate chroma components into the trained network and outputting the chroma components from the trained network. This means that the trained network uses only the intermediate chroma as input for downsampling the intermediate chroma. Thus, in this example, no side information about the ruma is used by the size transformation network layer. Consequently, the trained network may be designed with low complexity and provides low processing overhead because the ruma is not used as side information.

[0133] This is illustrated in Figure 8, which shows that the chroma component may not be used as input to the size transformation network by the decoder. In this example, the two chroma components U and V refer to the color format YUV444, and U and V have the same size as chroma Y. Otherwise, the processing of the intermediate chroma by the decoder is the same as that already discussed with reference to Figure 4. In the exemplary implementation of the trained network, the trained network comprises at least two processing layers. Of these at least two layers, the first layer processes the input intermediate chroma component, and the second layer of the at least two processing layers downsamples the output of the first layer.

[0134] In the example in Figure 8, the decoder-side size conversion network consists of multiple convolutional layers and activation layers Conv. N×K×K, act. The first Conv. N×K×K, Act. on the left takes the intermediate chroma components U and V as input. The last layer of the network (Conv. 2×K×K / 2↓) downsamples the signal by a coefficient of 2 in both the horizontal and vertical directions, outputting two channels. These two channels are used as the reconstructed U and V signals in YUV420 format. Once all intermediate Conv. N×K×K, Act. layers are removed, the last Conv. 2×K×K / 2↓ points to the second layer.

[0135] In this example, the Y information is not downsampled compared to J. Balle's method. Therefore, the lumens and spatial information of Y are largely preserved. As described above, chroma downsampling can be performed without using lumens information as side information.

[0136] In another exemplary implementation, the downsampling of the intermediate chroma component is based on the samples of the luma component. This means that the downsampling is not based on size, but rather on the contents of the luma channel, i.e., the values ​​of each luma sample. A luma sample can refer to one or more luma samples. This includes using all luma samples for upsampling.

[0137] When chroma is processed by a trained network, the processing step by the trained network includes inputting luma components into the trained network. In one exemplary implementation, the first layer of the trained network performs the step of processing intermediate chroma components together with luma components. In other words, luma is used as side information for chroma downsampling. This preserves inter-component information between luma channels and chroma channels. In particular, luma components may contain more detailed information than the subsampled (original) chroma components. By using luma as side information, some details of the chroma can be reconstructed based on the luma components. An upsampling network may be specifically trained for such upsampling.

[0138] On the decoder side, the first processing layer of the size transformation network may not need to perform an "addition operation," as opposed to the encoder side. Such an operation may be performed as shown in Figure 5 (at least some kind of explicit concatenation operation may be used in the size equalization network in the encoder). This is because the input to the first layer of the decoder's size transformation network, including the lumer Y and (one or more) intermediate chroma U, V, already has the same size, as shown in Figure 6. Rather, the three equally sized channels can be input directly to the first CNN layer without requiring a separate addition operation. This is not the case for the input to the first layer of the encoder's size equalization network shown in Figure 5. In this case, the chroma is first upsampled, and then the upsampled chroma and lumer are added together. This is then fed to the CNN layer.

[0139] Figure 6 shows an example implementation of the decoding side of a VAE structure, taking the output of the encoding side of the VAE structure described above as its input. As mentioned above, here each input to the size conversion network has three components of the same size (i.e., YUV444). The first channel is used directly as the reconstructed lumen component and has the same size as the original lumen component in the YUV420 format. Conv. 2×K×K / 2↓ represents the same as Conv N×K×K, but the output is further downsampled (downconverted) by a coefficient of 2 in both the horizontal and vertical directions. This is indicated by the "2" and the down arrow. As mentioned above, such an arrangement can be advantageous in order to preserve intercomponent information, for example, to enhance the chroma with further detail from the lumen, which originally had a higher resolution (dimension) than the chroma.

[0140] In general, a trained network using a ruma on the decoder side of a VAE structure may be constructed in the same manner as one without a ruma, except that it considers an additional ruma as an input. This is illustrated in Figure 6, where the size-transformed network is again designed with multiple convolutional and activation layers Conv. N×K×K, Act. In this case as well, the last layer of the network (Conv. 2×K×K / 2↓) downsamples the signal by a factor of 2 in both the horizontal and vertical directions, outputting two channels. These two channels are used as the reconstructed U and V signals in YUV420 format.

[0141] For example, if the color format is YUV420, the size conversion network layer downsamples the intermediate chroma components U and V by a factor of 2 both horizontally and vertically. In another example, if the color format is YUV422, the size conversion network layer downsamples the intermediate chroma components U and V by a factor of 2 horizontally. In either case, the final output has the same size as the original components of the input frame with color format YUV420 or YUV422, respectively.

[0142] In an exemplary implementation of the method, downsampling includes the step of removing sample lines after every n lines of the intermediate chroma component sample, where lines are rows and / or columns, and n is an integer greater than or equal to 1. This means, for example, that the line samples may be removed or deleted after each one (n=1) or two (n=2) rows and / or columns. Thus, downsampling of the intermediate chroma can be performed quickly by simply removing each line sample. Alternatively, downsampling may be performed, for example, by averaging (possibly weighted) the sample values ​​closest to each chroma sample.

[0143] Examples of line sample removal are shown in Figures 10 and 12. Note that the sample configurations in Figures 10 and 12 correspond to the sample configurations in Figures 9 and 11, respectively, as a result of the upsampling process of the chroma component on the encoder side.

[0144] Specifically, Figures 10 and 12 show 4x4 samples of the intermediate chroma. Black circles indicate samples of the original chroma. White circles marked with an "x" indicate samples that may be non-zero as a result of chroma upsampling by the encoding-side size equalization network (with and without the use of lumens as side information) and / or chroma processing through the first (one or more) layers of the decoding-side size transformation network.

[0145] In the examples shown in Figures 10 and 12, the size transformation network is simplified as a single-layer network (M=1). In Figures 10 and 12 (decoder side), U and V downsampling is simplified to discard odd-numbered row and column samples for the YUV420 color format. For YUV422, odd-numbered column samples are removed. In both of these cases shown in Figures 10 and 12, the respective networks are not convolutional neural networks, but can be understood as a single layer of a fully connected neural network (FCNN). Therefore, the network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to the trained network.

[0146] However, this is merely an example of a trained network for size transformation having one layer. As will be apparent to those skilled in the art, the size transformation layer may have two or more layers, as shown through Figures 6 and 8. In this case, the size transformation network may be trained to perform chroma upsampling.

[0147] It should be noted that the encoder-side size equalization network shown in Figure 5 and the decoder-side size transformation network shown in Figure 6 can be trained together using a VAE structure. This can further improve the encoding-decoding process of subsampled chroma components, particularly within the VAE framework. This includes, for example, encoding-decoding performance and / or the quality of the reconstructed picture / image.

[0148] Alternatively, they may be trained separately and used as pre-trained VAE networks. In either case, each network refers to a trained network. Note that up / down sampling (sampling coefficient) with a coefficient of 2 is an unrestricted example but is one of the typical values ​​used. In general, the sampling coefficient may be any integer greater than or equal to 1. Alternatively, the sampling coefficient may be a fraction given as a ratio, for example, p / q of two positive integers p and q.

[0149] To summarize this part of the embodiment, a size equalization network (for upsampling) and a size transformation network (for downsampling) are used to adjust the size of the subsampled chroma channels so that the three components of YUV420 have the same size. These can then be handled / processed by an existing VAE structure.

[0150] Furthermore, as already described, the disclosure also provides a device configured to perform the steps of the method described above.

[0151] Figure 15 shows an encoder 1500 for encoding a picture portion, which includes a rumor component and a chroma component having a lower resolution than the rumor component. The encoder comprises an upsampling module 1510 configured to upsample the chroma component to obtain an upsampled chroma component that matches the resolution of the rumor component, and an encoding module 1520 configured to encode the rumor component and the upsampled chroma component into a bitstream.

[0152] In one implementation example, the encoder may include one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the encoder is configured to perform the above method when the program is executed by the one or more processors.

[0153] Figure 16 shows a decoder 1600 for decoding a picture portion from a bitstream, the picture portion comprising a lumen component and a chroma component having a lower resolution than the lumen component. The decoder comprises a decoding module 1610 configured to decode the lumen component and an intermediate chroma component matching the resolution of the chroma component from the bitstream, and a downsampling module 1620 configured to downsample the intermediate chroma component to obtain the chroma component of the picture portion.

[0154] In particular, the decoder may include one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the decoder is configured to perform the above method when the program is executed by the one or more processors.

[0155] A computer program stored on a non-temporary medium, according to one embodiment of the present disclosure, which includes code that, when executed on one or more processors, performs any of the steps of the above method.

[0156] Some embodiments of this disclosure may offer the following advantages, which can be summarized as follows: • Ruma's sample information is fully preserved. • Ruma's spatial information is completely preserved. The network used for size equalization and size transformation (i.e., the trained network) (both trained networks may be CNNs) may be trained together with the VAE network structure. This may offer the advantage of further improving the results. • Flexibility in handling various other chroma subsampling formats through VAE network structures such as YUV422. • Inter-component information can be stored.

[0157] Second Embodiment The second embodiment may be combined with any of the exemplary implementations of the first embodiment. As described above, the first embodiment deals with the handling of subsampled chroma components compared to luma components. Upsampling of the chroma component creates such a resized chroma component along with a luma component suitable for further processing within a trainable network-based framework such as an ANN or VAE. Thereafter, the luma component may remain largely unaffected, and thus the sample and spatial information of the luma may be preserved. This can be desirable for many applications where high quality of the reconstructed picture (or part thereof) is required. This means that high resolution must be maintained, and therefore the encoding-decoding process must not tamper with the luma.

[0158] However, in other applications, high resolution is not necessarily the most important goal; rather, a faster encoding-decoding process is preferable. Furthermore, the desired goal may be a low bitrate, resulting in a lower resolution, such as that provided by lumar downsampling. Therefore, it may be desirable to choose between encoding-decoding processes that provide either low-resolution or high-resolution pictures.

[0159] This problem may be solved by a second embodiment that addresses adaptive switching between lumar downsampling and chroma upsampling when matching the resolution of two or more input channels. Lumar downsampling and chroma upsampling may also be called sampling modes. Which of the two sampling modes is performed is determined at the encoder side based on the lumar and chroma channels, signaled to the decoder side, and / or determined by the decoder side.

[0160] In particular, as will be discussed further below with reference to various exemplary implementations and aspects of this disclosure, lumar downsampling matching the chroma size (resolution) and lumar upsampling matching the luma size (resolution) are adaptively selected by the encoder. Which of the sampling modes is selected by the encoder may be indicated in various ways, as will be further detailed below, and may include indication by the received bitstream and / or determination from the received bitstream. Depending on how one of the sampling modes (i.e., lumar downsampling or lumar upsampling) is indicated, the decoder then performs either lumar upsampling or lumar downsampling to generate a reconstructed picture portion having the same resolution (i.e., the same subsampling format as the input frame) as the picture portion input to the encoder.

[0161] According to one embodiment of the present disclosure, a method for encoding a picture portion is provided. The picture portion comprises a ruma component and a chroma component having a resolution lower than the resolution of the ruma component. The picture portion refers to the number of samples of an image and / or picture, the number being less than the total number of samples of the image / picture. The picture portion may also include all the samples of the picture. The term sample may also refer to a pixel. The picture portion may be a still picture or a video picture. The terms picture, image, and picture portion may be used synonymously. The picture portion may include one or more chroma components. For example, the picture portion of the ruma-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution of the picture portion may be lower than the resolution of the ruma channels for the entire picture or image. Alternatively, the chroma resolution may be lower than the resolution of the ruma channels for a portion of the image. In the case of this "partial" lumen resolution, the lumen channel and (one or more) chroma channels relate to the same portion of the picture. This portion of the picture can also be called the region of the picture.

[0162] Downsampling lumens is beneficial for saving bitrate at low resolutions, while upsampling chromens is beneficial for preserving lumens information at high resolutions. Therefore, adaptive switching allows for the selection of sampling modes according to requirements and / or limitations on bitrate or picture or image quality. In either case of selected picture processing of the picture portion, the processed lumens-chromens components have matching resolutions, so the output can be handled within a VAE framework or another framework that enables efficient encoding.

[0163] This method is illustrated in Figure 31, and as picture processing for the picture portion A step of downsampling the lumens component to obtain a downsampled lumens component that matches the resolution of the chromens component, and Steps to upsample the chroma component to obtain an upsampled chroma component that matches the resolution of the chroma component. The procedure includes step 3110, which involves selecting one of the two options.

[0164] Resolution is given in terms of the size (i.e., dimensions) of a two-dimensional (2D) lumen channel. However, it should be noted that resolution can also be given by the dimensions of channels other than the lumen, such as chroma channels, or other types of channels, such as depth channels. Resolution is typically given by the number of samples along the x and y directions of the 2D lumen and / or chroma channels, or by the total number of samples in the channels at a given moment (the total number of samples in the picture).

[0165] Prior to lumar downsampling and / or chroma upsampling, the resolutions of the lumar and / or chroma components may be obtained, for example, from memory or storage, and the resolutions may be stored together with the picture portion to be encoded. The picture portion may be a part of a picture or the entire picture. Thus, the picture portion may contain one or more samples of the lumar and one or more samples of the chroma. In some embodiments, the picture portion may be one of a plurality of picture portions that make up a picture. These may be obtained by subdividing the picture into picture portions. Generally, the picture portions may have the same size or may have different sizes (dimensions).

[0166] A picture may be part of a video. The resolution of each channel may be provided as the number of samples. Alternatively, the channel resolution may be provided as the size of the picture, i.e., as a ratio to all samples of the picture.

[0167] The method in Figure 31 further includes step 3120, which processes the picture portion according to a selected picture processing. In other words, the selected picture processing may be either luma downsampling or chroma upsampling. Furthermore, the method includes step 3130, which generates a bitstream according to the selected picture processing, wherein generating includes encoding the processed picture portion. The encoding of the picture portion may be performed by an autoencoder, for example, a variational autoencoder (VAE), or by another type of autoencoder. VAEs are well known in the field for efficiently performing tasks related to the encoding and decoding of pictures and have a good compromise between the required bitrate and the quality of the reconstructed picture.

[0168] Picture processing that selects either lumar downsampling or chroma upsampling will, in either case, provide resized lumar and chroma components with equal resolution (e.g., equal dimensions). Therefore, regardless of the selected picture processing, equally sized lumar-chroma channels can be handled within a common framework such as an ANN framework, for example, a VAE.

[0169] This includes subsampled chroma formats such as YUV422 and YUV420. The color format may also include any other chroma-chroma format in which any of the channels may be of different sizes. The encoder portion of the VAE shown in Figure 2 can then process each of the three equally sized inputs in the manner described above with reference to Figure 2. Furthermore, the picture processing selection allows for adaptive switching between chroma downsampling and chroma upsampling. Thus, low-resolution or high-resolution picture processing can be selected.

[0170] Figure 17 illustrates the encoder side in an example of the YUV420 color format, where the chroma inputs U and V have a lower resolution than the lumen component Y. As shown in Figure 15, on the encoding side, the YUV input undergoes size equalization. This involves, on the one hand, downsampling of the lumen component as shown in the upper branch of Figure 1710. The output of this example is three YUV components of the same size, where the same size refers to the size (dimensions) of the chroma components U and V. On the other hand, the chroma component is upsampled through size equalization performed in the lower branch of Figure 1720. Again, the output is three YUV components of the same size, where the same size refers to the lumen component. After encoding the size-equalized YUVs from the upper and lower branches, for example by a VAE, the cost with respect to a cost function is determined. This may be based, for example, on rate-distortion optimization (RDO)1730. The final sampling mode used (i.e., chroma upsampling or chroma downsampling) depends on whether the lower branch processing or the upper branch processing is the one providing the lowest cost. In the example in Figure 17, the lowest-cost sampling mode is indicated by a flag (indicator) and is either encoded into the bitstream or separately signaled to the decoder. Various methods for selecting a sampling mode based on cost are discussed in detail below.

[0171] In RDO, the weighted sum of rate and distortion is optimized. However, this disclosure is not limited to cost minimization based on rate and disclosure. For example, complexity may be considered in addition to, or instead of, rate and / or distortion in the cost function. Other cost parameters may be added or used instead of those mentioned above.

[0172] In one implementation example of the method, upsampling and / or downsampling are performed by a trained network.

[0173] As already described in Embodiment 1, the trained network on the encoding side may be any type of network suitable for downsampling or upsampling. This includes, for example, artificial neural networks (ANNs) such as neural networks and convolutional neural networks (CNNs). The trained network used for lumar downsampling and / or chroma upsampling may also be called a size equalization network, since both networks provide resized color (lumar, chroma) components as output that have a resolution matching the resolution of the chroma channel (lumar downsampling 1710) or the lumar channel (chroma upsampling 1720). In other words, after processing by the size equalization network, the output Y, U, and V channels have the same resolution (i.e., size).

[0174] Figures 19 and 20 show example implementations of size equalization networks on the encoder side for the YUV420 color format. Figure 19 shows a size equalization network that performs lumar downsampling (upper branch), and Figure 20 shows a size equalization network that performs upsampling of two chroma U and V. In this example, each layer belongs to a convolutional neural network (CNN).

[0175] The convolutional network layers shown in Figures 19 and 20 are defined in the same way as in Embodiment 1 (for example, Figures 5 to 8). In summary, they are as follows: • Conv N×K×K: Represents a convolutional network layer having K×K convolutional kernels, where K is a positive integer. Typical values ​​for K are 3 or 5. However, K is not limited to these numbers and may be set to any other positive integer. In particular, values ​​of K greater than or equal to 2 are possible. N is a positive integer and represents the number of output channels in the convolutional neural network. • Act.: Refers to an activation layer that can be a nonlinear activation function, such as the aforementioned ReLU or GDN. ·Conv N×K×K / 2↑: Represents the same as Conv N×K×K, but this layer further upsamples (upconverts) the output by a coefficient of 2 in both the horizontal and vertical directions. This is indicated by the "2" and the upward arrow. • Conv. 2×K×K / 2↓: Represents the same as Conv N×K×K, but further downsamples (downconverts) the output by a coefficient of 2 in both the horizontal and vertical directions. This is indicated by the "2" and the down arrow. • Conv. N×K×K, Act: Represents the convolutional layer + activation layer.

[0176] As shown in Figures 19 and 20, the color format is YUV420 for the picture portion to be encoded. This means that the chroma size is w × h and the chroma size is w / 2 × h / 2. However, as will be obvious to those skilled in the art, other formats (types of chroma subsampling), such as YUV422, are also feasible.

[0177] Luma Processing - The size equalization network for luma processing in Figure 19 comprises a first layer Conv. N×K×K / 2↓ having K×K kernels and N channels as outputs. This layer downsamples the luma components by a coefficient of 2 in both the horizontal and vertical directions. This is followed by an activation layer sequence Act. and a convolutional layer Conv. N×K×K, Act. with activation. The size equalization network is terminated by a final layer which is a convolutional layer Conv. 1×K×K, thus having one channel as an output, i.e., a downsampled luma component Y. Here, the luma component is resized to have the same size (resolution) as the chroma components U and V. This processing is performed at the upper branch (1710) in Figure 17.

[0178] Chroma Processing - The size equalization network for chroma component processing in Figure 20 comprises a first layer Conv N×K×K / 2↑ having K×K kernels and N channels as outputs. This layer upsamples two chroma components U and V by a coefficient of 2 both horizontally and vertically. This is followed by an activation layer sequence Act. and a convolutional layer Conv. N×K×K, Act. with activation. The size equalization network is terminated by a final layer which is a convolutional layer Conv. 2×K×K, thus having two channels as outputs, i.e., upsampled chroma components U and V. Here, chroma components U and V are resized to have the same size (resolution) as the lumen component Y. This processing is performed at the lower branch (1720) in Figure 17. Note that in this case, the lumen component is preserved with all the advantages discussed in the first embodiment.

[0179] As is evident from Figures 19 and 20, in both cases, the outputs of the respective convolutional layers Conv. N×K×K / 2↓ and Conv N×K×K / 2↑ have N channels and are further processed in an activation layer (i.e., a nonlinear activation function, e.g., ReLU or GDN). The resized lumers and (one or more) chromers may also be called intermediate lumers and (one or more) intermediate chromers, respectively. Each equally sized output lumer-chroma (one or more) is then used as input to the respective VAEs of the upper and lower branches. The VAEs of the upper and lower branches consequently output coded bitstreams, which can then be reconstructed to obtain quality (loss). Quality (inversely proportional to distortion) and rate (e.g., measured by the number of bits in the bitstream) can then be used in the RDO1730 to select the appropriate processing that the VAE following the RDO may perform.

[0180] The size equalization process described above is not limited to the YUV420 color format, but may also be applied to other subsampled chroma formats such as YUV422.

[0181] In the first embodiment, it was explained that encoder-side chroma upsampling may utilize the lumen component as side information. This may be combined with the above processing of (one or more) chroma components in the second embodiment. Furthermore, the same procedure may be applied to encoder-side lumen downsampling, in that the chroma is used as side information.

[0182] Figure 23 further illustrates this with an example case where the selection of picture processing for the picture portion can be signaled by a flag (indicator) encoded in the encoded bitstream, for an encoder side showing an encoder structure similar to that of Figure 17. As shown in Figure 23, the difference from the schematic diagram of Figure 17 is that lumern downsampling (upper branch) utilizes information about chroma components U and V. Similarly, chroma upsampling (one or more) (lower branch) utilizes information about lumern Y. Otherwise, the sampling processes for lumern and (one or more) chroma are performed in the same manner as discussed above with reference to Figure 17.

[0183] Therefore, whether lumar downsampling and / or chromar upsampling are performed, in either case, cross-information is used for each sampling. This means that the correlation between the lumar and (one or more) chromars is preserved.

[0184] Figures 25 and 26 show exemplary implementations of size equalization networks for lumar downsampling and chroma upsampling on the encoder side, respectively. As shown in Figures 25 and 26, each size equalization network has the same design in terms of the order in which the convolutional and activation layers are arranged as shown in Figures 19 and 20. The difference is that an additional layer is added to provide each size equalization network with (one or more) chromas in Figure 25 or lumas in Figure 26. In Figure 25, the added layer is a convolutional layer Conv N×K×K / 1↑ for the chroma component, having K×K kernels and N channels as outputs, which is input to an activation layer Act. In Figure 26, the added layer is a convolutional layer Conv. N×K×K / 1↓ for the lumas, having K×K kernels and N channels as outputs, which is input to an activation layer Act. In the examples shown in Figures 25 and 26, the respective output signals are added to the output of the activation layer Act. of the size equalization network after the first convolutional layers Conv. N×K×K / 2↓ and Conv N×K×K / 2↑, respectively.

[0185] For example, each addition may involve adding (one or more) chroma components to (one or more) downsampled luma components element by element, and adding (one or more) luma components to (one or more) upsampled chroma components element by element. Thus, chroma upsampling and luma downsampling using luma as side information can be performed efficiently. Moreover, details that may exist in the luma which originally had a higher resolution can be restored in the originally subsampled chroma. Similarly, details that may exist in the chroma which originally had a lower resolution can be restored in the original luma.

[0186] The combining (combining) process of lumar and chroma components (in both lumar downsampling and / or chroma upsampling) is not limited to addition. Alternatively, chroma and lumar may be concatenated. This typically results in a concatenated lumar-chroma synthesis with a size larger than each of the lumar and chroma components. Other types of combination operations may be used instead and / or combined with addition or concatenation. Thereafter, the information of lumar and chroma is utilized for chroma upsampling and / or lumar downsampling by their respective size equalization networks.

[0187] As mentioned earlier, whether picture processing is performed on the encoding side can be determined in various ways. This will be discussed next.

[0188] In an exemplary implementation of the method, selection involves evaluating the rate and distortion cost functions. According to one implementation of the (encoding) method, selection further includes determining the costs of the intermediate downsampled lumar and chroma components, and the costs of the lumar component and the intermediate upsampled chroma component, respectively, according to the cost function. Furthermore, in the selection, the picture processing of the picture portion with the lower determined cost is selected. Thus, by using the sampling mode with the lowest cost (i.e., lumar downsampling or chroma upsampling), the overall encoding of the lumar component and the subsampled chroma component is performed efficiently.

[0189] For example, the cost may be a specified loss in relation to the distortion between the reconstructed picture portion and the original picture portion, and the bitrate. In this case, the loss may be expressed as follows: Loss = Rate + λ Distortion or Loss = λ rate + distortion.

[0190] In the formula, λ represents the Lagrange multiplier, which is a number that can be predefined. This number may be determined empirically based on the desired application and / or result. As described above, alternative or additional parameters of the cost function may be used. Furthermore, it should be noted that this disclosure does not necessarily include RDO. For example, selection may be performed by a trained network (e.g., ANN) that has been trained for such selection. Various other methods may also be possible.

[0191] The selection may further include downsampling the lumar component to obtain an intermediate downsampled lumar component, and upsampling the chroma component to obtain an intermediate upsampled chroma component. In Figure 17, after RDO 1730, and after either lumar downsampling or chroma upsampling, the VAE is applied to the color components of matching size. However, this is not required. Rather, instead of using the VAE again, the bitstream generated by the VAE of either the selected upper (1710) or lower (1720) branch may be taken. In other words, the coded portion may be reused, thus reducing some complexity.

[0192] Figure 17 shows three VAEs. Applying a VAE three times can be complex and time-consuming. Therefore, to reduce complexity, in some exemplary implementations, the VAEs at the upper and lower branches are not complete networks. Rather, they may be replaced by models that do not actually generate bitstreams, but rather estimate rate and distortion. After the best branch is estimated / selected, the actual VAE is used after the RDO1730 to generate the bitstream that is sent (transmitted) to the decoder. This can be advantageous if the rate and distortion estimation by the model VAE at the branch is faster than that of the VAE following the RDO. On the encoder side, both cases are possible (two VAEs versus three VAEs).

[0193] In one exemplary implementation, cost determination includes encoding the intermediate downsampled lumen and chroma components into a first bitstream, and encoding the lumen component and the intermediate upsampled chroma component into a second bitstream. Subsequently, the costs of the first and second bitstreams are determined according to the cost function. Thus, the actual cost can be determined precisely, since it is based on the encoded first and second bitstreams. However, as stated above, this disclosure is not limited to such methods. Rather, instead of performing encoding to bitstreams, the cost may be estimated by an estimation function. Furthermore, the encoding for obtaining the rate does not have to be done by actually generating bitstreams. For example, the cost may be estimated with respect to the number and / or distribution of quantized data, or by applying a trained network trained for such estimation, or by any conceivable method.

[0194] According to one implementation example, generating a bitstream for carrying encoded picture or video data includes encoding a selection indicator in the bitstream that indicates the selected picture processing. This provides low-complexity signaling of the picture processing performed on the bitstream. For example, the selection indicator may have a length of 1 bit and may distinguish between two values, one representing lumen downsampling and the other representing chroma upsampling.

[0195] This is shown in Figure 17. As previously mentioned, sampling processes of lumern downsampling (upper branch) and chroma upsampling (lower branch) are performed on the picture portion having a lumern component and a subsampled chroma component. In both cases, lumern-chroma components of equal size are provided and input to, for example, the upper branch and lower branch VAEs, respectively. Both VAEs calculate the cost for their respective lumern-chroma inputs, which may be based on bitrate and distortion (RDO). The respective picture processing is selected depending on which cost is lower. A bitstream is generated based on the selected picture processing, but without encoding an indicator in the bitstream. The selection of the picture processing (i.e., lumern downsampling or chroma upsampling on the encoder side) may be indicated by a selection indicator. The indicator may be, for example, a flag. By inserting (which may include encoding) the flag into the bitstream, the selected picture processing performed on the encoding side may be signaled to the decoder in the bitstream. The decoder then parses the indicator from the bitstream and decides which picture processing to perform based on the indicator. In particular, such a selection indicator can take a first value and a second value. The first value of such a selection indicator may indicate that the chroma is downsampled in the encoder (1710) and therefore should be upsampled in the decoder. The second value of such a selection indicator may indicate that the chroma is upsampled in the encoder (1720) and therefore should be downsampled in the decoder. In Figure 17, the function block "Up or Down" 1740 represents the insertion of a selection indicator into the bitstream.

[0196] Alternatively, the selected picture processing may be indicated without an explicit indicator (e.g., a 1-bit flag) encoded in the bitstream.

[0197] According to another implementation example, generating a bitstream involves encoding instructions for the width W and / or height H of the lumen component into the bitstream. Thus, the width and height of the original picture portion can be signaled to the encoder. Such instructions may specify the resolution of the lumen component. If the ratio between the lumen resolution and the chroma component is known, the chroma resolution can be determined accordingly based on the signaled W and H. This signaling may be part of the picture or video coding. Typically, the bitstream of the encoded picture or video contains instructions for the dimensions of the picture or video.

[0198] An exemplary implementation is illustrated in Figure 29. Similar to the example illustrated in Figure 17, sampling is performed on the picture portion having a lumen component and a chroma component with a lower resolution than the lumen component (thus, "subsampled" with respect to the lumen component), using lumen downsampling (upper branch) 2910 and chroma upsampling (lower branch) 2920. The equally sized lumen-chroma components of the upper and lower branches are input to the VAEs, respectively. Both VAEs calculate the possible cost for their respective lumen-chroma inputs, based on the bitrate and distortion (RDO) 2930. Depending on which cost is lower, the respective picture processing is selected. A bitstream is generated based on the selected picture processing. In this exemplary implementation, apart from Figure 17, no selection indicator (e.g., a flag) is encoded in the bitstream. The decoder can then determine which picture processing to perform based on the bitstream, for example, based on the width and height of the lumen encoded in the bitstream. Alternatively or additionally, further parameters signaled in the bitstream, for example for coding purposes, may be used in the decoder to derive which processing (luma upsampling or chroma downsampling) should be applied. Such parameters may be predefined and known in the encoder and decoder so that they consistently use them for processing selection.

[0199] According to one embodiment of the present disclosure, a method is provided for reconstructing a picture portion from a bitstream. Such a method may be applied on the decoder side. As described above with reference to Embodiment 1, the picture portion includes a lumen component and a chroma component having a resolution lower than the resolution of the lumen component. The picture portion refers to the number of samples of an image and / or picture, the number being less than the total number of samples of the image / picture. The picture portion may also include all samples of the image or picture. The picture may be a still picture or a video (a sequence of pictures). The picture portion may include one or more chroma components. For example, the picture portion of the lumen-chroma format YUV420 includes two chroma components U and V (also called chroma channels). The chroma resolution may be lower than the resolution of the lumen channel (in the picture portion or across the entire picture). The lumen and chroma components of a picture portion are typically spatially correlated, as they point to the same picture region.

[0200] The method, illustrated in Figure 32, includes step S3210 of decoding lumen and chroma components having the same resolution from a bitstream. As described above, the resolution may be given by the size (i.e., dimension) of the two-dimensional (2D) lumen channel. Decoding of the processed picture portion from the bitstream may be performed by an autoencoder such as a variational autoencoder (VAE). VAEs are well known in the field for performing tasks related to encoding and decoding images or pictures very efficiently and have a good compromise between the required bitrate and the quality of the reconstructed picture. However, this disclosure is not limited to the application of autoencoders. In general, any encoder may be applied, including encoders that use a network trained on only a part of the operation or function of the decoder (e.g., an ANN or an arbitrary ML / DL based network).

[0201] The method is as picture processing, The steps include: upsampling the decoded chroma component and obtaining a reconstructed picture portion including the decoded chroma component and the upsampled chroma component; The steps involve downsampling the decoded chroma component and obtaining a reconstructed picture portion containing the decoded chroma component and the downsampled chroma component. The process further includes step S3220, which determines one of the two based on the bitstream.

[0202] The decision may be made by parsing the bitstream for selection indicators that indicate the processing to be applied. However, as described above, in some embodiments, the selection instructions do not need to be explicitly signaled. Rather, they may be derived from other parameters in the bitstream related to the encoded picture or video data.

[0203] Furthermore, the method includes step S3230, which involves acquiring the picture portion according to the determined picture processing. Thus, the picture portion encoded in the bitstream can be processed by the decoder portion of the same type of trained network, such as the VAE shown in Figure 2, since the encoded luma and chroma components have the same size. Furthermore, the picture processing decision, either luma upsampling or chroma downsampling, in either case provides resized luma or chroma components that have different sizes corresponding to the components input for encoding (the original picture components). This means that the picture portion is reconstructed to its original color format, where the chroma has a lower resolution than the chroma. Thus, regardless of the determined picture processing, encoded equal-sized luma-chroma channels can be handled within a common VAE framework. Furthermore, the picture processing decision allows for adaptive switching between luma upsampling and chroma downsampling. This ensures that in either case, the picture portion is reconstructed with its original color format. Therefore, adaptive switching allows for the selection of low-resolution or high-resolution picture processing based on several optimization criteria, such as RDO.

[0204] Figure 18 illustrates the decoder side and its operation. Referring to the example in Figure 17, where the sampling mode is indicated by a flag (indicator) encoded in the bitstream, the flag is parsed from the bitstream on the decoder side. Depending on the value of the flag in function block 1840 (i.e., whether it indicates downconversion (luma) or upconversion (chroma) on the encoder side), the luma is upconverted 1810 or the chroma is downconverted 1820 in each of the upper and lower branches. This means that, depending on the flag's indication, the luma-chroma component of the input bitstream has either the size of the (one or more) chroma or the size of the luma of the originally coded picture. In either case, each upsampling and / or downsampling process may be performed by a size conversion process, providing a reconstructed frame of its original format as output. However, as shown in Figure 18, the upper branch 1810 performs luma upsampling, and the lower branch performs chroma component (U and / or V) downsampling 1820. Each size transformation can be performed by a corresponding size transformation network. As in Figure 17, the indication of the sampling mode by an encoded flag is one example that allows the decoder to determine which sampling mode to use for reconstructing the original picture portion. Various means by which the decoder determines which sampling mode to use may be used, as will be discussed below. Furthermore, it should be noted that in these examples, the resized color components (channels) match the size of the originally coded picture. However, this is not an limitation of this disclosure. In some applications, it may be beneficial to provide color components with dimensions different from those of the originally coded picture.

[0205] In one implementation example, upsampling and / or downsampling are performed by a trained network. The trained network on the encoding side may be any type of network suitable for downsampling or upsampling. This includes, for example, artificial neural networks (ANNs) such as neural networks and convolutional neural networks (CNNs). The trained network on the decoder may be any resizing network that adapts the size of the input channels to the desired resolution. The result may be channels with different dimensions (sizes), such as the dimensions of the originally encoded picture or other desired dimensions.

[0206] Figures 21 and 22 show an example implementation of a decoder-side size transformation network for the YUV444 color format, where the lumern and chroma components U and V have the same size. Figure 21 shows a size transformation network that performs lumern upsampling (upper branch corresponding to 1810). Figure 22 (corresponding to 1820) shows a size transformation network that performs downsampling of the two chroma U and V. In this example, each layer can be seen as a layer of a convolutional neural network.

[0207] Referring to Figure 5, the convolutional network layers shown in Figures 21 and 22 are similarly defined. Correspondingly, the same terminology and / or notation are applied here as well. As shown in Figures 21 and 22, the input color format is YUV444 for the picture portion to be decoded from the bitstream. This means that the lumens and chroma components decoded from the bitstream have the same size (i.e., resolution).

[0208] Luma processing (Branch 1810 above in Figure 18 and Figure 21): The size transformation network for luma processing consists of a sequence of convolutional layers + activation layers Conv. N×K×K, Act. The size transformation network is terminated by a final layer which is a convolutional layer Conv 1×K×K / 2↑, which upsamples the luma component by a coefficient of 2 both horizontally and vertically. The output is a single channel, i.e., a reconstructed luma component with the same size as the original luma.

[0209] Chroma processing (Branch 1820 below Figure 18 and Figure 22): The size transformation network for chroma component processing similarly consists of a convolutional layer + an activation layer Conv. N×K×K. Act. The size transformation network is terminated by a final layer which is a convolutional layer Conv. 2×K×K / 2↓, downsampling the two chroma components by a coefficient of 2 in both the horizontal and vertical directions. The output is two channels, i.e., two reconstructed chroma components having the same size as the original chroma components. Note that in this case, the chroma components can be preserved with all the advantages discussed in the first embodiment.

[0210] Since the input rumor-chroma components to the decoding-side (one or more) size-transformation network have the same resolution (i.e., size), they can be efficiently processed within the same existing framework, such as an ANN architecture, that has the same input size. Thus, a picture portion with any color format having (one or more) subsampled chromas can be reconstructed using the same network architecture applied to the rumor. This also makes it possible to preserve inter-component correlation information when processing different components (channels).

[0211] In the first embodiment, it was explained that the chroma downsampling on the decoder side can use the luma component as side information. This may be combined with the above processing of the (one or more) chroma components of the second embodiment. Further, the same procedure can be applied to the upsampling of the luma on the decoder side in that the chroma is used as side information.

[0212] FIG. 24 further illustrates an exemplary case in which, for the decoder side having the same decoder structure as in FIG. 18, the selection of picture processing of the picture part can be signaled by a flag (indicator) encoded in the encoded bitstream. As shown in FIG. 24, the difference from the schematic diagram of FIG. 18 is that the luma upsampling (upper branch) utilizes information regarding the chroma components U and V. Similarly, the (one or more) chroma downsamplings (lower branch) utilize information regarding the luma Y. Otherwise, the respective sampling processes for the luma and the (one or more) chroma are performed in the same manner as discussed above with reference to FIG. 18.

[0213] Therefore, regardless of whether luma upsampling and / or chroma downsampling is performed, in either case, cross information is used for each sampling. This means that the correlation between the luma and the (one or more) chroma can be utilized to reconstruct the channel. The same can be applied to any channel not limited to the color channels of other color spaces, or generally, additional information related to depth, motion, or other pictures, etc.

[0214] Figures 27 and 28 each show an exemplary implementation of a size conversion network for luma upsampling and chroma downsampling on the decoder side. As shown in Figures 27 and 28, the size conversion network has the same design with respect to the order in which convolutional layers and activation layers are arranged as those shown in Figures 21 and 22. The difference is that in the case of Figure 27, the chroma components U and V are directly input into the first convolutional layer Conv. N×K×K, Act. together with the downsampled luma. Similarly, in the case of Figure 28, the luma is directly input into the first convolutional layer Conv. N×K×K, Act together with the upsampled chroma components U and V.

[0215] This means that on the decoder side, the first processing layer of the size conversion network may not need to perform an addition operation, in contrast to the encoder side, and such types of operations can be performed as shown in Figures 25 and 26. The combination (combination or combination processing) in the encoder's size equalization network helped to utilize the inter-component correlation. There is no need to invert such processing on the decoder side because, as illustrated in Figure 24, the input to the first layer of the decoder's size conversion network, which includes the luma and chroma components, already has the same size as obtained in the encoder.

[0216] As described above, how picture processing is performed on the decoding side can be determined in various ways. This is discussed next.

[0217] In one exemplary implementation, determining which processing to apply further includes decoding a selection indicator from the bitstream that indicates picture processing, and determining the processing indicated by the selection indicator as the picture processing. This enables efficient signaling of picture processing. Correspondingly, the decoding side can quickly determine which picture processing (i.e., luma downsampling or chroma upsampling) was used on the encoder side based on a simple indicator contained in the bitstream. The selection indicator is described above with reference to the encoder side. As described above, this can be an indicator that can take at least a first value indicating luma upsampling (on the decoder side, which corresponds to luma downsampling on the encoder side) and a second value indicating chroma downsampling (on the decoder side, which corresponds to chroma upsampling (on the encoder side)). Note that the selection indicator can generally take one of three or more values. For example, the selection indicator may take a third value indicating no resizing and / or a fourth value indicating that two or more components will be resized, or it may further specify how the resizing should be performed (which allocations only, or to which desired resolution). This applies to both the encoder and decoder sides, as they should have the same understanding (interpretation or semantics) of the selection indicator. On the other hand, providing a one-bit flag (indicator) that can take only one of two values ​​(a first value and a second value) may be efficient as it does not introduce high additional signaling overhead.

[0218] The decoder's operation is illustrated in Figure 18. As mentioned earlier, the decoder receives a bitstream with lumen and chroma components of the same size. This is the result of picture processing on the encoder side. If a selection indicator (flag, e.g., RDO) is encoded in the bitstream, the decoder parses the bitstream and decides which picture processing to perform based on the selection indicator. The decoder then upsamples the lumen component if the flag indicates that the lumen has been downsampled on the encoder side. Conversely, the decoder downsamples the chroma component if the flag indicates that the chroma component has been upsampled on the encoder side. In either case, the picture portion of the original size is reconstructed.

[0219] In some embodiments, the determination may include (in addition to or as an alternative to other exemplary implementations) decoding the width W and / or height H of the rumor component of the picture portion from the bitstream. This allows the picture processing to be determined based on the width and height of the rumor channel of the picture portion. In other words, the channel carrying important information of the picture portion is used as a reference for the determination.

[0220] Furthermore, according to one exemplary implementation, the width W and height H are the respective widths of the decoded luma components.

number

number

number

number

number

number

number

number

[0221] It should be noted that this picture processing of the picture portion on the encoding and decoding sides, namely chroma upsampling (encoder) and chroma downsampling (decoder), corresponds to the processing of the first embodiment as far as sampling processing (up / down) is concerned.

[0222] In the above example, if W and H of the original picture do not match

Number

Number

[0223] However, the present disclosure is not limited to this. The default function may instead include upsampling the luma on the decoder side assuming that the encoder has downsampled the luma.

[0224] Correspondingly, as a result of the determination, if either W or H

Number

Number

Number

Number

Number

number

[0225] Therefore, the picture processing involves the widths of the W and H of the ruma and the width of the decoded ruma.

number

number

[0226] It should be noted that both chroma downsampling and luma upsampling described above can be applied by the decoder. For example, the decoder and encoder may have rules that determine which of the two processing types is used. These rules may be based on several conditions that depend on several syntactic elements (encoding parameters) such as width and height, and / or quality parameters or other parameters.

[0227] In another exemplary implementation, determining the processing to be applied involves determining the size S of the bitstream. The size S may be any unit, such as bits, bytes, or words. Determining the latent representation of the picture portion from the bitstream.

number

number

number

number

number

number

number

[0228] As a result of determining the processing to be applied, if the ratio is smaller than a predetermined threshold, in one exemplary implementation, the decoded luma component is upsampled. In this case, the luma is downsampled on the encoding side (low-resolution result).

[0229] Therefore, picture processing is determined by the bitstream size and the size of the latent space.

number

[0230] This is shown in Figure 30. In this case, the decoder decides which picture processing to perform on the bitstream. This decision may be made, for example, by a decision module that parses the bitstream. For example, if the width W and height H of the rumor component of the picture portion are encoded in the bitstream, the decoder parses W and H from the bitstream. Furthermore, the rumor and chroma components are decoded from the bitstream. Thus, the width of the decoded rumor component

number

number

number

number

number

number

[0231] Alternatively or additionally, the decision module considers the size S of the bitstream and the latent space representation.

number

number

number

number

[0232] If so, i.e., R>=T, the decoded chroma component is downsampled. In this case, the chroma is upsampled on the encoding side (high-resolution result). Otherwise (i.e., R>T), the decoded luma component is upsampled. In this case, the luma is downsampled on the encoding side (low-resolution result). According to one implementation example, the determination is made from the bitstream to the latent representation

number

[0233] Such neural network-based decisions may also be made by a decision module, in which case the module decides on picture processing based on the classification result. Classification is performed, for example, by reconstructing the latent space as input.

number

[0234] For example, chroma upsampling (on the encoding side) may include inserting a line of sample after every n lines of the sample of the chroma component, where lines refer to rows and / or columns, and n is an integer greater than or equal to 1. This means, for example, that the line samples may be inserted after each one (n=1) or two (n=2) rows and / or columns. This allows for further adaptation of chroma upsampling by inserting line samples at specific rows and / or columns. A similar process may be applied to the chroma on the decoder side.

[0235] Alternatively, chroma upsampling (on the encoding side) may involve inserting a line of sample with a value of 0 after every n lines of sample of the chroma component. This may be called zero-filling. This is a simple method of upsampling by simply setting each line sample to 0. Therefore, no further signaling and / or processing overhead is required. A similar process may be applied to the chroma on the decoder side.

[0236] This is illustrated in Figure 9, which shows 4x4 samples of upsampled chroma. Black circles indicate samples of the original chroma. White circles indicate samples with a value of 0. In this example, zero-value samples are inserted after each of the first rows and columns of the chroma component. Furthermore, in this example, the size equalization network is simplified to a single-layer network (M=1) with no further layers after the “addition” operation in Figure 5. In Figure 9 (encoder side), the upsampling of U and V is simplified as filling each row and each column with 0 for the YUV420 color format (for the YUV422 format, the zero filling is for each column). In this case, the network is not a convolutional neural network, but can be understood as one layer of a fully connected neural network (FCNN). Therefore, this network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to a trained network.

[0237] In another implementation, chroma upsampling can be performed by inserting a line of sample with a value equal to the nearest neighbor sample in the chroma component after every n lines of sample in the chroma component. The value insertion may be performed, for example, by copying the chroma value to each neighbor sample. Similar to the zero-filling described above, this allows for further adaptation of chroma upsampling by inserting line samples in specific rows and / or columns. Furthermore, using the values ​​of the inserted line samples that reference the nearest neighbor samples of the chroma makes the accuracy of chroma upsampling more precise because it uses neighbor information that references the original chroma (reflected by the neighbor chroma sample values).

[0238] This is illustrated in Figure 11, which shows a 4x4 sample of the upsampled chroma. Again, the black circles point to the original chroma samples. Here, the white circles point to samples with values ​​corresponding to each chroma neighbor sample. In Figure 11, the insertion of chroma values ​​into each neighbor sample is indicated by arrows. In this example, the sample values ​​are inserted after each of the first rows and columns of the chroma component. Furthermore, in this example, the size equalization network is simplified to a single-layer network (M=1) with no further layers after the “addition” operation in Figure 5. In Figure 11 (encoder side), the upsampling of U and V is simplified as upsampling of the nearest neighbor samples in each row and each column, as shown in Figure 11 for the YUV420 format. For the YUV422 format, the upsampling of nearest neighbor samples is for each column. In the above example of chroma upsampling, the network is not a convolutional neural network, but can be understood as a single layer of a fully connected network. Therefore, although this network itself is not trainable, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to the trained network.

[0239] However, this is merely an example of a trained network for size equalization having one layer. As will be apparent to those skilled in the art, the size equalization layer may have two or more layers, as shown through Figures 5 and 7. In this case, the size equalization network may be trained to perform chroma upsampling.

[0240] Similar picture processing that upsamples the luma on the decoding side can be performed, at least in terms of filling every n lines with some additional samples. However, it may be advantageous to perform luma upsampling by using an interpolation filter, i.e., by determining the value of the added luma samples as a function of neighboring samples, for example, as a mean or weighted average. It may also be advantageous to perform luma upsampling by using a trained (pre-trained) network, such as an ML / CL-based network.

[0241] For example, chroma downsampling (on the decoder side) may be performed by removing sample lines after every n lines of the intermediate chroma component sample, where lines are rows and / or columns, and n is an integer greater than or equal to 1. This means, for example, that after each one (n=1) or two (n=2) rows and / or columns, the lines (rows and / or columns) may be removed (deleted). Thus, downsampling of intermediate chroma can be performed quickly by simply removing each line sample.

[0242] Examples of line sample removal are shown in Figures 10 and 12. Note that the sample configurations in Figures 10 and 12 correspond to the sample configurations in Figures 9 and 11, respectively, as a result of the upsampling process of the chroma component on the encoder side. Specifically, Figures 10 and 12 show 4x4 samples of the intermediate chroma. Black circles indicate samples of the original chroma. White circles marked with an "x" indicate samples that may be non-zero as a result of chroma upsampling by the size equalization network on the encoding side (with and without the use of chroma as side information) and / or chroma processing via the first (one or more) layers of the size transformation network on the decoding side.

[0243] In the examples shown in Figures 10 and 12, the size transformation network is simplified as a single-layer network (M=1). In Figures 10 and 12 (decoder side), U and V downsampling is simplified to discard odd-numbered row and column samples for the YUV420 color format. For YUV422, odd-numbered column samples are removed. In both of these cases shown in Figures 10 and 12, the respective networks are not convolutional neural networks, but can be understood as a single layer of a fully connected neural network (FCNN). Therefore, in this example, the network itself is not trainable. However, it can be trained together with a VAE structure for better coding performance. Thus, each FCNN can also refer to the trained network.

[0244] However, this is merely an example of a trained network for size transformation having one layer. As will be apparent to those skilled in the art, the size transformation layer may have two or more layers, as shown through Figures 6 and 8. In this case, the size transformation network may be trained to perform chroma upsampling.

[0245] It should be noted that the encoder-side size equalization network shown in Figure 5 and the decoder-side size transformation network shown in Figure 6 can be trained together using a VAE structure. This can further improve the encoding-decoding process of subsampled chroma components, particularly within the VAE framework. This includes, for example, encoding-decoding performance and / or the quality of the reconstructed picture / image.

[0246] Alternatively, they may be trained separately and used as pre-trained VAE networks. In either case, each network refers to a trained or pre-trained network. Similar picture processing may be performed on the encoding side to downsample the luma. Generally, training may be performed beforehand, i.e., before the network is used in the inference phase for encoding, upsampling, decoding, etc.

[0247] Furthermore, as already described, the disclosure also provides a device configured to perform the steps of the method described above.

[0248] Figure 33 shows an encoder 3300 for encoding a picture portion. The picture portion includes a rumor component and a chroma component having a resolution lower than that of the rumor component. The encoder includes a selection module 3310 configured to select, as a picture processing of the picture portion, either downsample the rumor component to obtain a downsampled rumor component that matches the resolution of the chroma component, or upsample the chroma component to obtain an upsampled chroma component that matches the resolution of the rumor component. Furthermore, the encoder includes a processing module 3320 configured to process the picture portion according to the selected picture processing. The encoder further includes a generation module 3330 configured to generate a bitstream according to the selected picture processing, the generation of which includes encoding the processed picture portion.

[0249] In one implementation example, the encoder may include one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the encoder is configured to perform the above method when the program is executed by the one or more processors.

[0250] Figure 34 shows a decoder 3400 for reconstructing a picture portion from a bitstream. The picture portion includes a lumen component and a chroma component having a lower resolution than the lumen component. The decoder includes a decoding module 3410 configured to decode the lumen and chroma components having the same resolution from the bitstream. Furthermore, the decoder includes a determination module 3420 configured to determine, as a picture process, either upsample the decoded lumen component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled lumen component, or downsample the decoded chroma component to obtain a reconstructed picture portion including the decoded lumen component and the downsampled chroma component. The decoder further includes an acquisition module 3430 configured to obtain the picture portion according to the determined picture process.

[0251] In one implementation example, the decoder may comprise one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the decoder is configured to perform the above method when the program is executed by the one or more processors.

[0252] A computer program stored on a non-temporary medium, which, when executed on one or more processors, includes code that performs any of the steps of the above method, according to one embodiment of the present disclosure.

[0253] Some embodiments of this disclosure may offer advantages that can be summarized as follows: • Luma sample information is fully preserved. • Luma spatial information is completely preserved. Size-equalizing CNNs and size-transforming CNNs can be trained together using a VAE network structure. Simultaneous training may further improve the results. Alternatively, size-equalizing CNNs and size-transforming CNNs may also be trained separately from the VAE. Other types of chroma subsampling formats (e.g., YUV422) can be handled by VAE network structures. • Inter-component lumens-chromatic information can be preserved. Adaptive switching between chroma downsampling for low resolutions to save bitrate and chroma upsampling for high resolutions to preserve chroma information.

[0254] Several exemplary implementations in hardware and software A corresponding system capable of deploying the above encoder-decoder processing chain is shown in Figure 35. Figure 35 is a schematic block diagram illustrating exemplary coding systems, e.g., video, image, audio, and / or other coding systems (or simply coding systems) that may utilize the technology of the present application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video coding system 10 represent an example of a device that may be configured to perform the technology according to the various examples described in the present application. For example, the video coding and video decoding may use neural networks as shown in Figures 2A to 2C, these neural networks may be distributed and may include the above encoded bitstream, which is evaluated in advance whether the chroma is upsampled (Embodiment 1) or whether to upsample the chroma or downsample the chroma (Embodiment 2).

[0255] As shown in Figure 35, the coding system 10 includes, for example, a source device 12 configured to provide encoded picture data 21 to a destination device 14 for decoding encoded picture data 13.

[0256] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22.

[0257] It should be noted that Embodiments 1 and 2 of this disclosure, and their exemplary implementations described above in relation to encoding, may be implemented as part of the encoder 20. However, alternatively, these embodiments may be implemented by the preprocessor 18. Similarly, on the decoder side, Embodiments 1 and 2 of this disclosure, and their exemplary implementations described above in relation to decoding, may be implemented as part of the decoder 30. However, alternatively, these embodiments may be implemented by the postprocessor 32.

[0258] The picture source 16 comprises, or may comprise, any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also comprise any type of memory or storage for storing any of the aforementioned pictures.

[0259] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 may also be called the raw picture or raw picture data 17.

[0260] The preprocessor 18 is configured to receive (raw) picture data 17, perform preprocessing on the picture data 17, and obtain a preprocessed picture 19 or preprocessed picture data 19. Preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an optional component. It should also be noted that preprocessing may also use a neural network (for example, as shown in any of Figures 1 to 7). As described above, preprocessing may include processing by a neural network (a trained network) to upsample the chroma (Embodiment 1 and / or Embodiment 2) or downsample the luma (Embodiment 2).

[0261] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0262] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) to another device, such as the destination device 14 or any other device, via the communication channel 13 for storage or direct reconstruction.

[0263] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0264] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) from, for example, the source device 12, or directly from any other source, such as a storage device, such as a storage device for encoded picture data, and to provide the encoded picture data 21 to the decoder 30.

[0265] Communication interfaces 22 and 28 may be configured to send and receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network or any combination thereof, any type of private and public network, or any type of combination thereof.

[0266] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.

[0267] The communication interface 28 forms the other end of the communication interface 22 and may be configured, for example, to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0268] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows on the communication channel 13 in Figure 35 pointing from the source device 12 to the destination device 14, for example, to send and receive messages, to set up connections, and to confirm and exchange any other information relating to the communication link and / or data transmission, such as the transmission of encoded picture data. The decoder 30 is configured to receive the encoded picture data 21 and provide decoded picture data 31 or the decoded picture 31 (for example, using a neural network based on one or more of Figures 1 to 7).

[0269] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example the decoded picture 31, to obtain post-processed picture data 33, for example the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., YCbCr to RGB), color correction, cropping, or resampling, or any other processing, to prepare the decoded picture data 31 for display by, for example, the display device 34.

[0270] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to, for example, a user or viewer. The display device 34 may be any type of display for representing the reconstructed picture, for example, an integrated or external display or monitor, or may include such a display. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0271] Although Figure 35 depicts the source device 12 and the destination device 14 as separate devices, the device embodiment may also comprise both or both functions, i.e., the source device 12 or its corresponding function and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0272] As will be apparent to those skilled in the art based on the description, the functions or the presence and (exact) division of different units within the source device 12 and / or destination device 14 shown in Figure 35 may vary depending on the actual device and application.

[0273] An encoder 20 (e.g., a video encoder 20) or a decoder 30 (e.g., a video decoder 30), or both an encoder 20 and a decoder 30, may be implemented via processing circuitry such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including neural networks, such as those shown in any or part of Figures 2A to 2C. The decoder 30 may be implemented via processing circuitry 46 to embody various modules discussed with respect to Figures 2A to 2C and / or any other decoder systems or subsystems described herein. The processing circuitry may be configured to perform various operations, as described below. Where the technology is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium and execute those instructions in hardware using one or more processors to perform the technology of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a combined encoder / decoder (CODEC) within a single device, for example, as shown in Figure 36.

[0274] The source device 12 and destination device 14 may comprise any type of handheld or stationary device, including a wide range of devices such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video gaming console, a video streaming device (such as a content service server or content distribution server), a broadcast receiving device, or a broadcast transmitting device, and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Thus, the source device 12 and destination device 14 may be wireless communication devices.

[0275] In some cases, the video coding system 10 illustrated in Figure 35 is merely an example, and the technology of this application may be applied to video coding configurations (e.g., video coding and video decoding) that do not necessarily involve data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory or streamed over a network. A video encoding device may encode the data and store it in memory, and / or a video decoding device may retrieve the data from memory and decode it. In some examples, encoding and decoding are performed by devices that simply encode the data into memory and / or retrieve the data from memory and decode it, without communicating with each other.

[0276] Figure 37 is a schematic diagram of a video coding device 1000 according to one embodiment of the present disclosure. The video coding device 1000 is suitable for implementing embodiments of the disclosure described herein. In one embodiment, the video coding device 1000 may be a decoder, such as the video decoder 30 in Figure 35, or an encoder, such as the video encoder 20 in Figure 35.

[0277] The video coding device 1000 comprises an inlet port 1010 (or input port 1010) and a receiver unit (Rx) 1020 for receiving data, a processor, logic unit, or central processing unit (CPU) 1030 for processing data, a transmitter unit (Tx) 1040 and an exit port 1050 (or output port 1050) for transmitting data, and memory 1060 for storing data. The video coding device 1000 may also comprise optical-electrical (OE) and electrical-optical (EO) components coupled to the inlet port 1010, receiver unit 1020, transmitter unit 1040, and exit port 1050 for the entry and exit of optical or electrical signals.

[0278] The processor 1030 is implemented by hardware and software. The processor 1030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 1030 communicates with the input port 1010, the receiver unit 1020, the transmitter unit 1040, the output port 1050, and the memory 1060. The processor 1030 includes a coding module 1070. The coding module 1070 implements embodiments of the disclosure described above. For example, the coding module 1070 implements, processes, prepares, or provides various coding operations. Thus, including the coding module 1070 provides a significant improvement to the functionality of the video coding device 1000, resulting in the conversion of the video coding device 1000 to different states. Alternatively, the coding module 1070 is implemented as instructions stored in the memory 1060 and executed by the processor 1030.

[0279] Memory 1060 may comprise one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device for storing such programs when they are selected for execution, and for storing instructions and data read during program execution. Memory 1060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random-access memory (RAM), tri-level associative memory (TCAM), and / or static random-access memory (SRAM).

[0280] Figure 38 is a simplified block diagram of a device 800 that can be used as either or both of the source device 12 and destination device 14 in Figure 35, according to one exemplary embodiment.

[0281] The processor 1102 within the device 1100 may be a central processing unit. Alternatively, the processor 1102 may be any other type of device, existing or to be developed in the future, capable of manipulating or processing information, or a number of such devices. The implementation of the disclosure may be carried out using a single processor, e.g., processor 1102, as shown in the illustration, but advantages in speed and efficiency may be achieved by using two or more processors.

[0282] The memory 1104 within the device 1100 may be a read-only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as memory 1104. Memory 1104 may contain code and data 1106 accessed by the processor 1102 using the bus 1112. Memory 1104 may further contain an operating system 1108 and an application program 1110, the application program 1110 including at least one program that enables the processor 1102 to perform the methods described herein. For example, the application program 1110 may include applications 1 through N, the applications further including a video coding application that performs the methods described herein.

[0283] The device 1100 may also include one or more output devices, such as a display 1118. In one example, the display 1118 may be a touch-sensitive display that combines a display with a touch-sensitive element that can operate to sense touch input. The display 1118 may be coupled to the processor 1102 via a bus 1112.

[0284] Although described here as a single bus, the bus 1112 of device 1100 can consist of multiple buses. Furthermore, secondary storage can be directly coupled to other components of device 1100 or accessed via a network, and can comprise a single integrated unit such as a memory card or multiple units such as multiple memory cards. Thus, device 1100 can be implemented in a wide variety of configurations.

[0285] Overview of some exemplary embodiments Some embodiments of this disclosure provide methods and apparatus for encoding and decoding picture portions having a lumen component and a subsampled chroma component as input. Since the lumen component can be processed by downsampling, the encoding-decoding process can be performed more quickly at a lower bitrate and may result in a lower resolution. The chroma component can be processed by chroma upsampling to provide a high-resolution result. Spatial correlation between the lumen and chroma can also be preserved in this way. In either case, the resized lumen-chroma components have a matched resolution, and encoding by a trained network is applied to the matched components. The encoder may adaptively select between lumen downsampling and chroma upsampling, and the decoder performs the reverse picture processing of lumen upsampling and chroma downsampling. Thus, the encoding-decoding process can provide either a low-resolution or high-resolution picture / image depending on the selected picture processing.

[0286] According to one aspect of the present disclosure, a method for encoding a picture portion, the picture portion comprising a rumar component and a chroma component having a resolution lower than the resolution of the rumar component, the method comprising: selecting as processing the picture portion one of the following steps: downsampling the rumar component to obtain a downsampled rumar component that matches the resolution of the chroma component, and upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the rumar component; processing the picture portion according to the selected picture processing; and generating a bitstream according to the selected processing, the generated bitstream comprising encoding the processed picture portion.

[0287] This provides a lumen-chroma output of equal size, regardless of whether the lumen component is downsampled or the chromen component is upsampled. Thus, the output can then be processed by a trained network (e.g., within a machine learning framework) with a similar structure, particularly one with the same input resolution. If all color components have the same resolution, the machine learning framework can utilize cross-color correlation in the same way at all sample locations. In one of the picture processes, the lumen channel may remain unaffected, in which case there is no loss of quality.

[0288] Downsampling the lumens is beneficial for saving bitrate at low resolutions, while upsampling the chromens is beneficial for preserving lumens information at high resolutions. Therefore, adaptive switching allows for the selection of sampling modes according to requirements and / or limitations on bitrate or picture quality. In either case of selected picture processing of the picture portion, the processed lumens-chromens components have matching resolutions, so the output can be handled within a VAE framework or another framework that enables efficient encoding.

[0289] In some exemplary implementations, the selection step involves evaluating the cost functions of rate and distortion. Thus, the selection of the sampling mode (i.e., lumar downsampling and chroma upsampling) is cost-based.

[0290] According to an exemplary implementation of the method, the selection step includes downsampling the lumar component to obtain an intermediate downsampled lumar component, and upsampling the chroma component to obtain an intermediate upsampled chroma component. In either case, the selection of sampling modes, including lumar downsampling and chroma upsampling, provides intermediate lumar and intermediate chroma components having matching resolutions (i.e., sizes), respectively. The equal-sized lumar-chroma components may then undergo further processing, for example, by a VAE.

[0291] In a further implementation of the method, the selection step includes determining the costs of the intermediate downsampled lumar and chroma components, as well as the costs of the lumar component and the intermediate upsampled chroma component, respectively, according to the cost function, wherein the picture processing of the picture portion with the lower determined cost is selected. Thus, by using the sampling mode with the lowest cost (i.e., lumar downsampling or chroma upsampling), the overall encoding of the lumar and subsampled chroma components is performed efficiently.

[0292] According to one implementation example of the method, the determination step includes encoding the intermediate downsampled lumen and chroma components into a first bitstream, encoding the lumen component and the intermediate upsampled chroma component into a second bitstream, and determining the costs of the first and second bitstreams according to the cost function. Thus, the actual costs can be accurately determined because they are based on the actual encoded first and second bitstreams.

[0293] In one example of the method, the step of generating a bitstream includes the step of encoding a selection indicator in the bitstream that indicates the selected picture processing. This provides low-complexity signaling of the picture processing performed on the bitstream.

[0294] In another example, the step of generating a bitstream includes encoding instructions for the width W and / or height H of the rumor component into the bitstream. Thus, the width and height of the original picture portion can be signaled to the encoder.

[0295] In an exemplary implementation of the method, upsampling and / or downsampling are performed by a trained network. Coding using a trained network can offer improved performance thanks to the possibility of training such a network with the desired data.

[0296] For example, the encoding of the processed picture portion is performed by a variational autoencoder (VAE). VAEs can be advantageous for performing tasks related to encoding and / or decoding images and / or pictures very efficiently, and they offer a good compromise between the required bitrate and the quality of the reconstructed image or picture.

[0297] According to one aspect of the present disclosure, a method is provided for reconstructing a picture portion from a bitstream, wherein the picture portion comprises a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the method includes: decoding a rumor component and a chroma component having the same resolution from a bitstream; determining from the bitstream, as a picture process, one of the following: upsampling the decoded rumor component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled rumor component, or downsampling the decoded chroma component to obtain a reconstructed picture portion including the decoded rumor component and the downsampled chroma component; and obtaining the picture portion according to the determined picture process.

[0298] Therefore, the determined picture processing, either lumar upsampling or chroma downsampling, in either case provides resized lumar or chroma components having different sizes corresponding to the components input for encoding (the original picture components). Thus, the picture portion is reconstructed to its original color format, and the chroma has a lower resolution than the chroma. As a result, the encoded equal-sized lumar-chroma channels can be handled within a common VAE framework, regardless of the determined picture processing. In general, the advantages provided on the decoding side may be similar to the advantages provided on the encoder side by the corresponding additional features of each encoding method.

[0299] According to one implementation example of the method, the determination step further includes decoding a selection indicator from the bitstream that indicates a picture process, and determining the process indicated by the selection indicator as the picture process. This enables efficient signaling of picture processes. Accordingly, the decoding side can quickly determine which picture process was used by the encoder side (i.e., chroma downsampling or chroma upsampling) based on a simple indicator contained in the bitstream.

[0300] According to another exemplary implementation, the determination step further includes decoding the width W and / or height H of the rumor component of the picture portion from the bitstream. This allows the picture processing to be determined based on the width and height of the rumor channel of the picture portion. In other words, the channel carrying important information of the picture portion is used as a reference for the determination.

[0301] In one example, the step to determine the width W and height H of the decoded lumens component is

number

number

number

number

number

number

number

number

[0302] Further implementation examples show that the steps to be determined are: determining the size S of the bitstream, and determining the latent representation of the picture portion from the bitstream.

number

number

number

number

number

number

[0303] In one exemplary implementation, the steps to determine the size S and

number

number

number

[0304] According to another implementation example, the decision step is to determine the latent representation from the bitstream.

number

[0305] In one exemplary implementation, upsampling and / or downsampling are performed by a trained network. By providing a bitstream based on downsampled chroma or upsampled chroma, the decoding side can reconstruct the picture (still or video) at higher quality and / or a lower rate. The trained network provides adaptive and efficient coding, thus delivering improved quality at the same rate or improved rate at the same quality.

[0306] For example, decoding of the processed picture portion is performed by a variational autoencoder (VAE). Therefore, decoding can be performed efficiently, with a good compromise between the required bitrate and / or the quality of the reconstructed picture or image.

[0307] According to one aspect of the present disclosure, a computer program stored on a non-temporary medium is provided which, when executed on one or more processors, includes code that performs any of the steps of the above method.

[0308] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion comprising a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, the encoder comprising one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the encoder is configured to perform the above-described encoding method when the program is executed by the one or more processors.

[0309] According to one aspect of the present disclosure, a decoder is provided for reconstructing a picture portion from a bitstream, wherein the picture portion comprises a rumor component and a chroma component having a resolution lower than that of the rumor component, and the decoder comprises one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the decoder is configured to perform the above-described decoding method when the program is executed by the one or more processors.

[0310] According to one aspect of the present disclosure, an encoder for encoding a picture portion is provided, the picture portion comprising a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the encoder comprises a processing circuit configured to perform the steps of the above-described encoding method.

[0311] According to one aspect of a pre-configured disclosure, a decoder for reconstructing a picture portion from a bitstream, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than that of the rumor component, and the decoder comprises a processing circuit configured to perform the steps of the above-described decoding method.

[0312] According to one aspect of the present disclosure, an encoder is provided for encoding a picture portion, the picture portion comprising a ruma component and a chroma component having a resolution lower than the resolution of the ruma component, the encoder comprising: a selection module configured to select, as a picture processing of the picture portion, downsample the ruma component to obtain a downsampled ruma component that matches the resolution of the chroma component, and upsample the chroma component to obtain an upsampled chroma component that matches the resolution of the ruma component; a processing module configured to process the picture portion according to the selected picture processing; and a generation module configured to generate a bitstream according to the selected picture processing, the generation of which includes encoding the processed picture portion.

[0313] According to one aspect of the present disclosure, a decoder is provided for decoding a picture portion from a bitstream, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the decoder comprises: a decoding module configured to decode a rumor component and a chroma component having the same resolution from a bitstream; a determination module configured to determine, as a picture process from the bitstream, either upsample the decoded rumor component to obtain a reconstructed picture portion including the decoded chroma component and the upsampled rumor component, or downsample the decoded chroma component to obtain a reconstructed picture portion including the decoded rumor component and the downsampled chroma component; and an acquisition module configured to obtain the picture portion according to the determined picture process.

[0314] The embodiment allows for adaptive switching between chroma upsampling and luma downsampling of the input picture portion depending on the desired application. For example, some applications aim to reduce complexity or bitrate when luma downsampling may be more advantageous, while others aim for higher quality when chroma upsampling may be more advantageous. Accordingly, the picture portion having luma components and subsampled chroma components can be efficiently encoded and / or decoded in either of the sampling modes of picture processing, including luma downsampling and chroma upsampling.

[0315] Please note that this disclosure is not limited to any particular framework. Furthermore, this disclosure is not limited to image or video compression, but may also apply to object detection, image generation, and recognition systems.

[0316] The present invention can be implemented in hardware and / or software.

[0317] For clarity, any one of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of this disclosure.

[0318] In summary, this disclosure relates to video encoding and decoding, and more particularly to the handling of chroma subsample formats in deep learning-based video(de)coding. The corresponding apparatus and method enable processing for encoding and decoding of each picture portion, which includes a luma component and a chroma component having a lower resolution than the luma component. To handle such luma-chroma channels of different sizes, the chroma component is upsampled such that the resulting upsampled chroma component has a resolution matching one of the luma components. The luma and upsampled chroma components are then encoded into a bitstream. To reconstruct the picture portion, the luma component and an intermediate chroma component matching the resolution of the luma component are decoded from the bitstream, and then the intermediate chroma component is downsampled. Thus, the subsampled chroma format can be handled by an autoencoder / autodecoder framework while preserving the luma channel.

[0319] Furthermore, this disclosure relates to the encoding and decoding of pictures or videos. A picture portion includes a lumen component and a chroma component having a lower resolution than the lumen component. A choice is made between downsampling the lumen component and upsampling the chroma component to obtain a matched resolution for the lumen and chroma components. The selected process is then applied to the picture portion. This then includes generating a bitstream based on the selected process and encoding the processed picture portion. The choice between chroma upsampling and lumen downsampling allows for the adaptation of the input picture portion for a desired application. For example, some applications aim to reduce complexity or bitrate when lumen downsampling may be more advantageous, while others aim for higher quality when chroma upsampling may be more advantageous. [Explanation of Symbols]

[0320] 410 NN Size Equalization Module 420 VAE encoder 470 VAE Decoder 480 Downsampling Network 800 equipment 1500 encoders 1510 Upsampling Module 1520 Encoding Module 1600 Decoder 1610 Decoding Module 1620 Downsampling Module 1710 Luma Downsampling 1720 Chroma Upsampling 1730 Rate Distortion Optimization (RDO) 1740 Function block "Up or Down" 1810 Up-branched luma upsampling 1820 Lower branch chroma downsampling 1840 Functional Blocks 2910 Luma Downsampling 2920 Chroma Upsampling 2930 bitrate and distortion (RDO) 3300 encoders 3310 Selection Module 3320 Processing Module 3330 Generation Module 3400 Decoder 3410 Decoding Module 3420 Decision Module 3430 Acquisition Module 10 Video Coding Systems 12. Source device 13 Communication Channels 14 Destination device 16 Picture Sources 17 Picture Data 18 preprocessors 19 Preprocessed picture data 20 Video Encoders 21 Encoded picture data 22 Communication Interfaces 28 Communication Interfaces 30 video decoders 31 Decoded picture data 32 Post-Processors 33 Post-processed picture data 34 Display Devices 46 Processing Circuit 1000 video coding devices 1010 Entrance Port 1020 Receiver Unit 1030 processor 1040 Transmitter Unit 1050 Exit Port 1060 memory 1070 Coding Module 1100 equipment 1102 Processor 1104 memory 1106 Codes and Data 1108 Operating Systems 1110 Application Program 1112 Bus 1118 Display

Claims

1. A method for encoding a picture portion, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the method is The steps include: upsampling the chroma component to obtain an upsampled chroma component that matches the resolution of the chroma component; The steps include encoding the chroma component and the upsampled chroma component into a bitstream. Includes, The encoding step is performed by a variational autoencoder (VAE). A method wherein the step of upsampling further comprises the step of processing the chroma component by a trained network, the trained network being trained together with the VAE.

2. The upsampling of the chroma component is based on a sample of the luma component, according to claim 1.

3. The method according to claim 1, wherein the step of processing by the trained network includes inputting the lumens and chromens components into the trained network and outputting the upsampled chromens component from the trained network.

4. The trained network comprises at least two processing layers, The first of the at least two processing layers performs upsampling based on the input chroma component. The method according to claim 1 or 3, wherein the second of the at least two processing layers processes the output of the first layer together with the luma component.

5. The method according to claim 4, wherein processing the output of the first layer together with the rumor component includes adding the rumor component and the upsampled chroma component element by element.

6. The method according to claim 1, wherein the step of upsampling includes the step of inserting a line of sample after every n lines of the chroma component sample, where the lines are rows and / or columns and n is an integer of 1 or more.

7. The method according to claim 1 or 6, wherein the step of upsampling includes inserting a line of sample having a value of 0 after every n lines of the chroma component sample, wherein the lines are rows and / or columns and n is an integer of 1 or more.

8. The method according to claim 1 or 6, wherein the step of upsampling includes inserting a line of sample having a value equal to the nearest neighbor sample in the chroma component after every n lines of the sample in the chroma component, wherein the lines are rows and / or columns and n is an integer of 1 or more.

9. A method for decoding a picture portion from a bitstream, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the method is The steps include decoding the lumens component and an intermediate chroma component that matches the resolution of the lumens component from the bitstream, The steps include downsampling the intermediate chroma component to obtain the chroma component of the picture portion, and Includes, The decoding step is performed by a variational autoencoder (VAE). A method wherein the step of downsampling further comprises processing the intermediate chromatic components by a trained network, the trained network being trained together with the VAE.

10. The method according to claim 9, wherein the step of processing by the trained network includes inputting the intermediate chroma component into the trained network and outputting the chroma component from the trained network.

11. The trained network comprises at least two processing layers, The first of the at least two processing layers performs processing on the input intermediate chroma component. The method according to claim 9 or 10, wherein the second of the at least two processing layers performs downsampling of the output of the first layer.

12. The method according to claim 11, wherein the first layer is processed together with the luma component to obtain the intermediate chromatic component.

13. The downsampling of the intermediate chroma component is based on a sample of the luma component, according to any one of claims 9 to 12.

14. The method according to any one of claims 9 to 13, wherein the step of processing by the trained network includes the step of inputting the luma component into the trained network.

15. The method according to claim 9, wherein the downsampling step includes a step of removing lines of the sample after every n lines of the intermediate chromatic component of the sample, wherein the lines are rows and / or columns and n is an integer of 1 or more.

16. A computer program stored on a non-temporary medium, which includes code that, when executed on one or more processors, performs the method described in any one of claims 1 to 15.

17. An encoder for encoding a picture portion, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the encoder is One or more processors, A non-temporary computer-readable storage medium coupled to one or more processors and storing a program for execution by the one or more processors, wherein the program is configured to perform the method described in any one of claims 1 to 8 when the program is executed by the one or more processors, and An encoder equipped with the following features.

18. A decoder for decoding a picture portion from a bitstream, wherein the picture portion includes a rumor component and a chroma component having a resolution lower than the resolution of the rumor component, and the decoder One or more processors, A non-temporary computer-readable storage medium coupled to one or more processors and storing a program for execution by the one or more processors, wherein the decoder is configured to perform the method described in any one of claims 9 to 15 when the program is executed by the one or more processors. A decoder equipped with a decoder.

Images