Decoding using segmentation information signaling

Abstract

To provide a method and a device for decoding data.SOLUTION: Two or more sets of segmentation information elements are obtained from a bitstream. The two or more sets of segmentation information elements are then input to two or more segmentation information processing layers of a plurality of cascaded layers, respectively. At each of the two or more segmentation information processing layers, a respective set of segmentation information is processed. Decoding data for picture or video processing is obtained on the basis of segmentation information processed by the plurality of cascaded layers. Therefore, data can be decoded from the bitstream in an efficient method in a hierarchical structure.SELECTED DRAWING: Figure 9

Description

[Technical field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure relate generally to the field of decoding data for image or video processing from a bitstream using multiple processing layers. In particular, some embodiments relate to methods and apparatus for such decoding. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically coded block by block by predicting the block and further coding only the difference between the original block and its prediction. In particular, such coding can include transforming, quantizing, and generating a bitstream, and usually includes some entropy coding. Typically, the three components of a hybrid coding method, namely transforming, quantizing, and entropy coding, are optimized separately. Modern video compression standards such as High Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use a transformed representation to code the residual signal after prediction.

[0003] Recently, machine learning has been applied to image and video coding. Generally, machine learning can be applied to image and video coding in various different ways. For example, some end-to-end optimized image or video coding schemes have been discussed. Furthermore, machine learning has been used to determine or optimize some parts of end-to-end coding, such as prediction parameter selection or compression. These applications have in common that they generate some feature map data that is communicated between the encoder and the decoder. An efficient structure of the bitstream can greatly contribute to the reduction of the number of bits to code the image / video source signal.

[0004] A neural network usually includes two or more layers. A feature map is the output of a layer. In a neural network that is split between devices, for example, between an encoder and a decoder, between a device and a cloud, or between different devices, the feature map at the output of the split location (e.g., a first device) is compressed and sent to the remaining layers of the neural network (e.g., a second device).

[0005] It may be desirable to use the trained network architecture to further improve the encoding and decoding. Summary of the Invention [Means for solving the problem]

[0006] Some embodiments of the present disclosure provide methods and apparatus for decoding pictures in an efficient manner and allowing some scalability to adapt to desired parameters and content.

[0007] These and other objects are achieved by the subject matter of the independent claims. Further implementations are evident from the dependent claims, the description and the drawings.

[0008] According to one aspect, there is provided a method for decoding data for picture or video processing from a bitstream, the method comprising the steps of: obtaining two or more sets of segmentation information elements from the bitstream; inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers of a plurality of cascaded layers, respectively; and processing each of the two or more segmentation information processing layers, wherein the step of obtaining the decoded data for picture or video processing is based on the segmentation information processed by the plurality of cascaded layers.

[0009] Such a method may provide improved efficiency since it allows for decoding of data in various segments that may be configured on a layer-by-layer basis in a hierarchical structure. The provision of segments may take into account characteristics of the data to be decoded.

[0010] For example, the step of obtaining a set of segmentation information elements is based on segmentation information processed by at least one segmentation information processing layer of the plurality of cascaded layers.

[0011] In some exemplary embodiments, the step of inputting the set of segmentation information elements is based on processed segmentation information output by at least one of the plurality of cascaded layers.

[0012] Cascaded segmentation information processing allows for efficient analysis of the segmentation information.

[0013] For example, the segmentation information processed in each of the two or more segmentation information processing layers has different resolutions.

[0014] In some embodiments and examples, processing the segmentation information in two or more segmentation information processing layers includes upsampling.

[0015] The hierarchical structure of the segmentation information allows for a small amount of side information to be inserted into the bitstream, thus improving efficiency and / or processing time.

[0016] In particular, upsampling the segmentation information includes nearest neighbor upsampling. Nearest neighbor upsampling has a low computational complexity and can be easily implemented. Moreover, it is efficient especially for logical representations such as flags. For example, upsampling the segmentation information includes transposed convolution. By performing upsampling, the upsampling quality can be improved. Further, such a convolutional upsampling layer may be provided as trainable or configurable in a decoder, and as a result, the convolutional kernel may be controlled by instructions parsed from a bitstream or derived in other ways.

[0017] In an exemplary implementation, for each segmentation information processing layer j of a plurality of N segmentation information processing layers among a plurality of cascade layers, the input step, when j = 1, is to input initial segmentation information from a bitstream, and otherwise, to input the segmentation information processed by the (j - 1)-th segmentation information processing layer, and to output the processed segmentation information.

[0018] For example, the step of processing the segmentation information input by each layer j < N of a plurality of N segmentation information processing layers is to parse segmentation information elements from a bitstream and associate the parsed segmentation information elements with the segmentation information output by the previous layer, wherein the position of the parsed segmentation information elements within the associated segmentation information is determined based on the segmentation information output by the previous layer, and further includes the step of determining the amount of segmentation information elements parsed from the bitstream based on the segmentation information output by the previous layer. For example, the parsed segmentation information elements are represented by a set of binary flags.

[0019] Such a hierarchical structure provides processing that may be parallelizable and easily run on GPU / NPU, allowing one to exploit parallelism. A fully trainable scheme for gradient transfer allows its use in an end-to-end trainable video coding solution.

[0020] In some example embodiments and examples, obtaining decoding data for picture or video processing includes determining, based on the segmentation information, at least one of the following: intra-picture or inter-picture prediction mode, picture reference index, single-reference or multiple-reference prediction (including bi-prediction), present or absent prediction residual information, quantization step size, motion information prediction type, motion vector length, motion vector resolution, motion vector prediction index, motion vector difference size, motion vector difference resolution, motion interpolation filter, in-loop filter parameters, post-filter parameters. The decoding of this disclosure is quite generally applicable to any kind of data related to picture or video coding.

[0021] The method of the above embodiment or example further includes: obtaining a set of feature map elements from the bitstream, and inputting the set of feature map elements into a feature map processing layer among the multiple layers respectively based on the segmentation information processed by the segmentation information processing layer; and obtaining decoded data for picture or video processing based on the feature maps processed by the multiple cascade layers.

[0022] In particular, at least one of the multiple cascade layers is a segmentation information processing layer and a feature map processing layer. In other embodiments, each layer of the multiple layers is either a segmentation information processing layer or a feature map processing layer.

[0023] The separated layer functions provide a clean design and functional separation, however, the present disclosure can also work when layers implement both functions.

[0024] According to one aspect, a computer program product stored on a non-transitory medium is provided which, when executed on one or more processors, performs a method according to any of the above examples and embodiments.

[0025] According to one aspect, there is provided a device for decoding images or videos, the device including a processing circuit configured to perform a method according to any of the above examples and embodiments.

[0026] According to one aspect, a device for decoding data for picture or video processing from a bitstream is provided, the device including: an acquisition unit configured to acquire two or more sets of segmentation information elements from the bitstream; an input unit configured to input each of the two or more sets of segmentation information elements to two or more segmentation information processing layers of a plurality of cascaded layers, respectively; a processing unit configured to process each set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data acquisition unit configured to acquire the decoded data for picture or video processing based on the segmentation information processed in the plurality of cascaded layers.

[0027] Any of the above mentioned devices can be embodied on an integrated chip. The present invention can be implemented in hardware (HW) and / or software (SW). Furthermore, HW-based implementations can be combined with SW-based implementations.

[0028] It should be noted that the present disclosure is not limited to any particular framework. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, image generation, and recognition systems.

[0029] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0030] In the following, embodiments of the invention are explained in more detail with reference to the accompanying drawings. [Brief description of the drawings]

[0031] [Figure 1] FIG. 2 is a schematic diagram showing channels processed by layers of a neural network. [Diagram 2] FIG. 1 is a schematic diagram showing an autoencoder type neural network. [Figure 3A] FIG. 1 is a schematic diagram illustrating an example network architecture for the encoder and decoder side including a hyper-prior model. [Figure 3B] FIG. 1 is a schematic diagram showing a general network architecture for the encoder side including a hyper-priori model. [Figure 3C] FIG. 1 is a schematic diagram showing a general network architecture for the decoder side including a hyper-priori model. [Figure 4] FIG. 1 is a schematic diagram illustrating an example network architecture for the encoder and decoder side including a hyper-prior model. [Figure 5A] FIG. 1 is a block diagram illustrating an end-to-end video compression framework based on neural networks. [Figure 5B] FIG. 2 is a block diagram illustrating some example details of the application of neural networks for motion field compression. [Figure 5C] FIG. 2 is a block diagram illustrating some example details of the application of a neural network for motion compensation. [Figure 6] FIG. 1 is a schematic diagram of a layer of U-Net. [Figure 7A] FIG. 2 is a block diagram illustrating an example hybrid encoder. [Figure 7B]FIG. 2 is a block diagram illustrating an exemplary hybrid decoder. [Figure 8] 4 is a flowchart illustrating an exemplary method for encoding data for picture / video processing, such as encoding. [Figure 9] FIG. 1 is a block diagram showing the structure of a network for transporting information from different resolution layers in a bitstream. [Figure 10A] FIG. 1 is a schematic diagram showing max pooling. [Figure 10B] FIG. 1 is a schematic diagram showing average pooling. [Figure 11] FIG. 1 is a schematic diagram illustrating an example encoder-side processing of feature maps and segmentation information. [Figure 12] FIG. 2 is a block diagram showing generalized processing of motion information feature maps by the encoder and decoder sides. [Figure 13] FIG. 1 is a block diagram showing the structure of a network for transferring information from different resolution layers in a bitstream for processing motion vector related information. [Figure 14] FIG. 2 is a block diagram illustrating an example cost computation unit having high cost tensor resolution. [Figure 15] FIG. 2 is a block diagram illustrating an example cost computation unit having low cost tensor resolution. [Figure 16] FIG. 2 is a block diagram illustrating the functional structure of the signal selection logic. [Figure 17] FIG. 1 is a block diagram illustrating a functional structure of signal selection logic having a cost calculation unit that provides multiple coding options. [Figure 18] FIG. 1 is a block diagram showing the structure of a network for transferring information from layers of different resolutions in a bitstream having convolutional downsampling and upsampling layers. [Figure 19] FIG. 1 is a block diagram showing a structure for transporting information from layers of different resolutions in a bitstream having additional layers. [Figure 20]FIG. 1 is a block diagram showing a structure for transferring information from layers of different resolutions in a bitstream having layers that allow downsampling or upsampling filter selection. [Figure 21] FIG. 1 is a block diagram showing the structure of a network for transferring information from layers of different resolutions in a bitstream having layers that enable convolutional filter selection. [Figure 22] FIG. 1 is a block diagram illustrating a functional structure of a network-based RDO decision unit for selecting a coding mode. [Figure 23] FIG. 13 is a block diagram of an example cost calculation unit that may be used in a network-based RDO decision unit for selecting a coding mode. [Figure 24] FIG. 13 is a block diagram of an example cost calculation unit that may be used in a network-based RDO decision unit for selecting a coding mode that supports multiple options. [Diagram 25] FIG. 2 is a schematic diagram showing possible block divisions or filter shapes; [Figure 26] FIG. 2 is a schematic diagram illustrating the derivation of segmentation information; [Figure 27] FIG. 13 is a schematic diagram illustrating processing of segmentation information at the decoder side. [Figure 28] FIG. 2 is a block diagram illustrating an example signal feeding logic for dense optical flow reconstruction. [Figure 29] FIG. 2 is a block diagram illustrating an example signal feeding logic for dense optical flow reconstruction. [Diagram 30] FIG. 2 is a block diagram illustrating a convolution filter set. [Diagram 31] FIG. 2 is a block diagram illustrating a set of upsampling filters. [Figure 32A] FIG. 1 is a schematic diagram illustrating the upsampling process at the decoder side employing nearest neighbor copying. [Figure 32B] FIG. 1 is a schematic diagram illustrating a decoder-side upsampling process employing a convolution process. [Diagram 33] 1 is a flowchart of an example method for decoding data such as feature map information for use in decoding a picture or video. [Diagram 34] 1 is a flowchart of an example method for encoding data such as segmentation information for use in picture or video encoding. [Diagram 35] 1 is a block diagram illustrating an example of a video coding system configured to implement embodiments of the present invention. [Diagram 36] 2 is a block diagram illustrating another example of a video coding system configured to implement embodiments of the present invention. [Figure 37] FIG. 1 is a block diagram showing an example of an encoding device or a decoding device. [Figure 38] FIG. 13 is a block diagram showing another example of an encoding device or a decoding device. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0032] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and which show, by way of example, certain aspects of embodiments of the invention or in which embodiments of the invention may be used. It is understood that embodiments of the invention may be used in other ways and may include structural or logical changes not depicted in the drawings. Therefore, the following detailed description should not be taken in a limiting sense, and the scope of the present invention is defined by the appended claims.

[0033] For example, it is understood that disclosure related to a described method may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more particular method steps are described, the corresponding device may include one or more units, e.g., functional units, performing one or more of the described method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the multiple steps, respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, when a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps performing one or more of the multiple units, respectively), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that the features of various exemplary embodiments and / or aspects described herein can be combined with each other, unless otherwise specified.

[0034] Some embodiments aim to improve the quality of the encoded and decoded pictures or video data and / or reduce the amount of data required to represent the encoded pictures or video data. Some embodiments provide for efficient selection of information signaled from the encoder to the decoder. In the following, an overview is provided of some of the used technical terms and frameworks in which embodiments of the present disclosure may be employed.

[0035] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems loosely inspired by the biological neural networks that make up animal brains. Such systems "learn" to do tasks by considering examples and are not generally programmed with task-specific rules. For example, in image recognition, they may learn to identify images containing cats by analyzing example images that have been manually labeled as "cat" or "not cat" and using the results to identify cats in other images. They do this without prior knowledge of cats, e.g., that cats have fur, tails, whiskers, and cat-like faces. Instead, they automatically generate discriminative characteristics from the examples they process.

[0036] ANNs are based on a collection of connected units or nodes called artificial neurons that loosely model neurons in a biological brain. Each connection can send signals to other neurons, similar to synapses in a biological brain. An artificial neuron that receives a signal can then process the signal and signal to neurons connected to it.

[0037] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. The connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have a threshold such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal progresses from the first layer (input layer) to the last layer (output layer), possibly after traversing the layer multiple times.

[0038] The original goal of the ANN approach was to solve problems in the same way that the human brain solves them. Over time, attention shifted to performing specific tasks, leading to a deviation from biology. ANNs have been used for a variety of tasks including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even in activities traditionally considered to be exclusively human, such as painting.

[0039] The name "convolutional neural network" (CNN) indicates that the network employs a mathematical operation called convolution, which is a special kind of linear operation. Convolutional networks are neural networks that use convolution instead of the common matrix multiplication in at least one of their layers.

[0040] Figure 1 shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer and an output layer, as well as multiple hidden layers. The input layer is the layer where the input (such as a portion of an image as shown in Figure 1) is provided for processing. The hidden layers of a CNN are typically composed of a series of convolutional layers that convolve with multiplication or other dot products. The result of the layer is one or more feature maps (f.map in Figure 1), sometimes referred to as channels. There may be subsampling involved in some or all of the layers. As a result, the feature maps may be smaller, as shown in Figure 1. The activation function in a CNN is typically a ReLU (rectified linear unit) layer, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, which are referred to as hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layers are colloquially referred to as convolutions, but this is only by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important to the index into the matrix in that it affects how the weight is determined at a particular index point.

[0041] When programming a CNN to process images, the input is a tensor with shape (number of images) x (image width) x (image height) x (image depth), as shown in Figure 1. Then, after passing through a convolutional layer, the image is abstracted into a feature map with shape (number of images) x (feature map width) x (feature map height) x (feature map channels). A convolutional layer in a neural network must have the following attributes: A convolution kernel defined by width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) must be equal to the number of input feature map channels (depth).

[0042] In the past, traditional multi-layer perceptron (MLP) models have been used for image recognition. However, due to the full connections between nodes, MLP models suffer from high dimensionality and do not scale well with high-resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be conveniently handled efficiently at scale with full connections. Also, such network architectures do not take into account the spatial structure of the data, treating input pixels that are far apart the same as pixels that are close to each other. This ignores the locality of reference in image data, both computationally and semantically. Thus, for purposes such as image recognition, where spatially local input patterns predominate, the full connectivity of neurons is wasteful.

[0043] Convolutional neural networks are biologically inspired variants of multi-layer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of CNNs. The layer's parameters consist of a set of learnable filters (kernels, as shown above) that have small receptive fields but extend across the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, computing the dot product between the filter's entries and the input, generating a two-dimensional activation map for that filter. As a result, the network learns filters that activate when it detects some particular type of feature at some spatial location in the input.

[0044] Stacking the activation maps of all filters along the depth dimension forms the full output volume of the convolutional layer. Thus, every entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature map and activation have the same meaning. It is referred to in some papers as an activation map because it is a mapping that corresponds to the activation of different parts of the image, and also as a feature map because it is also a mapping of where a particular type of feature can be found in the image. High activation means that a particular feature has been found.

[0045] Another important concept in CNNs is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions to implement pooling, of which max pooling is the most common. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such subregion.

[0046] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers progressively reduce the spatial size of the representation, reducing the number of parameters in the network, the memory footprint, and the amount of computation, and thus also helping to control overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. The pooling operation provides another form of translation invariance.

[0047] The pooling layer operates independently on each depth slice of the input and resizes it spatially. The most common form is a pooling layer with a filter of size 2 × 2, applied with a stride of 2 for each depth slice of the input by 2 along both the width and height, discarding 75% of the activations. In this case, all max operations are over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can also use other functions such as average pooling or L2 norm pooling. Average pooling has traditionally been used often, but has fallen out of favor in recent times compared to max pooling, which often works better in practice. There is a recent trend to use smaller filters or to discard the pooling layer altogether for aggressive reduction in the size of the representation. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0048] The ReLU, above, stands for rectified linear unit, and applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. It increases the non-linearity of the decision function and the entire network without affecting the receptive fields of the convolutional layers. Other functions, such as the saturated hyperbolic tangent and sigmoid functions, are also used to increase the non-linearity. ReLU is often preferred over other functions because it trains neural networks several times faster without significantly compromising generalization accuracy.

[0049] After several convolutional and max-pooling layers, higher-level inference in neural networks is performed via fully connected layers, whose neurons are connected to all activations of the previous layer, as in regular (non-convolutional) artificial neural networks. Their activations can thus be computed as affine transformations, matrix multiplications followed by bias offsets (vector addition of a learned or fixed bias term).

[0050] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is usually the final layer of a neural network. Various loss functions suitable for different tasks may be used: softmax loss is used to predict a single class of K mutually exclusive classes; sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0051] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, an input image is passed through a convolutional layer and abstracted into a feature map containing multiple channels corresponding to the number of filters in the set of learnable filters of this layer. The feature map is then subsampled, for example using a pooling layer, which reduces the dimensionality of each channel in the feature map. The data then reaches another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish the connectivity of the network, these parameters need to be synchronized between two connected layers, such that the number of input channels of the current layer is equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is usually equal to the number of channels of the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation.

[0052] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram of it is shown in Figure 2. The goal of an autoencoder is to learn a representation (encoding) of a dataset, typically for dimensionality reduction, by training the network to ignore signal "noise". Along with the reduction side, a reconstruction side is learned, and the autoencoder tries to generate a representation from the reduced encoding that is as close as possible to its original input, hence its name. In the simplest case, given one hidden layer, the encoder stage of an autoencoder takes the input x and maps it to h h = σ(Wx + b).

[0053] This image h is usually referred to as the code, latent variable, or latent representation. Here, σ is an element-wise activation function such as the sigmoid function or the rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are usually initialized randomly and then updated iteratively during training via backpropagation. The decoder stage of the autoencoder then maps h to a reconstruction x' of the same shape as x: x'=σ'(W'h'+b') Here, σ', W' and b' of the decoder may be independent of the corresponding σ, W and b of the encoder.

[0054] Variational autoencoder models make strong assumptions about the distribution of the latent variables. They use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayesian (SGVB) estimator. The data is represented by a directed graphical model p θ (x|h), and the encoder estimates the posterior distribution p θ Approximation q for (x|h) Φ Suppose we are learning (h|x), where Φ and θ represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than standard autoencoders. The objective of a VAE has the following form:

number

[0055] In the formula, D KL represents the Kullback-Leibler divergence. The prior distribution for the latent variables is a centrally isotropic multivariate Gaussian distribution p θ It is usually set that (h) = N(0,Ι). In general, the shapes of the variational and likelihood distributions are chosen such that they are factored Gaussians: q Φ (h|x)=N(ρ(x),ω 2 (x)I) pΦ (x|h)=N(μ(h),σ 2 (h)I) where ρ(x) and ω 2 (x) is the encoder output, μ(h) and σ 2 (h) is the decoder output.

[0056] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have enabled researchers to apply neural network-based techniques to the task of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0057] Correspondingly, data compression is considered a fundamental and well-studied problem in engineering, and is commonly formulated with the objective of designing a code for a given discrete data ensemble with minimum entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0058] In this context, known as the lossy compression problem, one must trade off two competing costs: the entropy of the discretized representation (rate) and the error resulting from quantization (distortion). Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0059] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in a high-dimensional space is intractable. For this reason, most existing image compression methods work by linearly transforming the data vector into an appropriate continuous-valued representation, quantizing its elements independently, and then encoding the resulting discrete representation using a reversible entropy code. This scheme is called transform coding because of the central role of the transform.

[0060] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method, transform, quantization, and entropy code, are optimized separately (often by manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. Several transforms are used for that purpose, such as the discrete cosine transform and discrete sine transform (DCT, DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0061] Variational Image Compression The variable autoencoder (VAE) framework can be regarded as a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in FIG. 3A, which shows the VAE framework.

[0062] The transformation process can be divided into four main parts, and FIG. 3A illustrates the VAE framework. In FIG. 3A, the encoder 101 maps the input image x to a latent representation (denoted by y) via a function y=f(x). This latent representation may also be referred to as a part of the "latent space" or a "point in the latent space" in the following. The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 converts the latent representation y into a quantizer function Q,

number

number

number

[0063] A latent space can be understood as a compressed representation of data where similar data points are close to each other in the latent space. Latent spaces are useful for learning data features and finding simpler representations of data for analysis. The quantized latent representation T

number

number

number

number

number

number

[0064] In FIG. 3A, component AE105 is an arithmetic coding module that generates a quantized latent representation

number

number

number

number

[0065] Arithmetic Decoding (AD) 106 is the process of reversing the binarization process, where binary numbers are converted back into sample values. Arithmetic decoding is provided by an arithmetic decoding module 106.

[0066] It should be noted that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0067] In FIG. 3A, there are two sub-networks connected to each other. A sub-network in this context is a logical division between parts of the overall network. For example, in FIG. 3A, modules 101, 102, 104, 105 and 106 are referred to as the "encoder / decoder" sub-network. The "encoder / decoder" sub-network is responsible for encoding (generation) and decoding (analysis) of a first bitstream, "Bitstream 1". The second network in FIG. 3A includes modules 103, 108, 109, 110 and 107 and is referred to as the "hyperencoder / decoder" sub-network. The second sub-network is responsible for generating a second bitstream, "Bitstream 2". The purpose of the two sub-networks is different.

[0068] The first sub-network is Transformation 101 of an input image x into its latent representation y (this makes it easier to compress x), · The latent representation y is converted to a quantized latent representation

number

number

number

[0069] The purpose of the second sub-network is to obtain statistical properties of the samples of "Bitstream 1" (e.g., the mean, variance and correlation between samples of Bitstream 1) so that the first sub-network can compress Bitstream 1 more efficiently. The second sub-network generates a second bitstream "Bitstream 2" that includes the information (e.g., the mean, variance and correlation between samples of Bitstream 1).

[0070] The second network is the quantized latent representation

number

number

number

number

number

number

number

number

number

number

number

number

number

[0071] 3A illustrates an example of a VAE (Variational Autoencoder), the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler that targets extracting cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0072] 3A shows the encoder and the decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and the decoder may, and very often are, embedded in different devices.

[0073] FIG. 3B shows an encoder, and FIG. 3C shows the decoder component of the VAE framework in isolation. As input, the encoder receives a picture according to some embodiments. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (as shown in FIG. 3B) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the first sub-network of the encoder, and bitstream 2 is the output of the second sub-network of the encoder.

[0074] Similarly, in FIG. 3C, two bitstreams, namely bitstream 1 and bitstream 2, are received as input and the reconstructed (decoded) image is

number

[0075] Specifically, as seen in Figure 3B, the encoder includes an encoder 121 that converts an input x into a signal y, which is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and a hyper-encoder 123. The hyper-encoder 123 provides bitstream 2, already described above, to a hyper-decoder 147, which in turn provides this information to the arithmetic coding module 105 (125).

[0076] The output of the arithmetic coding module is bitstream 1. Bitstream 1 and bitstream 2 are the output of the coding of the signal and are then provided (sent) to the decoding process. Unit 101 (121) is referred to as an "encoder", but the complete sub-network described in FIG. 3B can also be referred to as an "encoder". The process of coding generally refers to a unit (module) that converts an input into a coded (e.g. compressed) output. As can be seen from FIG. 3B, unit 121 can actually be considered as the core of the entire sub-network, since it performs a conversion of the input x into a compressed version y. The compression in the encoder 121 may be achieved, for example, by applying a neural network, or generally any processing network with one or more layers. In such a network, the compression may be performed by cascading processes including downsampling to reduce the size and / or number of channels of the input. Thus, the encoder may be referred to, for example, as a neural network (NN)-based encoder, etc.

[0077] The remaining parts in the figure (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE 125, in combination with the hyperencoder 123 and hyperdecoder 127 used to configure the AE 125, may perform binarization that may further compress the quantized signal by lossless compression. Therefore, the entire sub-network of FIG. 3B may also be referred to as an "encoder".

[0078] Most deep learning (DL) based image / video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits). For example, in the VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height and thus a smaller size, the dimensionality (size) of the signal is reduced, and therefore it is easier to compress signal y. It should be noted that in general, an encoder does not necessarily need to reduce the size in both (or generally all) dimensions. Instead, some example implementations may provide an encoder that reduces the size in only one dimension (or generally a subset of the dimensions).

[0079] J. Balle, L. Valero Laparra, and EP Simoncelli (2015). In "Density Modeling of Images Using a Generalized Normalization Transformation", In: arXiv e-prints, Presented at the 4th Int. Conf. for Learning Representations, 2016 (hereafter referred to as "Balle"), the authors propose a framework for end-to-end optimization of image compression models based on nonlinear transformations. They optimize for the mean squared error (MSE), but with a more flexible transformation built from a cascade of linear convolutions and nonlinearities. In particular, they use the Generalized Divided Normalization (GDN) combined nonlinearity, inspired by models of neurons in the biological visual system and proven effective for Gaussianized image densities. This cascade transformation is followed by a uniform scalar quantization (i.e., each element is rounded to the nearest integer) that effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0080] Such an example of a VAE framework is shown in Figure 4, which utilizes six downsampling layers marked 401-406. The network architecture includes an ultra-prior model. a ,g s ) shows the image autoencoder architecture, and on the right (h a ,h s ) corresponds to an autoencoder that implements a hyper-prior. The factorized prior model is then applied to the analysis and synthesis transformation g a and g s The same architecture is used for both. Q stands for quantization, and AE and AD stand for arithmetic encoder and decoder, respectively. The encoder applies g a , resulting in responses y (latent representations) with spatially varying standard deviations. a contains multiple convolutional layers with subsampling and generalized decomposition normalization (GDN) as the activation function.

[0081] The response is h a to summarize the distribution of standard deviations in z. Then z is quantized, compressed, and transmitted as side information. The encoder then calculates the quantized vector

number

number

number

number

number

number

number

[0082] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv Nx5x5 / 2↓" means that the layer is a convolutional layer with N channels and the convolution kernel is of size 5x5. As mentioned above, 2↓ means that a 2x downsampling is performed in this layer. A 2x downsampling results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of two. Since there are six downsampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, the output signal z^ 413 has a width and height equal to w / 64 and h / 64, respectively. The modules denoted AE, AD are the arithmetic encoder, arithmetic decoder, which are described with reference to FIG. 3A to FIG. 3C. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression scheme used to convert the values ​​of symbols into a binary representation that is a reversible process. Also, the "Q" in the figure corresponds to the quantization operation described above in connection with FIG. 4 and is further described in the "Quantization" section. Also, the quantization operation and corresponding quantization unit as part of components 413 or 415 may not necessarily be present and / or may be replaced with another unit.

[0083] Also shown in Figure 4 is a decoder including upsampling layers 407-412. A further layer 420 is provided between upsampling layers 411 and 410 in the input processing order which is implemented as a convolutional layer but which does not provide upsampling to the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such layers can be provided within the NN to perform operations on the input which do not change the size of the input but which do change certain characteristics. However, such layers need not be provided.

[0084] Viewed in processing order of the bitstream 2 through the decoder, the upsampling layers are performed in reverse order, i.e. from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to provide upsampling with an upsampling ratio of 2, denoted by ↑. Of course, it is not necessary that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8, etc. may be used. Layers 407-412 are implemented as convolutional layers (conv). In particular, they may be intended to provide an inverse operation on the input to that of the encoder, and thus the upsampling layers may apply a deconvolution operation to the received input such that its size is increased by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two nearest neighbor samples, or by nearest neighbor sample copying, etc.

[0085] In the first sub-network, some convolutional layers (401-403) are followed by Generalized Divided Normalization (GDN) on the encoder side and Inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLu. It should be noted that the present disclosure is not limited to such implementations and generally other activation functions may be used instead of GDN or ReLu.

[0086] End-to-end image or video compression DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations that are not used in traditional approaches. However, it is not trivial to directly apply these techniques to build end-to-end learning systems for video compression. First, learning how to generate and compress tuned motion information for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0087] A straightforward solution is to represent the motion information using learning-based optical flow. However, current learning-based optical flow approaches aim to generate the most accurate flow field possible. Accurate optical flow is often not optimal for a particular video task. In addition, the amount of data for optical flow increases significantly compared to the motion information in traditional compression systems, and directly applying existing compression techniques to compress the optical flow values ​​would significantly increase the number of bits required to store the motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective for both the residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of the reconstructed frames given the number of bits (or bit rate) for compression. RDO is critical for video compression performance. To take advantage of the power of end-to-end training for learning-based compression systems, an RDO strategy is needed to optimize the entire system.

[0088] Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao; “DVC: ​​An End-to-end Deep Video Compression Framework” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015. In this paper, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0089] Such an encoder is shown in FIG. 5A. In particular, FIG. 5A shows the overall structure of an end-to-end trainable video compression framework. To compress the motion information, a CNN was specified to convert the optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder type network is used to compress the optical flow. A motion vector (MV) compression network is shown in FIG. 5B. The network architecture is somewhat similar to the ga / gs in FIG. 4. In particular, the optical flow is fed into a series of convolution operations and nonlinear transformations, including GDN and IGDN. The number of output channels for convolution (deconvolution) is 128, except for the last deconvolution layer, which is equal to 2. Given an optical flow with a size of M×N×2, the MV encoder generates a motion representation with a size of M / 16×N / 16×128. The motion representation is then quantized, entropy coded, and transmitted to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0090] FIG. 5C shows the structure of the motion compensation unit, where the previous reconstructed frame x t-1Using the reconstructed motion information, the warping unit generates a warped frame (usually with the help of an interpolation filter such as a bilinear interpolation filter). A separate CNN with three inputs then generates a predicted picture. The architecture of the motion compensation CNN is also shown in Figure 5C.

[0091] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to convert the residual into a corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this approach can better utilize the power of nonlinear transform and achieve higher compression efficiency.

[0092] From the above overview, it can be seen that CNN-based architectures can be applied to both image and video compression, considering different parts of the video framework including motion estimation, motion compensation, and residual coding. Entropy coding is a common method used for data compression and is widely adopted in the industry, and is also applicable to feature map compression for either human perception or computer vision tasks.

[0093] Machine Video Coding Video coding for machines (VCM) is another computer science direction that is popular today. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms such as object segmentation, detection and recognition. In contrast to traditional image and video coding, which is intended for human perception, the quality characteristic is not the reconstructed quality but the performance of a computer vision task, e.g. object detection accuracy.

[0094] Recent work has proposed a new deployment paradigm, termed collaborative intelligence, where deep models are split between mobile and cloud. Extensive experiments under various hardware configurations and wireless connectivity modes have revealed that the optimal operating point in terms of energy consumption and / or computational latency usually involves splitting the model at a deep point in the network. It turns out that today's common solutions, where the model is located entirely in the cloud or entirely on the mobile, are rarely (if ever) optimal. The concept of collaborative intelligence has also been extended to model training. In this case, data flows both ways: from cloud to mobile during backpropagation in training, and from mobile to cloud during forward pass in training, as well as inference.

[0095] Lossy compression of deep feature data has been studied in the context of recent deep models for object detection based on HEVC intra-coding. A degradation in detection performance with increasing compression levels was noted, and compression-enhanced training was proposed to minimize this loss by producing models that are more robust to quantization noise in the feature values. However, this remains a suboptimal solution, as the codecs employed are highly complex and optimized for natural scene compression rather than deep feature compression.

[0096] The problem of deep feature compression for collaborative intelligence has been addressed by an approach for object detection tasks using the popular YOLOv2 network to study the trade-off between compression efficiency and recognition accuracy. Here, the term deep feature has the same meaning as feature map. The word "deep" comes from the concept of collaborative intelligence when the output feature map of some hidden (deep) layer is captured and transferred to the cloud to perform inference. This appears to be more efficient than sending compressed natural image data to the cloud and using the reconstructed image to perform object detection.

[0097] Efficient compression of feature maps is useful for image and video compression and reconstruction for both human perception and machine vision. Shortcomings of state-of-the-art autoencoder-based approaches to compression also hold true for machine vision tasks.

[0098] Artificial Neural Networks with Skip Connections Residual neural networks (ResNet) are a type of artificial neural network (ANN) that builds on known constructs from pyramidal cells in the cerebral cortex. Residual neural networks do this by utilizing skip connections or shortcuts to skip over some layers. A typical ResNet model is implemented with two or three layer skips with nonlinearity (ReLU) and batch normalization in between. An additional weight matrix may be used to learn the skip weights, and these models are known as HighwayNets. Models with multiple parallel skips are referred to as DenseNets. In the context of residual neural networks, non-residual networks may be described as plain networks.

[0099] One motivation for skipping layers is to avoid the problem of vanishing gradients by reusing activations from the previous layer until the adjacent layer has learned its weights. During training, weights are adapted to mute the upstream layers and amplify the previously skipped layers. In the simplest case, only the weights for the connections of the adjacent layer are adapted, and there are no explicit weights for the upstream layers. This works best when a single nonlinear layer is stepped over, or when the intermediate layers are all linear. If not, explicit weight matrices should be learned for the skipped connections (HighwayNet should be used).

[0100] Skipping effectively simplifies the network by using fewer layers in the initial training phase. This speeds up learning by reducing the effect of vanishing gradients since there are fewer layers to propagate. The network then gradually restores the skipped layers as it learns the feature space. Towards the end of training, when all layers are expanded, it remains closer to the manifold and therefore learns faster. A neural network without a residual part explores more of the feature space. This makes it more vulnerable to perturbations that move it away from the manifold and requires extra training data to restore.

[0101] Longer skip connections are introduced into the U-Net, as shown in Figure 6. The U-Net architecture originates from the so-called "fully convolutional network" first proposed by Long and Shelhamer. The main idea is to supplement the usual contraction network by successive layers, where the pooling operations are replaced by upsampling operators. Thus, these layers increase the resolution of the output. Furthermore, successive convolutional layers can then learn to assemble an accurate output based on this information.

[0102] One important variation in U-Net is the presence of a large number of feature channels in the upsampling part, which allows the network to propagate context information to the high-resolution layers. As a result, the dilation path is more or less symmetric to the erosion path, resulting in a U-shaped architecture. The network uses only the valid parts of each convolution, without a fully connected layer. The missing context is extrapolated by mirroring the input image to predict pixels in the border regions of the image. This tiling strategy is important for applying the network to large images, whose resolution would otherwise be limited by GPU memory.

[0103] Introducing skip connections allows for better capture of features at different spatial resolutions, which has been successfully applied for computer vision tasks such as object detection and segmentation. However, implying such skip connections for image or video compression is not a trivial task, since information from the encoding side needs to be transferred in a communication channel, and direct connection of layers requires a significant amount of data to be transferred.

[0104] Conventional and Hybrid Video Encoding and Decoding The neural network framework may also be employed in combination with or within conventional hybrid encoding and decoding, as illustrated later. In the following, a very brief overview is given of exemplary hybrid encoding and decoding.

[0105] FIG. 7A shows a schematic block diagram of an exemplary video encoder 20 configured to implement the techniques of the present application. In the example of FIG. 7A, the video encoder 20 includes an input 201 (or an input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output 272 (or an output interface 272). The mode selection unit 260 may include an inter prediction unit 244, an intra prediction unit 254, and a splitting unit 262. The inter prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). The video encoder 20 shown in FIG. 7A may also be referred to as a hybrid video encoder, or a video encoder with a hybrid video codec.

[0106] The encoder 20 may be configured to receive, for example via an input 201, a picture 17 (or picture data 17), for example a picture of a video or a sequence of pictures forming a video sequence. The received picture or picture data may be a preprocessed picture 19 (or preprocessed picture data 19). For simplicity, the following description refers to the picture 17. The picture 17 may also be referred to as a current picture or a picture to be coded (especially in video coding to distinguish the current picture from other pictures, for example previously encoded and / or decoded pictures of the same video sequence, i.e. a video sequence that also includes the current picture).

[0107] A (digital) picture is or can be considered as a two-dimensional array or matrix of samples with intensity values. The samples in the array are sometimes called pixels (short for picture element) or pels. The number of samples in the horizontal and vertical directions (or axes) of the array or picture defines the size and / or resolution of the picture. For color representation, typically three color components are employed, i.e. a picture can be represented or contain three sample arrays. In an RGB format or color space, a picture contains corresponding red, green and blue sample arrays. However, in video coding, each pixel is typically represented in a luminance and chrominance format or color space, e.g. YCbCr, containing a luminance component denoted by Y (sometimes L is also used instead) and two chrominance components denoted by Cb and Cr. The luminance (or short luma) component Y represents brightness or gray level intensity (e.g., as in a grayscale picture), and the two chrominance (or short chroma) components Cb and Cr represent chromaticity or color information components. Thus, a picture in YCbCr format includes a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). A picture in RGB format can be converted or translated to YCbCr format and vice versa, a process also known as color conversion or translation. If the picture is monochrome, the picture may include only a luminance sample array. Thus, a picture may be, for example, an array of luma samples in monochrome format, or an array of luma samples and two corresponding arrays of chroma samples in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0108] An embodiment of the video encoder 20 may include a picture division unit (not shown in FIG. 7A) configured to divide a picture 17 into multiple (typically non-overlapping) picture blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC) or coding tree blocks (CTBs) or coding tree units (CTUs) (H.265 / HEVC and VVC). The picture division unit may be configured to use the same block size for all pictures of a video sequence and a corresponding grid defining the block sizes, or to vary the block size among pictures or subsets or groups of pictures and divide each picture into the corresponding blocks. The abbreviation AVC stands for Advanced Video Coding.

[0109] In further embodiments, the video encoder may be configured to directly receive block 203 of picture 17, such as one, multiple, or all blocks that form picture 17. Picture block 203 may also be referred to as a current picture block or a picture block to be coded.

[0110] Similar to picture 17, picture block 203 is or can be considered as a two-dimensional array or matrix of samples having intensity values ​​(sample values), albeit with smaller dimensions than picture 17. In other words, block 203 may include, for example, one sample array (e.g., a luma array for a monochrome picture 17, or a luma or chroma array for a color picture), or three sample arrays (e.g., a luma and two chroma arrays for a color picture 17), or any other number and / or type of arrays, depending on the color format applied. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Thus, a block may be, for example, an M×N (M columns×N rows) array of samples, or an M×N array of transform coefficients.

[0111] An embodiment of video encoder 20 as shown in FIG. 7A may be configured to encode picture 17 on a block-by-block basis, eg, encoding and prediction are performed on each block 203.

[0112] The embodiment of video encoder 20 shown in FIG. 7A may be further configured to divide and / or encode pictures using slices (also referred to as video slices), where a picture may be divided into or encoded using one or more slices (typically non-overlapping), each of which may include one or more blocks (e.g., CTUs).

[0113] An embodiment of video encoder 20 as shown in FIG. 7A may be further configured to divide and / or encode a picture using tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where a picture may be divided into or encoded using one or more tile groups (typically non-overlapping), where each tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, where each tile may be, for example, rectangular in shape and may include one or more blocks (e.g., CTUs), e.g., full or partial blocks.

[0114] 7B illustrates an example of a video decoder 30 configured to implement the techniques of the present application. The video decoder 30 is configured to receive coded picture data 21 (e.g., coded bitstream 21), e.g., coded by encoder 20, to obtain a decoded picture 331. The coded picture data or bitstream includes information for decoding the coded picture data, e.g., data representing picture blocks of coded video slices (and / or tile groups or tiles) and associated syntax elements.

[0115] The entropy decoding unit 304 is configured to parse the bitstream 21 (or the coded picture data 21 in general) and perform entropy decoding, e.g., on the coded picture data 21, to obtain, e.g., quantization coefficients 309 and / or decoded coding parameters (not shown in FIG. 3), e.g., any or all of inter prediction parameters (e.g., reference picture indexes and motion vectors), intra prediction parameters (e.g., intra prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 may be configured to apply a decoding algorithm or scheme corresponding to an encoding scheme as described with respect to the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 may be further configured to provide the inter prediction parameters, intra prediction parameters and / or other syntax elements to the mode application unit 360 and provide other parameters to other units of the decoder 30. The video decoder 30 may receive syntax elements at a video slice level and / or a video block level. In addition to or as an alternative to slices and their respective syntax elements, tile groups and / or tiles and their respective syntax elements may be received and / or used.

[0116] The reconstruction unit 314 (e.g., an adder or summer 314) may be configured to add the reconstructed residual block 313 to the prediction block 365, e.g., by adding sample values ​​of the reconstructed residual block 313 and sample values ​​of the prediction block 365, to obtain a reconstructed block 315 in the sample domain.

[0117] An embodiment of video decoder 30 as shown in FIG. 7B may be configured to divide and / or decode pictures using slices (also referred to as video slices), where a picture may be divided into or decoded using one or more slices (typically non-overlapping), and each slice may include one or more blocks (e.g., CTUs).

[0118] An embodiment of video decoder 30 as shown in FIG. 7B may be configured to divide and / or decode a picture using tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), where a picture may be divided into or decoded using one or more tile groups (typically non-overlapping), where each tile group may include, for example, one or more blocks (e.g., CTUs) or one or more tiles, where each tile may be, for example, rectangular in shape and may include one or more blocks (e.g., CTUs), e.g., full or partial blocks.

[0119] Other variations of the video decoder 30 may be used to decode the encoded picture data 21. For example, the decoder 30 may generate an output video stream without a loop filtering unit 320. For example, the non-transform-based decoder 30 may inverse quantize the residual signal directly without the inverse transform processing unit 312 for some blocks or frames. In another implementation, the video decoder 30 may have the inverse quantization unit 310 and the inverse transform processing unit 312 combined into a single unit.

[0120] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current step may be further processed and then output to the next step. For example, after the interpolation filtering, the motion vector derivation, or the loop filtering, further operations such as clipping or shifting may be performed on the processing result of the interpolation filtering, the motion vector derivation, or the loop filtering.

[0121] Improved coding efficiency As mentioned above, image and video compression methods based on the variational autoencoder approach suffer from the lack of spatial adaptation processing and object segmentation targeting to capture real object boundaries. Thus, content adaptability is limited. Furthermore, for some types of video information, such as motion information or residual information, sparse representation and coding is desirable to keep signaling overhead at a reasonable level.

[0122] Therefore, some embodiments of the present disclosure introduce segmentation information coding and feature map coding from different spatial resolution layers of an autoencoder to enable content adaptability and sparse signal representation and transmission.

[0123] In some exemplary implementations, connections are introduced between layers of the encoder and layers of the decoder other than the low resolution layer (latent space) that are transmitted in the bitstream. In some exemplary implementations, to save bandwidth, only a portion of the feature maps of different resolution layers are provided in the bitstream. For example, signal selection and signal feeding logic is introduced to select, transmit, and use a portion of the feature maps from different resolution layers. At the receiver side, tensor combination logic is introduced to combine the output from the previous resolution layer with the information received from the bitstream corresponding to the current resolution layer.

[0124] In the following, some detailed embodiments and examples on the encoder side and the decoder side are provided.

[0125] Encoding method and device According to one embodiment, there is provided a method for encoding data for picture or video processing into a bitstream, the method comprising a step of processing the data, the processing of the data comprising generating feature maps in a number of cascaded layers, each feature map having a respective resolution, and at least two of the generated feature maps having different resolutions from each other.

[0126] In other words, the resolution of two or more of the cascade layers may be different from each other. Here, when referring to the resolution of a layer, what is meant is the resolution of the feature map processed by that layer. In an exemplary implementation, it is the resolution of the feature map output by the layer. A feature map that includes a resolution means that at least a portion of the feature map has the resolution. In some implementations, the entire feature map may have the same resolution. The resolution of a feature map may be given, for example, by a number of feature map elements in the feature map. However, it may be more specifically defined by the number of feature map elements in one or more dimensions (such as x, y, alternatively or additionally, the number of channels may be considered).

[0127] The term layer herein refers to a processing layer. It does not have to be a layer with trainable or trained parameters (weights) like the layers of some of the neural networks mentioned above. Rather, a layer may represent a particular processing of layer inputs to obtain a layer output. In some embodiments, a layer(s) may be trained or trainable. Training here refers to machine learning or deep learning.

[0128] When referring to cascaded layers, what is meant is that the layers have a certain predefined order, and the input to a first layer (in the given order) is processed sequentially by the first layer and then by further layers according to the given order. In other words, the output of layer j is the input of layer j+1, where j is an integer from 1 to the total number of cascaded layers. In a specific, non-limiting example, layer j+1 contains (or has) the same or lower resolution as layer j for all possible j values. In other words, the resolution of the layers does not increase (e.g., on the encoder side) with the sequence of the cascade. However, it should be noted that the present disclosure is not limited to such a specific cascaded layer. In some embodiments, the layers of the cascaded process may also include layers that increase the resolution. In any case, there may be layers that do not change the resolution.

[0129] A lower resolution of the feature maps may mean, for example, fewer feature elements per feature map. A higher resolution of the feature maps may mean, for example, more feature elements per feature map.

[0130] The method further includes generating a bitstream including selecting a layer from the plurality of layers that is different from the layer that generates the lowest resolution feature map and inserting information related to the selected layer into the bitstream.

[0131] In other words, in addition to (or instead of) outputting the results of processing by all layers in the cascade to the bitstream, information to another (selected) layer is provided. There may be one or more selected layers. The information related to the selected layer can be any kind of information, such as the output of the layer or segmentation information of a part of the layer (discussed later), or other information related to the feature map processed by the layer and / or the processing performed by the layer. In other words, in some examples, the information can be the elements of the feature map and / or the position of the elements in the feature map (within the layer).

[0132] The input to the cascade process is a picture or data for video processing. Such data may be related to predictive coding, such as inter- or intra-prediction. It may be motion vectors, or other parameters of prediction, such as prediction mode or reference picture or direction, or other parts of coding apart from prediction, such as transformation, filtering, entropy coding, or quantization. Bitstream generation may include any conversion of values ​​to bits (binarization), including fixed codewords, variable length codes, or arithmetic coding.

[0133] Here, a picture may be a still picture or a video picture. A picture refers to one or more samples, such as samples captured by a camera or generated by, for example, computer graphics. A picture may include samples representing luminance levels in grayscale, or may have multiple channels, including one or more of a luminance channel, a chrominance channel(s), a depth channel, or other channels. The picture or video encoding may be either a hybrid coding (e.g., similar to HEVC or VVC) or an autoencoder as described above.

[0134] Figure 8 is a flow diagram illustrating the above method. The method thus comprises a step 810 of processing input data. From the processed data, a portion is selected in a selection step 820 and included in the bitstream in a generation step 830. Not all the data generated in the processing step needs to be included in the bitstream.

[0135] According to an example implementation, the processing further includes downsampling by one or more of the cascaded layers. An example network 900 for implementing (performing in operation) such processing is shown in FIG.

[0136] In particular, FIG. 9 shows input data for image or video processing 901 entering the network 900. The input data for image or video processing can be any kind of data used for such processing, such as directly, image (picture) or video samples, prediction modes, motion vectors, etc., as already described above. The processing of FIG. 9 applied to the input 901 is performed by a number of processing layers 911-913, each processing layer reducing the resolution of each motion vector array. That is, the cascaded layers 911-913 are downsampling layers. It should be noted that when a layer is referred to as a downsampling layer, it performs downsampling. There may be embodiments in which the downsampling layers 911-913 perform downsampling as their only task, and there may be embodiments in which the downsampling layers 911-913 do not perform downsampling as their only task. Rather, the downsampling layers may generally also perform other types of processing.

[0137] As can be seen in FIG. 9, apart from the processed data inputs and outputs, the downsampling layers 911-913 also have additional selection outputs leading to signal selection logic 920. Note that the term "logic" here refers to any circuitry that implements a function (here signal selection). The signal selection logic 920 selects information to be included in the bitstream 930 from the selection outputs of any of the layers. In the example of FIG. 9, each layer 911-913 downsamples the layer input. However, layers that do not apply downsampling may be added between the downsampling layers. For example, these layers may process the input by filtering or other operations.

[0138] In the example shown in FIG. 9, the signal selection logic 920 selects information to be included in the bitstream from the output of layers 911 to 913. The goal of this selection may be to select information relevant to reconstructing an image or video from multiple feature maps output by different layers. In other words, the downsampling layer and the signal selection logic may be implemented as part of an encoder (picture or video encoder). For example, the encoder may be the encoder 101 shown in FIG. 3A, the encoder 121 of FIG. 3B, the MV encoder net (part of the end-to-end compression of FIG. 5A), the MV encoder of FIG. 5B, or part of an encoder according to FIG. 7A (e.g., part of the loop filtering 220 or the mode selection unit 260 or the prediction unit 244, 254), etc.

[0139] 9 further includes a decoder side portion (which may be referred to as the dilation path) including signal feeding logic 940 and upsampling layers 951 to 953. The input of the encoder side is the bitstream 930. The output 911 is, for example, the reconstructed input 901. The decoder side is described below with larger delays.

[0140] Downsampling may be performed, for example, via max pooling, average pooling, or any other operation that results in downsampling. Another example of such an operation includes a convolution operation. FIG. 10A shows an example of max pooling. In this example, four elements of each (adjacent 2×2 square) of array 1010 are grouped and used to determine one element in array 1020. Arrays 1020 and 1010 may correspond to feature maps in some embodiments of the present disclosure. However, the arrays may also correspond to parts of feature maps in this embodiment. The fields (elements) in arrays 1020 and 1010 may correspond to elements of the feature maps. In this illustration, feature map 1020 is determined by downsampling feature map 1010. The numbers in the fields of arrays 1010 and 1020 are merely illustrative. Instead of numbers, the fields may include, for example, motion vectors. In the example of max pooling shown in FIG. 10A, the top left four fields of array 1010 are grouped and the maximum of their values ​​is selected. This group of values ​​determines the top-left field of array 1020 by assigning this maximum value to this field. In other words, the maximum of the four top-left values ​​of array 1010 is inserted into the top-left field of array 1020.

[0141] Alternatively, in some implementations, min pooling may be used. In min pooling, instead of selecting the field with the maximum value, the field with the minimum value is selected. However, these downsampling techniques are only examples, and various downsampling strategies may be used in different embodiments. Some implementations may use different downsampling techniques in different layers, in different regions in the feature map, and / or for different types of input data.

[0142] In some implementations, downsampling is performed using average pooling, where the average of a group of feature map elements is calculated and associated with a corresponding field in the feature map of the downsampled feature map.

[0143] An example of average pooling is shown in Figure 10B, where the top left feature map element of feature map 1050 is averaged, and the top left element of feature map 1060 takes this average. Something similar is shown for the three groups in the top right, bottom right, and bottom left of Figure 10B.

[0144] In another embodiment, a convolution operation is used for downsampling in some or all of the layers. In a convolution, a filter kernel is applied to a group or block of elements in the input feature map. The kernel may itself be an array of elements having the same size as the block of input elements, with each element of the kernel storing a weight for the filter operation. In downsampling, the sum of elements from the input block, each weighted with a corresponding value taken from the kernel, is calculated. If the weights of all elements in the kernel are fixed, such a convolution may correspond to the filter operation described above. For example, a convolution with a kernel having the same fixed weights and a stride of the size of the kernel corresponds to an average pooling operation. However, the stride of the convolution used in this embodiment may be different from the kernel size, and the weights may be different. In one example, the kernel weights may be such that some features in the input feature map may be emphasized or distinguished from each other. Furthermore, the kernel weights may be learnable or pre-learned.

[0145] According to one embodiment, the information associated with a selected layer includes feature map elements 1120 of that layer. For example, the information may convey feature map information. In general, a feature map may include any feature associated with a motion picture.

[0146] 11 shows an example implementation with dense optical flow of motion vectors where feature map 1110 has width W and height H. Motion segmentation net 1140 includes three downsampling layers (e.g., corresponding to downsampling layers 911-913 in FIG. 9) and signal selection circuit (logic) 1100 (e.g., corresponding to signal selection logic 920). FIG. 11 shows an example of the outputs of different layers (L1-L3) in the contraction path on the right side.

[0147] In this example, the output of each layer (L1-L3) is a feature map with progressively lower resolution. The input to L1 is the dense optical flow 1110. In this example, one element of the feature map output from L1 is determined from 16 (4x4) elements of the dense optical flow 1110. Each square in the L1 output (bottom right of FIG. 11) corresponds to a motion vector obtained by downsampling (downspl4) from the 16 motion vectors of the dense optical flow. Such downsampling can be, for example, average pooling or another operation, as explained above. In this exemplary implementation, only a portion of the feature map L1 of that layer is included in the information 1120. A layer L1 is selected and the portion corresponding to the four motion vectors (feature map elements) associated with the selected layer is signaled in the selected information 1120.

[0148] The output L1 of the first layer is then input to the second layer (downspl2). The output L2 feature map element of the second layer is determined from the four elements of L1. However, in other examples, each element of the feature map with a low resolution can also be determined by a group composed of any other number of elements of the feature map with the next higher resolution. For example, the number of elements in the group that determines one element in the next layer may be any power of two. In this example, the output L2 feature map corresponds to three motion vectors that are also included in the selected information 1120, and therefore the second layer is also the selected layer. The third layer (downspl2) downsamples the output L2 from the second layer by two in each of the two dimensions. Thus, one feature map element of the output L3 of the third layer is obtained based on the four elements of L2. In the feature map L3, no elements are signaled, i.e., the third layer is not the selected layer in this example.

[0149] The signal selection module 1100 of the motion segmentation net 1140 selects the above mentioned motion vectors (elements of the feature maps from the output of the first and second layers) and provides them to the bitstream 1150. This provision may be a simple binarization or may, but need not, involve entropy coding.

[0150] The group of elements may be arranged in a square shape as in the example of FIG. 11. However, the group may be arranged in any other shape, such as, for example, a rectangle, with the long side of the rectangle oriented horizontally or vertically. These shapes are merely examples. In an implementation, any shape may be used. The shape may also be signaled in the bitstream 1150. The signaling may be implemented by a map of flags indicating which features belong to the shape and which do not. Alternatively, the signaling may be done using a more abstract description of the shape.

[0151] In this exemplary implementation, the feature map elements are grouped such that every element belongs to exactly one group of elements that determines one element of the feature map of the next layer. In other words, the feature map element groups do not overlap, and only one group contributes to the feature map elements of a higher (later in the cascaded processing order) layer. However, it is conceivable that an element of one layer may contribute to two or more elements of the next layer. In other words, in process 810, when a new layer output, for example, layer output L2, is generated based on the layer output L1 with high resolution, a filter operation may be used.

[0152] In this embodiment, the selection 820 (e.g., by signal selection 1100) selects elements from multiple output feature maps (L1-L3) to be included in the bitstream. The selection may be implemented to keep the amount of information relevant for decoding as large as possible while keeping the amount of data required to signal the selected data small. For example, rate-distortion optimization, or other optimizations, may be employed.

[0153] The above example shows processing with three layers. In general, the method is not limited thereto. Any number of processing layers (one or more) can be employed. In other words, according to a more generalized example, the method includes obtaining data to be coded. This may be a dense flow 1110 of motion vectors, as described above. However, the disclosure is not limited thereto, and instead of or in addition to motion vectors, other data may be processed, such as prediction modes, prediction directions, filtering parameters, or even spatial picture information (samples) or depth information.

[0154] The processing 810 of the data to be encoded includes, in this example, processing through each layer j of a plurality of N cascaded layers. The processing through the jth layer includes: - if j=1, take the data to be encoded as the layer input; otherwise, take the feature map processed by the (j-1)th layer as the layer input (i.e., if the jth layer is the currently processed layer, j-1 is the preceding layer); - processing the obtained layer inputs, the processing including downsampling; and - outputting the downsampled feature map.

[0155] In this example, j=1 is the highest resolution layer of the N processing layers. Note that the input of this layer may be dense optical flow (which may also be generally considered as a feature map). Thus, in some particular embodiments, the j=1 layer may be the input layer. However, this is not necessarily the case, since it is conceivable that the N processing layers are preceded by some pre-processing layers. Typically, it is a feature of the encoder that earlier processing layers have higher resolution than later processing layers (contraction path). This can be correspondingly reversed at the decoder side. It is conceivable that some of the processing layers do not change the resolution or do not further improve the resolution, but the present disclosure may still be applicable.

[0156] In the above example, the bitstream 1150 carries the selected information 1120. It can be, for example, a motion vector or any other feature. In other words, the bitstream 1150 carries feature map elements from at least one layer that is not an output layer of the processing network (encoder side processing network). In the example of FIG. 11, only a part of the selected feature map is transmitted in the bitstream. This part has one or more feature elements. To enable the decoder to determine which part of the feature map is transmitted, a rule for the decision can be defined. In some embodiments, segmentation information can be transmitted in the bitstream 1150 to configure which part of the feature map is transmitted. Such an exemplary embodiment is described below. However, it should be noted that the above embodiment is only exemplary and generally such additional signaling is not necessary, since there can be rules for deriving the information and relying on other known or signaled parameters.

[0157] In an example embodiment related to segmentation information, the information related to the selected layer includes (in addition to or instead of the selected information 1120) information 1130 indicating from which layer and / or from which portion of the feature map of that layer the element of the feature map of that layer was selected.

[0158] In the example shown in FIG. 11, the segmentation information is indicated by a binary flag. For example, on the right side, each low-resolution feature map or feature map portion is assigned either 0 or 1. For example, L3 is assigned zero (0) because it is not selected and no motion vector (feature element) is signaled for L3. Feature map L2 has four portions. Layer process L2 is the selected layer. Three feature map elements (motion vectors) of the four portions are signaled, and correspondingly, the flag is set to 1. The remaining one portion of feature map L2 does not contain motion vector(s), and therefore the flag is set to 0 because the motion vector corresponding to that portion is indicated by the L1 feature map. Since L1 is the first layer, it is implicit that the remaining motion vectors are provided in this layer. Note that the binary flag takes a first value (e.g., 1) when the corresponding feature map portion is portion selection information, and a second value (e.g., 0) when the corresponding feature map portion is not portion selection information. Since this is a binary flag, it can only have one of these two values.

[0159] Such segmentation information may be given in the bitstream. The left side of Fig. 11 shows the processing of the segmentation information 1130. It should be noted that the segmentation information 1130 may also be processed by a layer of the motion segmentation net 1140. It may be processed in the same layer as the feature map or in a different layer. The segmentation information 1130 may also be interpreted as follows: One superpixel of the layer with the lowest resolution covers a 16x16 cell of the feature map obtained by downsampling downspl4 of the dense optical flow 1110. Since the flag assigned to the superpixel covering the 16x16 cell is set to 0, this means that the feature map element(s) (here the motion vector) is not indicated for this layer (the layer is not selected). Thus, the feature map element may be indicated within an area corresponding to a 16x16 cell of the next layer, represented by four equally sized superpixels, each covering an 8x8 feature element cell. Each of these four superpixels is associated with a flag. For superpixels associated with a flag with value 1, a feature map element (motion vector) is signaled. For superpixels with the flag set to 0, no motion vector is signaled. Unsignaled motion vectors are signaled for layers with superpixels covering a 4x4 element cell.

[0160] Generally speaking, a method for encoding data for picture / video decoding may further include selecting (segmentation) information for insertion into a bitstream. This information relates to a first region (superpixel) within a feature map processed by layer j>1. The first region corresponds to a region of the feature map or initial data that is encoded at a layer smaller than j and includes a plurality of elements. The method further includes excluding from the selection a region corresponding to the first region from a selection in a feature map processed by layer k (where k is an integer greater than or equal to 1 and k<j). The correspondence of regions between different layers, as used herein, means that the corresponding regions (superpixels) spatially cover the same feature elements (initial data elements) within the feature map (initial data) being encoded. In the example of FIG. 11, the initial data to be segmented is L1 data. However, the association may be made with reference to the dense optical flow 1110.

[0161] In the particular configuration of FIG. 11, it is ensured that each feature element of the initial feature map (e.g., L1) is covered by only one superpixel of only one of the N layers. This configuration offers the advantage of efficiently coding the feature map as well as the segmentation information. The cascaded layer processing framework corresponds to the neural network processing framework and can be used in this way to segment the data and provide data for various segments with different resolutions. In particular, the advantage of downsampling in some of the layers may include reducing the amount of data required to signal the representation of the initial feature map. Specifically, in the example of signaling motion vectors, downsampling may allow a group of similar motion vectors to be signaled by one common motion vector. However, the prediction error caused by grouping the motion vectors needs to be small to achieve good inter prediction. This may mean that for different areas of the picture, different levels of grouping the motion vectors may be optimal to achieve the desired prediction quality, while at the same time requiring a small amount of data to signal the motion vectors. This may be achieved using multiple layers with different resolutions.

[0162] In one embodiment where the feature map elements are motion vectors, the length and direction of the motion vectors may be averaged for downsampling purposes, and the averaged motion vectors are associated with the corresponding feature map elements of the downsampled feature map. In normal averaging, all elements of a group of elements corresponding to one element in the downsampled feature map have the same weight. This corresponds to applying a filter with equal weights to a group or block of elements to calculate the downsampled feature map element. However, in other implementations, such filters may have different weights for different elements in the layer input. In other implementations, instead of calculating the average of a group or block of elements in the downsampling, the median of each group of elements may be calculated.

[0163] In the example of Figure 11, the downsampling filter operation uses a filter with a square shape of size 2x2 of the input elements and calculates a filter output that is mapped to one element in the downsampled feature map depending on the selected filter operation. The filter operation uses a stride of 2 equal to the edge length or a square filter. This means that between two filtering operations the filter is moved by a stride of the same size as the filter. As a result, in downsampling, the downsampled elements are calculated from non-overlapping blocks in the layer where the downsampling filter is applied.

[0164] However, in some further possible embodiments, the stride may be different from the edge length of the filter. For example, the stride may be smaller than the edge length of the filter. As a result, the filter blocks used to determine the elements in the downsampled layer may overlap, which means that one element from the downsampled feature map contributes to the calculation of more than one element in the downsampled feature map.

[0165] In general, the data associated with a selected layer includes an indication of the location of feature map elements within a feature map of the selected layer, where the feature map of the selected layer refers to the output from the selected layer, i.e., the feature map processed by the selected layer, similar to the concept of FIG.

[0166] For example, the locations of selected and non-selected feature map elements are indicated by a number of binary flags based on the location of the flags in the bitstream. In the above description with reference to FIG. 11, the binary flags are included in the bitstream 1150 as segmentation information 1130. To enable the decoder to parse and correctly interpret the segmentation information, an assignment between flags and areas in the layer and / or feature maps processed by the layer should be defined. This can be done by defining an order of binarizing the flags that is known to both the encoder and the decoder.

[0167] The above example is provided for data used in encoding a picture / video that is a motion vector. However, the present disclosure is not limited to such an embodiment. In one embodiment, the data to be encoded includes image information and / or prediction residual information and / or prediction information. Here, image information means sample values ​​of an original image (or an image to be coded). The sample values ​​may be samples of one or more colors or other channels.

[0168] The information about the selected layer is not necessarily a motion vector or a motion vector of a superpixel. Additionally or alternatively, in some embodiments, the information includes prediction information. The prediction information may include a reference index and / or a prediction mode. For example, the reference index may indicate which particular picture from the reference picture set should be used for inter prediction. The index may be relative to a current image in which the current block to be predicted is located. The prediction mode may indicate, for example, whether to use a single or multiple reference frames and / or whether to use a combination of different predictions, such as combined intra-inter prediction.

[0169] Nevertheless, if the data to be encoded is a motion vector field, efficient motion vector field coding and reconstruction can be achieved. The corresponding general block scheme of a device that can perform such encoding and decoding of motion fields is shown in FIG. 12. On the encoding side, motion information is obtained using some motion estimation or optical flow estimation module (unit) 1210. The input to the motion vector (optical flow) estimation is the current picture and one or more reference pictures (stored in a reference picture buffer). In FIG. 12, the picture is referred to as a "frame", a term sometimes used for a picture of a video. The optical flow estimation unit 1210 outputs an optical flow 1215. In different implementations, the motion estimation unit can output motion information already with different spatial resolution, for example for some N×N block, or for each pixel of the original resolution, which may be referred to as dense optical flow. The motion vector information is intended to be transmitted to the decoding side (embedded in the bitstream 1250) and used for motion compensation. To obtain a motion compensated region, each pixel of the region should have a defined motion vector. Transmitting motion vector information for each pixel of the original resolution may be too expensive. To reduce the signaling overhead, a motion specification (or segmentation) module 1220 is used. A corresponding module 1270 on the decoding side performs a motion generation (densification) task to reconstruct a motion vector field 1275. The motion specification (or segmentation) module 1220 outputs motion information (e.g., motion vectors, and / or possibly reference pictures) and segmentation information. This information is added (encoded) to the bitstream.

[0170] In this embodiment, the motion segmentation unit 1220 and the motion generation unit 1270 include only a downsampling layer dwnspl and a corresponding upsampling layer upspl, as shown in FIG. 13. Nearest neighbor methods can be used for downsampling and upsampling, and average pooling can be used for downsampling. Feature map data from layers of different spatial resolutions are selected by the encoder and sent in the bitstream as selected information 1120, together with segmentation information 1130 that instructs the decoder how to interpret and utilize the selected information 1120. The motion segmentation (sparsification) net 1220 is shown in FIG. 13 as network 1310. Thus, the dense optical flow 1215 is inserted into the motion segmentation (sparsification) net 1310. The net 1310 includes three downsampling layers and signal selection logic 1320 that selects the information to be included in the bitstream 1350. The functionality is similar to that already described with reference to the more general FIG. 9.

[0171] In the above embodiment, the signaling of information related to layers different from the output layer improves the scalability of the system. Such information may be information about hidden layers. In the following, embodiments and examples are presented on taking advantage of the scalability and flexibility provided. In other words, some approaches are provided on how to select layers and what the information may look like.

[0172] Some of the embodiments herein present image or video compression systems that use an autoencoder architecture that includes one or more dimensionality (or spatial resolution) reduction steps (implemented by layers incorporating downsampling operations) in the encoding part. Along with the reduction (encoding) side, a reconstruction (decoding) side is trained, where the autoencoder tries to generate from the reduced encoding a representation as close as possible to its original input, which usually implies one or more resolution increase steps (implemented by layers incorporating upsampling operations) on the decoding side.

[0173] Hereinafter, under encoder is meant the encoding part of an autoencoder which generates the latent signal representations contained in the bitstream. Such an encoder is for example 101 or 121 as described above. Under decoder is meant the generating part of an autoencoder which perceives the latent signal representations obtained from the bitstream. Such a decoder is for example the decoder 104 or 144 as described above.

[0174] As already described with reference to Figure 11, the encoder selects a portion (or portions) of feature map information (selected information 1120) from layers of different spatial resolutions according to signal selection logic 1100 and transmits the selected information 1120 in a bitstream 1150. Segmentation information 1130 indicates from which layer the selected information is taken and from which portion of the feature map of the corresponding layer.

[0175] According to one embodiment, the processing by layer j of the plurality of N cascaded layers is - determining a first cost resulting from reconstructing a portion of the reconstructed picture using the feature map elements output by the j-th layer; - determining a second cost resulting from reconstructing a portion of the picture using the feature map elements output by the (j-1)th layer; and - if the first cost is higher than the second cost, selecting the (j-1)th layer and selecting information relating to the portion in the (j-1)th layer.

[0176] The decision of which layer to select may be performed based on the distortion or a function of the distortion, e.g., in case of motion vector field coding, the reconstructed picture (or picture portion) may be a motion compensated picture (or picture portion).

[0177] To select the selected information in this exemplary implementation, the encoder includes a cost calculation unit (module) that estimates the cost of transmitting motion information from a specific resolution layer at a certain position. The cost is calculated by combining the distortion caused by motion compensation using the selected motion vector and an estimate of the amount of bits required to transmit the motion information multiplied by a Lagrangian multiplier. In other words, according to one embodiment, a rate-distortion optimization (RDO) is performed.

[0178] In other words, in some embodiments, the first cost and the second cost include an amount of data and / or distortion. For example, the amount of data includes an amount of data required to transmit data related to the selected layer. This may be motion information or other information. It may also be or include overhead caused by residual coding. The distortion is calculated by comparing the reconstructed picture to a target picture (the original picture to be coded or a part of such coded picture). It should be noted that RDO is just one possibility. This disclosure is not limited to such an approach. Furthermore, complexity or other factors may be included in the cost function.

[0179] FIG. 14 shows the first part of the cost calculation. In particular, the cost calculation (or estimation) unit 1400 obtains the optical flow L1 downsampled by the downsampling layer (downspl4) of the motion segmentation unit 1140. The cost calculation unit 1400 then upsamples 1415 the optical flow to its original resolution, for example, in this case by 4 in each of the two directions (x and y). Then, a motion compensation 1420 is performed using the upsampled motion vectors output from 1410 and the reference picture(s) 1405 to obtain a motion compensated frame (picture) or a part of a motion compensated frame (picture) 1420. Then, the distortion is calculated 1430 by comparing the motion compensated picture (part) 1420 with a target picture 1408. The target picture 1408 may be, for example, the picture to be coded (original picture). In some example implementations, the comparison may be performed by calculating the mean squared error (MSE) or sum of absolute differences (SAD) between the target picture 1408 and the motion compensated picture 1420. However, other types of measurements / metrics, such as more advanced metrics targeting subjective perception like MS-SSIM or VMAF, may be used alternatively or in addition. The calculated distortion 1430 is then provided to a cost calculation module 1460.

[0180] Furthermore, the rate estimation module 1440 calculates an estimate of the bit amount for each motion vector. The rate estimate may include not only bits used to signal the motion vector, but also bits used to indicate segmentation information (in some embodiments). The number of bits thus obtained may be normalized, for example, per pixel (feature map element) 1450. The resulting rate (amount of bits) is provided to the cost calculation module 1460. To obtain the rate (amount of bits) estimate, the evaluation of the amount of bits for each motion vector transmission is performed, for example, using a motion information coding module (for example, by performing coding and recording the resulting amount of bits), or, in some simplified implementations, using the length of the motion vector in its x or y component as a rough estimate. Other estimation techniques may be applied. To take the segmentation information into account, the segmentation information may be evaluated by the segmentation information coding module (for example, by generating and coding the segmentation information and counting the number of resulting bits), or, in a simpler implementation, by adding one bit to the total amount of bits.

[0181] The next step of the cost calculation in this example is cost calculation 1460, followed by downsampling by 4 1470 (downspl 4) to the resolution of the corresponding downsampling layer of the motion segmentation unit 1100. In some cases, only one motion vector can be transmitted for each point (picture sample value). Thus, the resulting cost tensor can have a corresponding size (dimension). Thus, the bit estimate can be normalized by the square of the downsampling filter shape (e.g., 4×4).

[0182] Then, using Lagrange multipliers, the cost estimation unit 1460 calculates, for example, Cost=D+λ*R, or Cost=R+β*D The cost is calculated using, where D denotes the distortion (calculated by 1430), the R bit estimate (the rate estimate output by 1440 or 1450), and λ and β are Lagrangian multipliers. Downsampling 1470 outputs a cost tensor 1480. As known in the art, the Lagrangian multipliers as well as λ and β may be obtained empirically.

[0183] As a result, a tensor 1480 is obtained with cost estimates for each location in the feature map (in this case the W×H locations of the dense optical flow). Note that using sequential average pooling and nearest neighbor upsampling results in averaging of motion vectors in an N×N (e.g., 4×4) region, where N×N is the average pooling filter shape and scaling factor for the upsampling operation. During nearest neighbor upsampling, values ​​from the low resolution layer are replicated (repeated) at all points in the high resolution layer that correspond to the filter shape. This corresponds to a translational motion model.

[0184] Various implementations of the cost selection unit are possible. For example, FIG. 15 shows another exemplary implementation. In this example, unlike FIG. 14, the motion vector field obtained after downsampling 1501 the dense optical flow by 4 in each of the x and y dimensions is not upsampled 1415. Rather, it is directly provided for motion compensation 1510 and rate estimation 1540. Instead, the reference picture 1505 and the target picture 1505 may be downsampled 1515, 1518 to the corresponding resolution before motion compensation 1510 and distortion evaluation 1530. This leads to excluding the step of initial motion field upsampling 1415 to the original resolution in FIG. 14, as well as excluding the final cost downsampling step 1470 in FIG. 14. This also makes the bit normalization 1450 unnecessary. This implementation may require less memory to store tensors during processing, but may give less accurate results. It should be noted that in order to accelerate or reduce the complexity of the RDO, it is possible to downsample the dense optical flow and the reference and target pictures even more than is done by L1, however, the accuracy of such an RDO may be further reduced.

[0185] By applying the cost estimation unit (1400, 1500) to each downsampling layer of the motion segmentation unit (1220, 1310), costs are obtained using different levels of motion vector averaging (different spatial resolutions). As a next step, the signal selection logic 1100 uses the cost information from each downsampling layer to select motion information of different spatial resolutions. To achieve that the signal selection logic 1100 performs pairwise comparison of costs from successive (cascaded) downsampling layers, the signal selection logic 1100 selects the minimum cost at each spatial location and propagates it to the next downsampling layer (in the processing sequence). Figure 16 shows an example architecture of the signal selection unit 1600.

[0186] The dense optical flow 610 enters three downsampling layers downspl4, downspl2 and downspl2, similar to that shown in FIG. 11. The signal selection logic 1600 of FIG. 16 is an example implementation of the signal selection logic 1100 of FIG. 11. In particular, the LayerMv tensor 611 is a subsampled motion vector field (feature map) that enters the cost calculation unit 613. The LayerMv tensor 611 also enters the layer information selection unit 614 of the first layer. The layer information selection unit 614 provides the selected motion vector to the bitstream, if there is a selected motion vector on this (first) layer. Its function is further described below.

[0187] The cost calculation unit 613 calculates the cost, for example as described for the cost calculation unit 1400 with reference to Fig. 14. It outputs a cost tensor, which is then downsampled by 2 to match the resolution at which the second layer operates. After processing by the second downsampling layer downspl2, the LayerMV tensor 621 is provided to the next layer (third layer) and to the cost calculation unit 623 of the second layer. The cost calculation unit 623 operates similarly to the cost calculation unit 1400. Instead of upsampling / downsampling by 4 as in the example described with reference to Fig. 14, downsampling by 2 in each direction is applied, as will be clear to those skilled in the art.

[0188] To perform a pair-wise comparison of the cost tensors from the cost calculation units 613 and 623, the cost tensor from the previous (first) downsampling layer is downsampled (by 2) to the current resolution layer (second). Then, a pooling operation 625 is performed between the two cost tensors. In other words, the pooling operation 625 maintains the lowest cost in the cost tensor per element. The selection of the layer with the lowest cost is captured per element index of the pooling operation result. For example, if at one particular tensor element, the cost of the first tensor has a lower value than the cost of the corresponding element of the second tensor, the index is equal to 0, otherwise the index is equal to 1.

[0189] To ensure gradient propagation for training purposes, a soft arg max can be used to obtain a pooled index with the gradient. If gradient propagation is not required, normal pooling with the index can be used. As a result 622 of the pooling operation 625, an index (LayerFlag tensor) indicating whether a motion vector from the current or previous resolution layer was selected is forwarded to the layer information selection unit 624 of the current (here the second) layer, together with the motion vector (LayerMv tensor) from the corresponding downsampling layer of the motion segmentation unit. The best pooled cost tensor is propagated to the next downsampling level (downspl2), and then the operation is repeated for the third layer.

[0190] In particular, the output LayerMv 621 of the second layer is further downsampled (downspl 2) by the third layer, and the resulting motion vector field LayerMv 631 is provided to the cost calculation unit 633 of the third layer. The calculated cost tensor is propagated from the second layer and compared element-wise 635 with the downsampled cost tensor provided by the MinCost pooling unit 625. After processing by MinCost pooling 635, an index (LayerFlag tensor) indicating whether a motion vector from the current (third) or previous (second) resolution layer is selected is forwarded to the layer information selection unit 634 of the current (here third) layer, together with the motion vector (LayerMv tensor) from the corresponding downsampling layer of the motion segmentation unit. In this example, there are only three layers for illustrative purposes. However, in general there may be more than three layers, and the further layers and the signal selection logic for these layers have similar functions as shown for the second and third layers.

[0191] To collect pooled information from each spatial resolution layer, the following process is performed in reverse order from low to high resolution layers using layer information selection units 634, 624, and 614. First, a TakeFromPrev tensor of the same size as the lowest resolution layer (here the third layer) is initialized with zeros 601. Then the same operation is repeated for layers of different resolutions as follows: The values ​​of the LayerFlag tensor (in the current layer) at the positions where the value of the tensor (NOT TakeFromPrev) is equal to 1 are selected to be transmitted in the bitstream as segmentation information. The (NOT TakeFromPrev) tensor is the element-wise negation of the TakeFromPrev tensor. In the third (here the last) layer, the (NOT TakeFromPrev) tensor therefore has all values ​​set to 1 (negated zeros set by 601). Thus, the segmentation information 1130 (LayerFlag) of the last (here the third) layer is always transmitted.

[0192] The TakeFromCurrent tensor is obtained using the logical operation TakeFromCurrent=(NOT TakeFromPrev) AND LayerFlag. The flag of this tensor TakeFromCurrent indicates whether motion vector information is selected to be sent in the bitstream from the current resolution layer. The layer information selection unit (634, 624, 614) selects motion vector information from the corresponding downsampling layer of the motion segmentation unit by taking the value of the LayerMv tensor, and the value of the TakeFromCurrent tensor is equal to 1. This information is sent in the bitstream as selected information 1120.

[0193] For the third (first in reverse order) processing layer, corresponding to the lowest resolution, TakeFromPrev is initialized by zero and then all flags are sent, then all values ​​of (NOT TakeFromPrev) are equal to 1. For the last processing layer, corresponding to the highest resolution layer, no LayerFlag flag needs to be sent. For all positions where motion information was not selected from the previous layer, it is assumed that a position should be selected from the current layer or the next (highest resolution) layer.

[0194] Note that the cost computation as shown in Fig. 16 is a parallelizable scheme that can be run on a GPU / NPU. This scheme is also trainable since it transfers gradients allowing it to be used in an end-to-end trainable video coding solution.

[0195] Note that the reverse order processing is similar to that performed by the decoder when parsing the segmentation and motion vector information, as shown below when describing the functionality of the decoder.

[0196] Another exemplary implementation of the signal selection logic 1700 is shown in FIG. 17. Compared with FIG. 16, the block diagram of FIG. 17 introduces multiple coding options at the same resolution layer. This is indicated by options 1 to N in the layer 1 cost calculation unit 710. It should be noted that in general, one or more or all layers may include more options. In other words, any of the cost calculation units 613, 623, 633 may provide more options. These options may be, for example, one or more or all of the following: different reference pictures used for motion estimation / compensation, single-hypothesis prediction, bi-hypothesis prediction or multi-hypothesis prediction, different prediction methods, for example, inter-frame prediction or intra-frame prediction, direct coding without prediction, multi-hypothesis prediction, presence or absence of residual information, quantization level of the residual, etc. A cost is calculated for each coding option in the cost calculation unit 710. The best option is then selected using a minimum cost pooling 720. An indicator (e.g., index) 705 of the best selected option is sent to the layer information selection module 730, and then the indicator BestOpt is forwarded in the bitstream if the corresponding point of the current layer is selected for transmitting information. In the given example, options are shown for only the first layer, but it should be understood that similar option selection logic may be applied to other layers of different resolutions or to all layers.

[0197] The above-mentioned approach is also suitable for segmentation and transfer of logical information such as flags or switches that control the picture reconstruction process, and also for information that is intended to remain unchanged after decoding and be kept the same as the encoding side. In other words, instead of the motion vector field (dense optical flow) processed in the exemplary implementation of Figure 16, any one or more other parameters including segmentation can be coded in a similar manner. It may be one or more or all of the following, for example, an indicator indicating different reference pictures used for motion estimation / compensation, a single-hypothesis, bi-hypothesis or multi-hypothesis prediction indicator, different prediction methods, for example, inter-frame or intra-frame prediction, an indicator of direct coding without prediction, multi-hypothesis prediction, the presence or absence of residual information, the quantization level of the residual, the parameters of the in-loop filter, etc.

[0198] Further variations of the above embodiments and examples According to a first variant, the downsampling layer of the motion segmentation unit 1310 and / or the upsampling layer of the motion generation unit 1360 include convolution operations. This is illustrated in FIG. 18. As can be seen in FIG. 18, compared to FIG. 13, the downsampling layer "dwnspl" and the upsampling layer "upspl" are replaced by a downsampling convolution layer "conv↓" in the motion segmentation unit 1810 and an upsampling convolution layer "conv↑" in the motion generation unit 1860, respectively. Part of the advantage of the convolution rescaling (downsampling, upsampling) layer is that it allows for a learnable downsampling and upsampling process. This allows, for example, in the case of use for densification of motion information, to find an optimal upsampling transformation, and thus may reduce the blocking effect caused by motion compensation using motion vector information averaged per block, as described in the above-mentioned embodiments and examples. The same is applicable to texture restoration processes, for example, original image intensity values ​​or prediction residual generation processed by cascaded layers.

[0199] In the above example of Figure 18, all downsampling and upsampling layers are convolutional layers. In general, this disclosure is not limited to such implementations. In general, within the segmentation unit (1310, 1810) and / or within the generation unit (1360, 1860), a subset (one or more) of the downsampling and corresponding upsampling operations may be implemented as convolutions.

[0200] It should be recalled that although the examples described herein are provided for dense optical flow / motion vector field processing and thus refer to motion segmentation units (1310, 1810) and / or motion generation units (1360, 1860), the present disclosure is not limited to such data / feature maps. Instead, any coding parameters, or textures such as samples of an image, or prediction residuals (prediction errors), etc., can be processed instead of or in addition to the motion vector field in any of the embodiments and examples herein.

[0201] For example, note that an encoder using motion information averaging as downsampling can be used in combination with a decoder having a convolutional upsampling layer. Furthermore, an encoder with a convolutional layer aimed at finding a better latent representation can be combined with a motion generation network (decoder) that implements a nearest neighbor-based upsampling layer. Further combinations are possible. In other words, the upsampling layer and the downsampling layer do not have to be of similar types.

[0202] According to a second variant, which can be combined with any of the above-mentioned embodiments and examples (as well as the first variant), the processing by the network includes one or more additional convolution layers between the cascade layers having different resolutions described above. For example, the motion segmentation unit 1310 and / or the motion generation unit 1360 further include one or more intermediate convolution layers between some or all of the downsampling layer and the upsampling layer. This is illustrated in FIG. 19, which shows an exemplary implementation of such a motion segmentation network (module) 1910 and a motion generation network (module) 1860. It should be noted that the terms "module" and "unit" are used interchangeably in this specification to indicate functional units. In this particular embodiment, the units 1910 and 1960 are more specifically network structures having multiple cascade layers.

[0203] For example, the motion segmentation unit 1910 has an additional convolution layer "conv" before each downsampling layer "conv↓" (which may be other types of downsampling) compared to the motion segmentation unit 1310. Furthermore, the motion generation unit 1960 has an additional convolution layer "conv" before each upsampling layer "conv↑" (which may be other types of upsampling) compared to the motion generation unit 1360.

[0204] This may further reduce blocking artifacts caused by sparsifying the motion information and increase the generalization effect of finding a better latent representation. Similar to what has been described above with respect to the first variant, the encoders and decoders from the different embodiments / variations described above can be combined into one compression system. For example, it is possible to have only an encoder with additional layers between the downsampling layers and a decoder without such additional layers, or vice versa. Alternatively or additionally, it is possible to have different numbers and positions of such additional layers in the encoder and the decoder.

[0205] According to the third variant, a direct connection with the input and output signals is provided, as also shown in Fig. 19. It should be noted that although shown here in the same figure, the second and third variants are independent. They may be applied together or separately to the previously described embodiments and examples as well as to other variants. The direct connection is shown by dashed lines.

[0206] In some embodiments, in addition to the bottleneck information of the autoencoder's latent representation (the output of the lowest resolution layer), information from the high resolution layer(s) is added to the bitstream. To optimize the signaling overhead, only a part of the information from different resolution layers is inserted into the bitstream, controlled by the signal selection logic. At the receiving (decoder) side, the corresponding signal supply logic supplies the information from the bitstream to the layers of different spatial resolution, as described in more detail below. Furthermore, information from the input signal before the downsampling layer can be added to the bitstream, which can further increase variability and flexibility. For example, the coding may be aligned to real object boundaries and segments with high spatial resolution, adjusted to the characteristics of a particular sequence.

[0207] According to a fourth variant, the shape of the downsampling filter and the upsampling filter can have shapes other than squares, for example rectangular with horizontal or vertical direction, asymmetric shapes, or even arbitrary shapes by adopting a mask operation. This allows to further increase the variability of the segmentation process to better capture the actual object boundary. This variant is illustrated in FIG. 20. In the motion segmentation unit 2010, a first downsampling layer, which can be the same as any of the previous embodiments, is followed by two further downsampling layers adopting filters selected from a filter shape set. This variant is not limited to the processing of motion vector information.

[0208] In general, downsampling by layers applies downsampling an input feature map using a first filter to obtain a first feature map, and downsampling the input feature map using a second filter to obtain a second feature map. The cost calculation includes determining a third cost resulting from reconstructing a portion of the reconstructed picture using the first feature map, and determining a fourth cost resulting from reconstructing a portion of the reconstructed picture using the second feature map. Then, in the selection, the first feature map is selected if the third cost is smaller than the fourth cost, and the second feature map is selected if the third cost is larger than the fourth cost. In this example, the selection was out of two filters. However, the present disclosure is not limited to two filters, rather, the selection from a predetermined number of filters may be performed in a similar manner, for example, by estimating the costs of all selectable filters and selecting the filter that minimizes the cost.

[0209] The shape of the first filter and the second filter can be either a square, a rectangle with horizontal and vertical orientation. However, the present disclosure is not limited to these shapes. Generally, any filter shape can be designed. The filter can further include a filter that can be defined with any desired shape. Such a shape can be indicated by obtaining a mask, the mask being composed of flags, the mask representing any filter shape, and one of the first filter and the second filter (generally any of the selectable filters from the filter set) has any filter shape.

[0210] In an exemplary implementation, to provide variability, the encoder further includes pooling between cost tensors obtained with the help of filters having different shapes. The index of the selected filter shape is signaled in the bitstream as (part of) the segmentation information, similar to what was described above for the motion vectors. For example, in case of a selection between a horizontally oriented rectangular shape and a vertically oriented rectangular shape, a corresponding flag may be signaled in the bitstream. For example, the method of selecting multiple coding options described with reference to FIG. 17 may be used for the selection of different filter shapes in the same resolution layer.

[0211] According to a fifth variant, a motion model from a predefined set of different motion models may be selected at the same resolution layer. In the previous embodiment, the specific case of downsampling and / or upsampling filters was described. In such a case, the motion information may be averaged over a square block representing a translational motion model. It should be noted that in this fifth variant, in addition to the translational motion mode, other motion models may be adopted. Such other motion models may be: -Affine motion model, -Higher order motion models, or -CNN layers specifically trained to represent certain motion models, e.g. zoom, rotation, affine, perspective, etc. It may include one or more of them.

[0212] In an exemplary implementation of the fifth variant, the autoencoder further includes a set of CNN layers and / or "handcrafted" layers that represent other than translational motion models. Such an autoencoder (and decoder) is shown in Figure 21. In Figure 21, a layer including a set of filters, shown as "conv fit set," is provided on the encoder side and the decoder side.

[0213] For example, at each spatial layer, the encoder selects an appropriate filter(s) from a set of filters that corresponds to a certain motion model and inserts an indication into the bitstream. At the receiving side, signaling logic interprets the indicator and performs the convolution at a particular layer using the corresponding filter(s) from the set.

[0214] The example methods described above use motion information, in particular motion vectors, as an exemplary input for encoding. It should be noted again that these methods are also applicable to compressing different types of image or video information, such as direct image sample values, prediction residual information, intra-frame and inter-frame prediction parameters, etc.

[0215] According to a sixth variant, the RDO exemplified above with reference to FIG. 16 or FIG. 17 may be applied to a conventional block-based codec.

[0216] Conventional video coding methods, such as state-of-the-art video coding standards like AVC, HEVC, VVC or EVC, use a block-based coding concept, according to which a picture is recursively divided into blocks of square or rectangular shape. For these blocks, signal reconstruction parameters are estimated or evaluated at the encoder side and transmitted in a bitstream to the decoder. Typically, an encoder aims to find optimal reconstruction parameters for a set of blocks representing a picture in terms of rate-distortion cost, and tries to maximize the reconstruction quality (i.e., minimize the distortion of the original picture) and minimize the amount of bits required to transmit the parameters for the reconstruction process. This task of parameter selection (or coding mode decision) is a complex and resource-demanding task, which is also the main source of encoder complexity. Due to processing time constraints, for example in real-time applications, an encoder may sacrifice the quality of the mode decision, which affects the quality of the reconstructed signal. Optimizing the mode decision process is always a desirable technical improvement.

[0217] One of the coding mode decisions is whether to split the current block (or coding unit (CU)) into multiple blocks according to a partitioning method.

[0218] According to a sixth variant, the above-mentioned motion segmentation unit 1310 (or 1810) is adapted to a partition mode decision based on cost minimization (eg rate-distortion optimization criterion). Figure 22 shows an example of such optimization.

[0219] As seen in Fig. 22, instead of downsampling the layers, a block partition structure is used to represent information of different spatial resolutions. For each block of a given size NxN (considering square blocks) of a picture or part of a picture, the cost calculation unit calculates a distortion tensor and further downsamples the resolution by a factor of 16 (to match the original resolution). In the given example of Fig. 22, the first block size is 16x16, and for example, the downsampling is performed by an average pooling operation to obtain a tensor, each element of which represents the average distortion per 16x16 block. In the first layer, the picture is divided into 16x16 blocks 2201 at the initial highest resolution. In the second layer, the resolution is reduced so that the block size 2202 in the picture is 32x32 (corresponding to combining 4 blocks of the previous layer). In the third layer, the resolution is reduced again so that the block size 2203 is 64x64 (corresponding to combining 4 blocks of the previous layer). It should be noted that the combination of four blocks from the previous layer in this case can be considered as a subsampling of the block-related information, since in the first layer, block-related information is provided for each 16x16 block, whereas in the second layer, block-related information is provided only for 32x32 blocks, i.e., four times fewer parameters are provided. Similarly, in the third layer, block-related information is provided only for 64x64 blocks, i.e., four times fewer parameters are provided than in the second layer and 16 times fewer parameters are provided than in the first layer.

[0220] Block-related information in this context is any information that is coded per block, such as prediction mode, motion vectors, prediction direction, prediction mode specific information such as reference pictures, filtering parameters, quantization parameters, transformation parameters, or other settings that may vary at the block (coding unit) level.

[0221] Then, the cost calculation units 2211, 2212, and 2213 of the first layer, second layer, and third layer, respectively, calculate costs based on the block reconstruction parameters of the block sizes 2201, 2202, and 2203, respectively, and the input picture of size W×H.

[0222] The output cost tensor is obtained as the average distortion per block combined with an estimate of the bits required to transmit the coding parameters of the N×N block (e.g., in the first layer 16×16) using a Lagrangian multiplier. An exemplary structure of a cost calculation unit 2300 for a block N×N (which may correspond to each or any of the cost calculation units 2211, 2212, and 2213) is shown in FIG.

[0223] FIG. 23 shows an exemplary block diagram of a cost calculation unit 2300 for a common block size 230x of N×N. The cost calculation unit 2300 obtains 2310 block reconstruction parameters (block-related parameters) associated with a block of size N×N. This obtaining may correspond to fetching parameters (parameter values) from a memory or the like. For example, the block-related parameters may be a particular prediction mode, such as an inter-prediction mode. In unit 2310, the block reconstruction parameters are obtained, and in reconstruction unit 2320, a part of the picture is reconstructed using these parameters (in this example, all blocks are reconstructed using the inter-prediction mode). Then, a distortion calculation unit 2330 calculates the distortion of the reconstructed part of the picture by comparing it with the corresponding part of a target picture, which may be the original picture to be coded. Since the distortion may be calculated per sample, a downsampling of the distortion (to one value per N×N block) 2340 may be performed to obtain it on a block basis. In the lower branch, the rate or number of bits required to code the picture is estimated 2360. In particular, the bit estimation unit 2360 can estimate the number of bits signaled per N×N sized block. For example, the number of bits per block required for inter prediction modes may be calculated. Once the estimated distortion and bit amount (or rate) are obtained, a cost can be calculated 2350, for example based on the Lagrangian optimization described above. The output is a cost tensor.

[0224] It should be noted throughout this specification that the term "tensor" may refer to a matrix when only a 2D image of a sample is observed, such as a grayscale image. However, there may be multiple channels, such as color or depth channels for a picture, so that the output may have more dimensions. A general feature map may also be provided in more than two or three dimensions.

[0225] The same cost evaluation procedure is performed for the first layer (with 16x16 block granularity) and for the next level of quadtree division into blocks of size 32x32 samples. To decide whether it is better to use one 32x32 block or four 16x16 blocks for the reconstruction parameters (block-related parameters), the cost tensor evaluated for the 16x16 block is downsampled by a factor of 2 (see FIG. 22). Then, a minimum cost pooling operation 2222 provides the optimal decision for each 32x32 block. An index of the pooled costs is passed to the layer information selection unit 2232 to be transmitted in the bitstream as split_flags. The reconstruction parameters blk_rec_params for the block best selected according to the pooled index are also passed to the layer information selection unit 2231. The pooled cost tensor is further passed (downsampled by 2) to the next quadtree aggregation level of blocks with size 64x64, i.e., MinCost pooling 2223. The MinCost pooling 2223 also receives the costs calculated for 64x64 block resolution 2203 in the cost calculation unit 2213. It passes the index of the pooled costs as split_flags to the layer information selection unit 2233 as indicated in the bitstream. It also passes the reconstructed parameters blk_rec_params for the best selected block according to the pooled index to the layer information selection unit 2233.

[0226] To collect pooled information from each block aggregation level, processing is performed in reverse order from the top (highest in this example) aggregation level (64 x 64 samples) to the bottom (lowest in this example) aggregation level (16 x 16 samples) using layer information selection units 2233, 2232, and 2231 in the manner described above with reference to Figure 16.

[0227] The result is a bitstream that encodes the quadtree partitioning obtained by the optimization together with the coded values ​​of the obtained partitions (blocks) and possibly further coding parameters. The above-mentioned method allows the determination of the partition flags of the block partitions. To obtain the reconstruction parameters of each block, conventional methods based on the evaluation of each or some of the possible coding modes can be used.

[0228] FIG. 24 shows an example of the seventh variant. The seventh variant is an extension of the sixth variant described above with reference to FIG. 22 and FIG. 23. The seventh variant represents a manner in which the evaluation of coding modes is incorporated into the design. In particular, as can be seen, the cost calculation unit 710 can evaluate N options. Note that the term "N" is here a placeholder for some integer. The "N" indicating the number of options does not need to be the same as the "N" in "N×N" indicating a common block size. In the cost calculation unit 710, for the same level of block partitioning, for example, for blocks of size 16×16 samples (as in the first layer), the encoder iterates over all possible (or a limited set of) coding modes for each block.

[0229] Consider an encoder with N options for coding each 16x16 block, denoted as blk_rec_params 0, blk_rec_params 1, ..., blk_rec_params N. The parameter combination blk_rec_params k (k is an integer from 0 to N) may be, for example, a combination of some prediction modes (e.g., from inter and intra), some transforms (e.g., from DCT and KLT), some filtering orders or filter coefficient sets (of predefined filters), etc. In some implementations, blk_rec_params k may be the value k of a single parameter, in case only one parameter is optimized. As will be apparent to one skilled in the art, any one or more parameters can be optimized by checking its usage cost.

[0230] With each given set of block reconstruction parameters (blk_rec_params_k), the cost calculation unit 2410 calculates a tensor representing the cost of each block. Then, using a minimum cost pooling 2420, the best coding mode for each block is selected and forwarded to the layer information selection unit 2430. The best pooled cost tensor is further downsampled by a factor of 2 and forwarded to the next quadtree aggregation level (in this case the second layer corresponding to aggregation with block size 32x32). Then, the splitting (partitioning) decision is made in a manner similar to the sixth variant above. In FIG. 24, options 0..N are evaluated only at the first layer (aggregation level 16x16). However, the present disclosure is not limited to such an approach. Rather, the evaluation of options 0..N may be performed at each aggregation level.

[0231] For example, at the next level of quadtree aggregation (32x32, 64x64), the encoder evaluates (by calculating the cost in each cost unit) and pools (by each MinCost pooling unit) the best coding mode for each block (not shown in the figure for clarity), which is compared with the previous aggregation level. The decision on the best mode and the corresponding reconstruction parameters set accordingly are provided to a layer information selection unit (such as the layer information selection unit 2430 shown for the first layer). To collect pooled information from each block aggregation level, the process is performed in reverse order from the upper aggregation level (64x64) to the lower aggregation level (16x16) using the layer information selection unit, as described in the sixth variant.

[0232] Different block shapes can be used to represent more advanced partitioning methods such as binary tree, ternary tree, asymmetric and geometric partitioning. Figure 25 illustrates such partitioning of blocks. In other words, optimization may not necessarily be performed only over different block sizes, but also for different partitioning types (e.g., by corresponding options). Figure 25 shows the following examples: - Quadtree partitioning 2510. In quadtree partitioning, a block is split (partitioned) into four blocks of equal size. - (Symmetric) Binary Tree Split 2520. In a symmetric binary split, a block is split into two blocks of equal size. The split can be a vertical or horizontal split. Vertical or horizontal is an additional parameter of the split. - Asymmetric Binary Tree Split 2530. In asymmetric binary splitting, a block is split into two blocks of different sizes. The size ratio may be fixed (to save overhead caused by signaling) or variable (in which case some ratio options may also be optimized, i.e. configurable). - Ternary tree partitioning 2540. In ternary partitioning, a block is divided into three partitions by two vertical lines or two horizontal lines, where vertical or horizontal is an additional parameter of the partitioning.

[0233] The present disclosure is not limited to these exemplary partitioning modes: triangular partitions or any other type of partitions may be employed.

[0234] In a seventh variant, a hybrid architecture applicable to common video coding standards is supported and empowered by a powerful (neural) network-based approach. The technical advantage of the described method is that it provides a highly parallelizable GPU / NPU-friendly scheme that can enable speeding up the calculations required for the mode decision process. It can enable global picture optimization since multiple blocks are considered at the same decision level, and incorporates a learnable part to speed up the decision, for example to evaluate the amount of bits required for reconstruction parameter coding.

[0235] In summary, processing with a cascaded layer structure according to the sixth or seventh variant includes processing data for the same picture segmented (i.e., divided / split) into blocks having different block sizes and / or shapes in different layers. Selecting a layer includes selecting a layer based on a cost calculated for a given set of coding modes.

[0236] In other words, different layers can process picture data with different block sizes. Thus, a cascade layer includes at least two layers that process different block sizes from each other. When referring to a "block" herein, what is meant is a unit, i.e., a portion of a picture on which coding is performed. A block may also be referred to as a coding unit, a processing unit, etc.

[0237] The predetermined set of coding modes corresponds to a combination of coding parameter values. Different block sizes may be evaluated in a single set of coding modes (combinations of values ​​of one or more coding parameters). Alternatively, the evaluation may include various combinations of block sizes and partition shapes (such as those in FIG. 25). However, the present disclosure is not limited thereto, and as specifically mentioned in the seventh variant, for example, there may be multiple predetermined sets of coding modes (combinations of coding parameter values) for each block, which may further include coding modes such as intra / inter prediction type, intra prediction mode, residual skip, residual data, etc.

[0238] For example, the process includes determining, for at least one layer, costs for different sets of coding modes (combinations of values ​​of coding parameters) and selecting one of the set of coding modes based on the determined costs. Figure 24 illustrates the case where only the first layer performs such a selection. However, this is not limited to this. At high speed, each cost calculation unit can have the same structure as the first cost calculation unit 2410, including 0..N options. This is not shown in the figure for simplicity.

[0239] As mentioned above, this is a GPU-friendly RDO that can be performed by the codec and selects the best coding mode for each block. In Fig. 24, the input image (picture) is the same in each layer. However, the coding of the picture (to calculate the cost) is performed in each layer with different block sizes. Apart from the block size, for one or more of the block sizes, further coding parameters can be tested and selected based on the RDO.

[0240] In particular, in these variations, the indication of data related to the selected layer includes a selected set of coding modes (eg, blk_rec_params).

[0241] In summary, in some embodiments, an encoder may be provided that corresponds in structure to a neural network autoencoder for video or image information coding. Such an encoder may be configured to analyze input image or video information by a neural network that includes layers of different spatial resolutions, to transfer in the bitstream latent representations that correspond to the lowest resolution layer outputs, and to transfer in the bitstream outputs other than the lowest resolution layer.

[0242] Decryption The encoder described above provides a bitstream that includes selected layer feature data and / or segmentation information. Correspondingly, the decoder processes data received from the bitstream in multiple layers. In addition, the selected layer receives additional (direct) input from the bitstream. The input may be some feature data information and / or segmentation information.

[0243] Correspondingly, in the following, embodiments are described that focus on the information related to the selected layer being feature data. Other described embodiments focus on the information related to the selected layer being segmentation information. There are also mixed embodiments in which the bitstream carries both feature data and segmentation information and the layers process both feature data and segmentation information.

[0244] As a simple example, a decoder of a neural network autoencoder may be provided for video or image information coding. The decoder may be configured to read latent representations corresponding to low resolution layer inputs from a bitstream, obtain layer input information based on corresponding information read from the bitstream for layers other than the low resolution layer(s), obtain combined inputs for the layer based on the layer information obtained from the bitstream and outputs from previous layers, feed the combined inputs to the layer, and synthesize an image based on the layer outputs.

[0245] Here, the term "low-resolution" refers to layers that process feature maps with low resolution, e.g., feature maps in the latent space provided from the bitstream. The low resolution may in fact be the lowest resolution of the network.

[0246] The decoder may be further configured to obtain segmentation information based on corresponding information read from the bitstream, and obtain a combined input for the layer based on the segmentation information. The segmentation information may be a quadtree, a dual (binary) tree, a ternary tree data structure, or a combination thereof. The layer input information may correspond to, for example, motion information, image information, and / or prediction residual information, etc.

[0247] In some examples, information obtained from the bitstream corresponding to the layer input information is decoded using a hyper-a priori neural network. Information obtained from the bitstream corresponding to the segmentation information may be decoded using a hyper-a priori neural network.

[0248] The decoder can be easily applied to the decoding of motion vectors (e.g., motion vector fields or optical flow). Some of those motion vectors may be similar or correlated. For example, in a video showing an object moving across a constant background, there may be two groups of similar motion vectors. The first group of motion vectors may be vectors used in predicting pixels showing the object, and the second group may be vectors used to predict pixels of the background. Therefore, instead of signaling all the motion vectors in the encoded data, it may be beneficial to signal groups of motion vectors to reduce the amount of data representing the encoded video. This may allow signaling a representation of the motion vector field that requires a smaller amount of data.

[0249] 9 shows a bitstream 930 received at a decoder and generated by an encoder as described above. On the decoding side, the decoder portion of the system 900, in some embodiments, includes signal providing logic 940 that interprets segmentation information obtained from the bitstream 930. According to the segmentation information, the signal providing logic 940 identifies a particular (selected) layer, spatial size (resolution) and location for a portion of a feature map where corresponding selected information (also obtained from the bitstream) should be placed.

[0250] It should be noted that in some embodiments, the segmentation information is not necessarily processed by the cascade network. It may be provided independently or may be derived from other parameters in the bitstream. In other embodiments, the feature data is not necessarily processed in the cascade network, but the segmentation information is. Thus, the two sections "Decoding with Feature Information" and "Decoding with Segmentation Information" describe examples of such embodiments, as well as combinations of such embodiments.

[0251] It should be noted that for the embodiments in both sections, the above encoder-side variants (1 to 7) are correspondingly applied to the decoder-side. For better clarity, the additional features of the variants are not copied in both sections. However, as will be clear to those skilled in the art, they may be applied alternatively or in combination to the decoding approaches in both sections.

[0252] Decoding using feature information In this embodiment, a method for decoding data for picture or video processing from a bitstream is provided, as shown in Figure 33. Correspondingly, an apparatus for decoding data for picture or video processing from a bitstream is provided. The apparatus may include a processing circuit configured to perform the steps of the method.

[0253] The method includes obtaining 3310 two or more sets of feature map elements from the bitstream, each set of feature map elements associated with a (respective) feature map. The obtaining may be performed by parsing the bitstream. In some example implementations, the bitstream parsing may also include entropy decoding. This disclosure is not limited to any particular method of obtaining data from the bitstream.

[0254] The method further includes a step 3320 of inputting each of the two or more sets of feature map elements into two or more feature map processing layers of the plurality of cascaded layers, respectively.

[0255] Cascade layers may form part of a processing network. In this disclosure, the term "cascade" means that the output of one layer is subsequently processed by another layer. The cascade layers need not be directly adjacent (the output of one of the cascade layers goes directly into the input of the second of the cascade layers). With reference to FIG. 9, data from the bit stream 930 is input to signal feed logic 940, which feeds sets of feature map elements to the appropriate layers (indicated by arrows) 953, 952, and / or 951. For example, a first feature element set is inserted into the first layer 953 (first in the processing sequence) and a second feature element set is inserted into the third layer 951. It is not necessary that the sets be inserted into the second layer. The number of layers and their positions (in the processing order) may vary and the disclosure is not limited to any particular one.

[0256] The method further includes obtaining 3330 the decoded data for picture or video processing as a result of processing by multiple cascaded layers. For example, the first set is a latent feature map element set that is processed by all layers of the network. The second set is an additional set that is provided to another layer. Referring to Figure 9, the decoded data 911 is obtained after processing the first set by three layers 953, 952, and 951 (in that order).

[0257] In an exemplary implementation, a feature map is processed in each of two or more feature map processing layers, and the feature maps processed in each of the two or more feature map processing layers differ in resolution, e.g., a first feature map processed by a first layer has a different resolution than a second feature map processed by a second layer.

[0258] In particular, the processing of the feature maps in two or more feature map processing layers includes upsampling. Figure 9 shows a network whose decoder includes three (directly) cascaded upsampling layers 953, 952, and 951.

[0259] In an exemplary implementation, the decoder includes only upsampling layers of different spatial resolutions, and a nearest neighbor approach is used for upsampling. The nearest neighbor approach repeats low-resolution values ​​in areas of high resolution corresponding to a given shape. For example, if one element of low resolution corresponds to four elements of high resolution, the value of one element is repeated four times in the high resolution area. In this case, the term "corresponding" means describing the same area in the highest resolution data (initial feature map, initial data). Such a method of upsampling allows information to be transmitted from the low-resolution layer to the high-resolution layer without modification, which may be suitable for some kind of data such as logical flags or indicator information, or information that is desired to remain the same as that obtained at the encoder side without modification by some convolutional layers, for example. An example of such data is prediction information, such as motion information that may include motion vectors estimated at the encoder side, reference indexes indicating which particular picture from the reference picture set should be used, prediction modes indicating whether a single reference frame or multiple reference frames should be used, or a combination of different predictions such as combined intra-inter prediction, presence or absence of residual information, etc.

[0260] However, the present disclosure is not limited to upsampling performed by a nearest neighbor approach. Alternatively, upsampling may be performed by applying some interpolation or extrapolation, or by applying convolution, etc. These approaches may be particularly suitable for upsampling data that is expected to have smooth characteristics, such as motion vectors or residuals or other sample-related data.

[0261] In FIG. 9, the encoder (e.g., 911-920) and the decoder (e.g., 940-951) have the same number of downsampling and upsampling layers, and the nearest neighbor method may be used for upsampling, and average pooling may be used for downsampling. The shape and size of the pooling layer are matched with the scale factor of the upsampling layer. In some other possible implementations, another method of pooling, such as max pooling, may be used.

[0262] As already illustrated in several encoder embodiments, the data for picture or video processing may include a motion vector field. For example, FIG. 12 shows the encoder and decoder side. At the decoder side, the bitstream 1250 is analyzed and motion information 1260 (possibly including segmentation information described below) is obtained therefrom. The obtained motion information is then provided to a motion generation network 1270. The motion generation network can increase the resolution of the motion information, i.e., densify the motion information. The reconstructed motion vector field (e.g., dense optical flow) 1275 is then provided to a motion compensation unit 1280. The motion compensation unit 1280 uses the reconstructed motion vector field to obtain predictive picture / video data based on the reference frame(s) and reconstructs a motion-compensated frame based thereon (e.g., by adding a decoded residual, as illustrated in the decoder part of the encoder in FIG. 5A, or the reconstruction unit 314 in FIG. 7B).

[0263] Figure 13 also shows a decoder-side motion generation (densification) network 1360. Network 1360 includes signal feeding logic 1370, similar in function to signal feeding logic 940 of Figure 9, and three upsampling (processing) layers. The main difference with the embodiment described above with reference to Figure 9 is that in Figure 13 network 1360 is dedicated to processing motion vector information to output a motion vector field.

[0264] As mentioned above, according to one embodiment, the method further includes obtaining segmentation information related to two or more layers from the bitstream. Then, obtaining feature map elements from the bitstream is based on the segmentation information. Inputting the set of feature map elements to two or more feature map processing layers, respectively, is based on the segmentation information. Some detailed examples regarding the use of segmentation information in analysis and processing are provided below in the section on decoding using segmentation information. For example, Figures 28 and 29 provide very specific (just exemplary) layer processing options.

[0265] In some embodiments, the multiple cascade layers further include multiple segmentation information processing layers. The method further includes processing the segmentation information in the multiple segmentation information processing layers. For example, the processing of the segmentation information in at least one of the multiple segmentation information processing layers includes upsampling. Such upsampling of the segmentation information and / or the upsampling of the feature map includes nearest neighbor upsampling in some embodiments. In general, the upsampling applied to the feature map information may be different from the upsampling applied to the segmentation information. Furthermore, since the upsampling in the same network may be different, one network (segmentation information processing or feature map processing) may include different types of upsampling layers. Such examples are shown in, for example, FIG. 20 or FIG. 21. It should be noted that the upsampling types other than nearest neighbor may include some interpolation approaches, such as polynomial approaches, e.g., bilinear, cubic, etc.

[0266] According to an exemplary implementation, said upsampling of segmentation information and / or said upsampling of feature maps comprises a (transposed) convolution. This corresponds to the first variant described above for the encoder. FIG. 18 shows a motion generation unit 1869 on the decoder side, which comprises a convolution operation "conv↑" instead of nearest neighbor upsampling. This allows a learnable upsampling process, e.g., to find the optimal upsampling transformation when using it for densification of motion information, and may reduce the blocking effect caused by motion compensation using block-wise averaged motion vector information, as described above with reference to the encoder. The same is applicable to a texture restoration process, e.g., for original image intensity values ​​or prediction residual generation. The motion generation unit 1869 also comprises signal supply logic, which corresponds functionally to the signal supply logic 940 of FIG. 9 or the signal supply logic 1370 of FIG. 13.

[0267] FIG. 30 shows a block diagram of an exemplary decoder-side layer processing according to the first variant. In particular, a bitstream 3030 is analyzed, and a signal providing logic 3040 (functionally corresponding to the signal providing logic 940 or 1370) provides a selection instruction to the convolutional upsampling filter 300. In some embodiments, the convolutional filter is selectable from a set of N filters (denoted as filters 1-N). The filter selection may be based on information indicating the selected filter and parsed from the bitstream. An indication of the selected filter may be given (generated and inserted into the bitstream) by the encoder based on an optimization approach such as RDO. In particular, the RDO illustrated in FIG. 17 or FIG. 24 may be applied (treating the filter size / shape / order as one of the options, i.e., as a coding parameter to be optimized). However, the present disclosure is not limited thereto, and in general, the filter may be derived based on other signaled parameters (such as coding mode, interpolation direction, etc.).

[0268] In summary, the signal supply logic unit controls the inputs of different layers with different filter shapes and selectively bypasses the layer output to the next layer according to the segmentation and motion information obtained from the bitstream. The convolution filter unit 3000 corresponds to a convolution by one layer. Multiple such convolution swamp filters can be cascaded as shown in FIG. 18. It should be noted that the present disclosure is not limited to variable or trainable filter settings. In general, the convolution upsampling can also be performed using a fixed convolution operation.

[0269] Aspects of this embodiment can be combined with aspects of other embodiments. For example, an encoder with motion information averaging in a downsampling layer can be used in combination with a decoder that includes a convolutional upsampling layer. An encoder with a convolutional layer aimed at finding a better latent representation can be combined with a motion generation network that includes a nearest neighbor-based upsampling layer. Further combinations are possible. In other words, the implementation of the encoder and decoder need not be symmetric.

[0270] Figure 32A shows two examples of reconstruction applying the nearest neighbor approach. In particular, Example 1 shows the case where the flag value of the segmentation information of the lowest resolution layer is set (to 1). Correspondingly, the motion information indicates one motion vector. Since the motion vector is already indicated in the lowest resolution layer, no further motion vector and no further segmentation information are signaled in the bitstream. From the one signaled motion vector, the network generates respective motion vector fields with high resolution (2x2) and highest resolution (4x4) by copying it during the nearest neighbor upsampling. The result is a 4x4 area with all 16 motion vectors identical and equal to the signaled motion vector.

[0271] FIG. 32B shows two examples of reconstruction applying a convolutional layer-based approach. Example 1 has the same input as Example 1 in FIG. 32A. In particular, the segmentation information of the lowest resolution layer has a flag value set (to 1). Correspondingly, the motion information shows one motion vector. However, instead of simply copying one motion vector, after applying a (possibly trained) convolutional layer, the motion vectors in the upper layer and the top layer are not completely identical.

[0272] Similarly, Example 2 in Figure 32A shows segmentation information 0 in the lowest resolution layer and segmentation information 0101 in the next (higher resolution) layer. Accordingly, two motion vectors for the positions indicated by the segmentation information are signaled in the bitstream as motion information. These are shown in the middle layer. As seen in the bottom layer, the signaled motion vectors are each copied four times to cover the highest resolution area. The remaining eight motion vectors of the highest resolution (bottom) layer are signaled in the bitstream. Example 2 in Figure 32B applies convolution instead of nearest neighbor copying. Motion vectors are no longer copied. The transition between the motion vectors copied in Figure 32A is now somewhat smoother, allowing for blocking artifact reduction.

[0273] Similar to the second variant described above for the encoder, in the decoder, the multiple cascaded layers include convolutional layers that do not perform upsampling between layers with different resolutions. Note that the encoder and decoder are not necessarily symmetric in this respect: the encoder may have such an additional layer and the decoder may not, or vice versa. Of course, the encoder and decoder may be designed symmetrically and may have additional layers between the corresponding downsampling and upsampling layers of the encoder and decoder.

[0274] With respect to the combination of segmentation information processing and feature map processing, obtaining feature map elements from the bitstream is based on processed segmentation information processed by at least one of a plurality of segmentation information processing layers. The segmentation layer may analyze and interpret the segmentation information as described in more detail below in the section on decoding using segmentation information. It should be noted that the embodiments and examples described herein may be applied in combination with the embodiments of this section. In particular, the layer processing of segmentation information described with reference to Figures 26 to 32B below may be performed in combination with the feature map processing described herein.

[0275] For example, inputting each of the two or more sets of feature map elements into two or more feature map processing layers respectively is based on processed segmentation information processed by at least one of the multiple segmentation information processing layers. The obtained segmentation information is represented by a set of syntax elements, and the position of an element in the set of syntax elements indicates to which feature map element position the syntax element relates. The set of syntax elements is a bitstream portion that may be binarized using an entropy code, such as a fixed code, a variable length code, or an arithmetic code, any of which may be context-adaptive. The present disclosure is not limited to any particular coding or form of the bitstream, as long as it has a predefined structure known to both the encoder and decoder sides. In this way, the parsing and processing of the segmentation information and the feature map information may be performed in conjunction. For example, the processing of the feature map includes, for each syntax element, (i) parsing from the bitstream the element of the feature map on the position indicated by the syntax element's position in the bitstream when the syntax element has a first value, and (ii) otherwise (or, more generally, when the syntax element has a second value), bypassing parsing from the bitstream the element of the feature map on the position indicated by the syntax element's position in the bitstream. The syntax elements may be binary flags that are ordered into the bitstream at the encoder and parsed in the correct order from the decoder by a particular layer structure of the processing network.

[0276] Note that options (i) and (ii) may also be provided for non-binary syntax elements. In this case, a first value means parsing and a second value means bypassing. The syntax element may take some further values ​​apart from the first and second values. These may also result in parsing or bypassing, or may indicate a particular type of parsing, etc. The number of parsed feature map elements may correspond to the amount of the syntax element that is equal to the first value.

[0277] According to an exemplary implementation, for each layer 1 < j < N of a plurality of N feature map processing layers, processing of the feature map by the j-th feature map processing layer includes analyzing the segmentation information element of the j-th feature map processing layer from the bitstream, obtaining the feature map processed by the previous feature map processing layer, further analyzing the feature map element from the bitstream, and associating the analyzed feature map element with the obtained feature map. The position of the feature map element in the processed feature map is indicated by the analyzed segmentation information element and the segmentation information processed by the previous segmentation information processing layer. The association can be, for example, replacement of a previously processed feature map element, or combination, such as addition, subtraction, or multiplication. Some exemplary implementations are provided below. The analysis may depend on the previously processed segmentation information, which provides the possibility of a very compact and efficient syntax.

[0278] For example, the method may include analyzing the feature map element from the bitstream when the syntax element has a first value, and bypassing the analysis of the feature map element from the bitstream when the syntax element has a second value or when the segmentation information processed by the previous segmentation information processing layer has the first value. This means that the analysis is bypassed when the relevant part has been analyzed in the previous layer. For example, the syntax element analyzed from the bitstream representing the segmentation information is a binary flag. As described above, it may be beneficial that the processed segmentation information is represented by a set of binary flags. The set of binary flags is a sequence of binary flags having a value of either 1 or 0 (corresponding to the first value and the second value described above).

[0279] In some embodiments, the upsampling of the segmentation information in each segmentation information processing layer j further includes determining, for each p-th position in the obtained feature map indicated by the input segmentation information, an indication of the feature map position that is included in the same region in the reconstructed picture as the p-th position as the upsampled segmentation information, which provides a spatial relationship between the reconstructed image (or reconstructed feature map or data in general), the positions in the subsampled feature map, and the corresponding segmentation flag.

[0280] As already mentioned above, similar to the encoder embodiment, the data for picture or video processing may include picture data (such as picture samples) and / or prediction residual data and / or prediction information data. Note that when referring to "residuals" in this disclosure, these may be pixel domain residuals or transform (spectral) coefficients (i.e., transformed residuals, residuals represented in a domain different from the sample / pixel domain).

[0281] Similar to the fourth variant described above for the encoder side, according to an exemplary implementation, a filter is used in upsampling the feature map, and the shape of the filter is one of a square, a horizontal rectangle, and a vertical rectangle. Note that the filter shape may be similar to the partition shape shown in FIG. 25.

[0282] An exemplary decoder-side layer processing is shown in Figure 20. A motion generation network (unit) 2060 includes signal supply logic and one or more (here two) upsampling layers using filters (upsampling filters) that can be selected from a set of predetermined or predefined filters. The selection can be performed at the encoder side, for example by the RDO or by other settings, and signaled in the bitstream. At the decoder side, an indication of filter selection is parsed from the bitstream and applied. Alternatively, the filter can be selected at the decoder without explicit signaling based on other coding parameters derived based on the bitstream. Such parameters can be any parameters that are correlated with the content, such as prediction type, direction, motion information, residual, loop filtering characteristics, etc.

[0283] FIG. 31 shows a block diagram of an upsampling filter unit 3100 supporting the selection of one of N filters 1 to N. The signaling of the filter selection may directly include an index of one of the N filters. It may include the filter orientation, the filter order, the filter shape, and / or the coefficients. At the decoding side, the signal feeding logic interprets the filter selection flag (e.g., an orientation flag to distinguish between vertical and horizontal filters or further orientations) and feeds the feature map value(s) to the layer with the corresponding filter shape set. In FIG. 31, a direct connection from the signal feeding logic to the selective bypass logic allows not to select any of the filters. The corresponding value of the filter selection indicator may also be signaled or derived in the bitstream.

[0284] In general, a filter is used in upsampling the feature map, and inputting information from the bitstream further includes obtaining information indicating a filter shape and / or a filter orientation and / or a filter coefficient from the bitstream. There may be implementations where each layer has a set of filters to select from, or where each layer is a filter and the signal supply logic determines which layer should be selected and which layer should be bypassed based on a filter selection flag (indicator).

[0285] In some embodiments, a flexible filter shape may be provided in that the information indicating the filter shape indicates a mask composed of flags, where a flag having a third value indicates a non-zero filter coefficient, and a flag having a fourth value different from the third value indicates a zero filter coefficient, in that the mask represents the filter shape. In other words, as already described for the encoder side, the filter shape may be defined by indicating the position of the non-zero coefficients. The non-zero coefficients may be derived based on a predefined rule or may also be signaled.

[0286] The above decoder embodiment may be implemented as a computer program product stored on a non-transitory medium, which, when executed on one or more processors, performs the steps of any of the above methods. Similarly, the above decoder embodiment may be implemented as a device for decoding an image or video, including a processing circuit configured to perform the steps of any of the above methods. In particular, a device for decoding data for picture or video processing from a bitstream may be provided, the device including: an acquisition unit configured to acquire two or more sets of feature map elements from the bitstream, each set of feature map elements relating to a feature map; an input unit configured to input each of the two or more sets of feature map elements to two or more feature map processing layers of a plurality of cascaded layers, respectively; and a decoded data acquisition unit configured to acquire decoded data for picture or video processing as a result of processing by the plurality of cascaded layers. These units may be implemented in software or hardware, or as a combination of both, as described in more detail below.

[0287] Decoding using segmentation information At the receiving side, the decoder of this embodiment performs analysis and interpretation of the segmentation information. Thus, as shown in Figure 34, a method for decoding data for picture or video processing from a bitstream is provided. Correspondingly, an apparatus for decoding data for picture or video processing from a bitstream is provided. The apparatus may include a processing circuit configured to perform the steps of the method.

[0288] The method includes obtaining 3410 two or more sets of segmentation information elements from the bitstream. The obtaining may be performed by parsing the bitstream. In some example implementations, the bitstream parsing may also include entropy decoding. The present disclosure is not limited to any particular method of obtaining data from the bitstream. The method further includes inputting 3420 each of the two or more sets of segmentation information elements into two or more segmentation information processing layers of the multiple cascaded layers, respectively. It should be noted that the segmentation information processing layer may be the same layer as the feature map processing layer or a different layer. In other words, one layer may have one or more functions.

[0289] Further, in each of the two or more segmentation information processing layers, the method includes processing a respective set of segmentation information. Obtaining 3430 the decoded data for picture or video processing is based on the segmentation information processed by the multiple cascaded layers.

[0290] FIG. 26 shows an example segmentation information for three-layer decoding. The segmentation information may be considered as selecting the layer at which the feature map element is analyzed or otherwise obtained (see the encoder-side description). The feature map element 2610 is not selected. Thus, the flag 2611 is set to 0 by the encoder. This means that the feature map element 2610 with the lowest resolution is not included in the bitstream. However, the flag 2611 indicating that the feature map element is not selected is included in the bitstream. For example, if the feature map element is a motion vector, this may mean that the motion vector 2610 of the largest block is not selected and is not included in the bitstream.

[0291] 26, in feature map 2620, of the four feature map elements used to determine the feature map elements of feature map 2610, three feature map elements are selected for signaling (indicated by flags 2621, 2622, and 2624) and one feature map element 2623 is not selected. In an example using motion vectors, this may mean that three motion vectors are selected from feature map 2620 and their respective flags are set to 1, while one feature map element is not selected and its respective flag 2623 is set to 0.

[0292] The bitstream may then include all four flags 2621-2624 and the three selected motion vectors. In general, the bitstream may include the four flags 2621-2624 and the three selected feature map elements. In the feature map 2630, one or more of the elements that determine the non-selected feature map elements of the feature map 2620 may be selected.

[0293] In this example, when a feature map element is selected, none of the elements of the high resolution feature map are selected. In this example, none of the feature map elements of feature map 2630 used to determine the feature map element signaled by flags 2621, 2622, and 2624 are selected. In one embodiment, none of these feature map element flags are included in the bitstream. Rather, only the flags of the feature map elements of feature map 2630 that determine feature map element 2623 having a flag are included in the bitstream.

[0294] In an example where the feature map elements are motion vectors, feature map elements 2621, 2622, and 2624 may each be determined by a group of four motion vectors each in feature map 2630. In each of the groups that determine motion vectors using flags 2621, 2622, and 2624, the motion vectors may have more similarity to each other than the four motion vectors in feature map 2620 that determine the motion vectors (feature map elements) in the unselected feature map 2630 (signaled by flag 2623).

[0295] Figure 26 has been described above with respect to the properties of the bitstream. It should be noted that the decoder will decode (parse) such bitstream accordingly, i.e., the decoder will determine what information is included (signaled) based on the values ​​of the flags as explained above, and will parse / interpret the parsed information accordingly.

[0296] In an exemplary implementation, the segmentation information is organized as shown in Figure 27. For 2D information such as images or videos considered as a sequence of images, the feature map of a layer may be represented in two-dimensional space. The segmentation information includes an indicator (binary flag) for a location in the 2D space that indicates whether the feature map value corresponding to this location is presented in the bitstream.

[0297] In FIG. 27, there is a starting layer (layer 0) for decoding segmentation information, e.g., the lowest resolution layer, i.e., the latent representation layer. In this starting layer, each 2D location contains a binary flag. If such flag is equal to 1, the selected information contains the feature map value for this location in this particular layer. On the other hand, if such flag is equal to 0, there is no information for this location in this particular layer. This set of flags (or generally a tensor of flags, here a matrix of flags) is called TakeFromCurrent. The TakeFromCurrent tensor is upsampled to the next layer resolution, e.g., using a nearest neighbor method. Let this tensor be TakeFromPrev. The flag in this tensor indicates whether the corresponding sample location was filled in the previous layer (here layer 0) or not.

[0298] As a next step, the signaling logic reads the flag of the position of the current resolution layer (LayerFlag). In this example implementation, only the positions that were not filled (not filled with feature map element value(s) that were not set to 1) in the previous layer are signaled. Using logical operations, it can be expressed as TakeFromPrev==0 or !TakeFromPrev==1, where "!" represents the logical NOT operation (negation).

[0299] The amount of flags needed for this layer can be calculated as the amount of zero (logical false) elements in the TakeFromPrev tensor, or the amount of values ​​with 1 (logical true) in the inverse (!TakeFromPrev) tensor. For non-zero elements in the TakeFromPrev tensor, no flags are needed in the bitstream. This is indicated in the figure by showing "-" at the positions that do not need to be read. From an implementation perspective, it may be easier to calculate the sum of the elements on the inverse tensor as sum(!TakeFromPrev). The signaling logic can use this arithmetic to identify how many flags need to be parsed from the bitstream. A read flag is placed at the positions where the value of TakeFromPrev is equal to 1 to obtain the LayerFlag tensor. The TakeFromCurrent tensor for the current resolution layer (here Layer1) is then obtained as a combination of the TakeFromPrev tensor and the LayerFlag tensor by keeping the flags at the positions read from the bitstream for the current resolution layer and zeroing the values ​​at the positions read in the previous resolution layer (positions marked by "-" in LayerFlag). This can be expressed and implemented using a logical AND operator as TakeFromCurrent=!TakeFromPrev AND LayerFlag. Then, to take into account the positions read in the previous resolution layer, the TakeFromCurrent tensor is obtained using a logical OR operation as TakeFromCurrent=TakeFromCurrent OR TakeFromPrev. It should be understood that Boolean operations can be implemented using regular mathematical operations, e.g., multiplication for AND and addition for OR. This gives the advantage of storing and transferring gradients, allowing the above method to be used in end-to-end training.

[0300] The obtained TakeFromCurrent tensor is then upsampled to the next resolution layer (here layer 2) and the above operations are repeated.

[0301] For generality and ease of implementation, it is beneficial to unify the processing of all resolution layers without special consideration of the first resolution layer where all flags are parsed from the bitstream. This can be achieved by initializing TakeFromPrev with zero before processing the first (low resolution) layer (Layer0) and repeating the steps described above for each resolution layer.

[0302] To further reduce the signaling overhead, in some further implementations, the LayerFlags for the last resolution layer (here the third layer, i.e. Layer2) does not need to be transferred (included in the encoder and parsed in the decoder) to the bitstream. This means that for the last resolution layer, the feature map values ​​are sent in the bitstream as selected information (see 1120 in FIG. 11) for all positions of the last resolution layer that were not acquired (in any of the previous resolution layers) in (all) previous resolution layers. In other words, for the last resolution layer, TakeFromCurrent=!TakeFromPrev, i.e. TakeFromCurrent corresponds to negated TakeFromPrev. Again, to keep the generality of the processing of the last resolution layer, LayerFlag can be initialized with 1, and the same formula TakeFromCurrent=!TakeFromPrev AND LayerFlag can be used.

[0303] In some further possible implementations, the last resolution layer has the same resolution as the original image. If the last resolution layer does not have additional processing steps, this means transmitting some values ​​of the original tensor and bypassing compression in the autoencoder.

[0304] In the following, an example of the signal providing logic 2800 is described with reference to FIG. 28. In FIG. 28, the signal providing logic 2800 of the decoder uses the segmentation information (LayerFlag) to obtain and utilize the selected information (LayerMv) transmitted in the bitstream. In particular, in each layer, the bitstream is parsed in the respective syntax interpretation units 2823, 2822, and 2821 (in that order) to obtain the segmentation information (LayerFlag) and possibly the selected information (LayerMv). As mentioned above, in the first layer (syntax interpretation 2823), the TakeFromPrev tensor is initialized to all zeros in 2820 to enable the same operation as the other layers. The TakeFromPrev tensor is propagated in the order of processing from the syntax interpretation of the previous layer (e.g., from 2823) to the syntax interpretation of the subsequent layer (e.g., 2822). The propagation here involves upsampling by a factor of 2, as described above with reference to FIG.

[0305] While interpreting the segmentation information (LayerFlag) at each resolution layer, a tensor TakeFromCurrent is obtained (generated). This tensor TakeFromCurrent contains a flag indicating whether feature map information (LayerMv) is present in the bitstream for each particular position of the current resolution layer. The decoder reads the feature map LayerMv values ​​from the bitstream and places them in the TakeFromCurrent tensor at positions where the flag is equal to 1. The total amount of feature map values ​​included in the bitstream for the current resolution layer can be calculated based on the amount of non-zero elements in TakeFromCurrent, or as sum(TakeFromCurrent) - the sum over all elements of the TakeFromCurrent tensor. As a next step, the tensor combination logic 2813, 2812, and 2811 of each layer combine (e.g., at 2812) the output of the previous resolution layer (e.g., generated at 2813 and upsampled at 2801 to match the resolution of the next layer processing 2812) by replacing the feature map values ​​at the locations where the value of TakeFromCurrent tensor is equal to 1 with the feature map value (LayerMv) sent in the bitstream as selected information. As mentioned above, in order to enable the same operations in the first layer (tensor combination 2813) as in the other layers, the combined tensor is initialized to all zeros at 2810. After processing (at 2811) the LayerFlags from all layers and generating the output tensor of the last layer, the combined tensor is upsampled by 4 at 2801 to obtain the original size of the dense optical flow, which is W×H.

[0306] The example implementation of Figure 28 provides a fully parallelizable scheme that can run on a GPU / NPU and allow exploiting parallelism. The fully trainable scheme of gradient transfer allows its use in an end-to-end trainable video coding solution.

[0307] Figure 29 shows another possible example implementation of the signal supply logic 2900. This implementation generates a LayerIdx tensor (called LayerIdxUp in Figure 29) that contains the indexes of layers of different resolutions that indicate which layers should be used to obtain the motion information (included in the encoder and parsed in the decoder) that is forwarded in the bitstream. In each syntax interpretation block (2923, 2922, 2923), the LayerIdx tensor is updated by adding the TakeFromCurrent tensor multiplied by the upsampling layer index numbered from the highest resolution to the lowest resolution. The LayerIdx tensor is then upsampled and forwarded (passed) to the next layer in the processing order, for example from 2923 to 2922 and from 2922 to 2921. To make the processing in all layers similar, the tensor LayerIdx is zero-initialized in 2920 and passed to the syntax interpretation 2923 of the first layer.

[0308] After the last (here the 3rd) layer, the LayerIdx tensor is upsampled to the original resolution (upsampling 2995 by 4). As a result, each location in LayerIdx contains the index of the layer to take motion information from. The LayerIdx location corresponds to the original resolution of the feature map data (here dense optical flow) at the same resolution, which is 2D (matrix) in this example. So for each location in the reconstructed optical flow, LayerIdx specifies where to take the motion information from (from which layer's MayerMV).

[0309] The motion information (LayerMv, also referred to as LayerMvUp in FIG. 29) is generated as follows: At each spatial resolution layer, the tensor combination block (2913, 2912, 2911) combines the LayerMv obtained from the bitstream (passed through the respective syntax interpretation units 2923, 2922, 2921) with an intermediate tensor based on the segmentation information (LayerFlag) obtained from the bitstream and an intermediate TakeFromCurrent Boolean tensor according to the method described above. The intermediate tensors can be initialized by zero (see initialization units 2910, 2919, 2918) or any other value. The initial values ​​are not important because these values ​​are not selected for the dense optical flow reconstruction 2990 by this method after all steps are finally completed. The combined tensor containing the motion information (output from each of 2913, 2912, 2911) is upsampled and concatenated with the combined tensor of the previous spatial resolution layer (2902, 2901). The concatenation is performed along an additional dimension corresponding to the motion information obtained from the layers of different resolution (i.e., the 2D tensor before concatenation 2902 becomes a 3D tensor after concatenation, and the 3D tensor before concatenation 2901 remains a 43D tensor after concatenation, but the size of the tensor increases). Finally, after completion of all upsampling steps for LayerIdxUp and LayerMvUp, a reconstructed dense optical flow is obtained by selecting the motion information from LayerMvUp using the value of LayerIdxUp as an index on the axis in LayerMvUp, where the axis is the dimension increased during the LayerMvUp concatenation step. In other words, the added dimension in LayerMvUp is the dimension over the number of layers, and LayerIdxUp selects the appropriate layer for each position.

[0310] The above specific exemplary implementations do not limit the present disclosure. Generally, segmentation can be performed in various possible ways and signaled in the bitstream. Generally, obtaining a set of segmentation information elements is based on segmentation information processed by at least one segmentation information processing layer of multiple cascaded layers. Such layers may include a syntax interpretation unit (2823, 2822, 2821) that parses / interprets the meaning (semantic) of the parsed segmentation information LayerFlag, as shown in FIG. 28.

[0311] More specifically, the input of the set of segmentation information elements is based on the processed segmentation information output by at least one of the multiple cascade layers. This is illustrated, for example, in FIG. 28 by the passing of the TakeFromPrev tensor between the syntax interpretation units (2823, 2822, 2821). As already described in the description of the encoder side, in some exemplary implementations, the segmentation information processed in two or more segmentation information processing layers respectively differs in resolution.

[0312] Furthermore, the processing of segmentation information in two or more segmentation information processing layers includes upsampling, as already illustrated with reference to FIG. 9, FIG. 13, etc. For example, said upsampling of segmentation information includes nearest neighbor upsampling. It should be noted that in this embodiment as well as the previous embodiment, the present disclosure is not limited by applying nearest neighbor upsampling. The upsampling may include interpolation rather than simple copying of neighboring sample (element) values. The interpolation may be any known interpolation, such as linear or polynomial, for example, cubic upsampling. It should be noted that with respect to copying, the copy performed by nearest neighbor is a copy of element values ​​from a predetermined (available) nearest neighbor, for example, from above or to the left. If there are neighbors at the same distance from the position to be filled, a predefinition of such neighbors to copy from may be required.

[0313] As described above for the first variant, in some exemplary implementations, the upsampling includes transposed convolution. In addition to or instead of applying convolutional upsampling to the feature map information, convolutional upsampling may also be applied to the segmentation information. Note that the upsampling type performed on the segmentation information is not necessarily the same as the upsampling type applied to the feature map elements.

[0314] In general, for each segmentation information processing layer j of the plurality of N segmentation information processing layers from the plurality of cascade layers, inputting - if j=1, input initial segmentation information from the bitstream (and / or based on initialization, e.g., initialization to 0 in 2820), otherwise input segmentation information processed by the (j-1)th segmentation information processing layer; - outputting the processed segmentation information.

[0315] This is the segmentation information related to the input layer and is not necessarily (still potentially) the entire segmentation information from the bitstream. The upsampled segmentation information in the j-th layer is the segmentation information upsampled in the j-th layer, i.e., the segmentation information output by the j-th layer. Generally, the processing by the segmentation layer includes upsampling (TakeFromPrev) and including new elements from the bitstream (LayerFlag).

[0316] For example, the processing of the input segmentation information by each layer j < N of a plurality of N segmentation information processing layers further includes parsing the segmentation information element (LayerFlag) from the bitstream and associating the parsed segmentation information element with the segmentation information (TakeFromPrev) output by the previous layer (e.g., in the syntax interpretation unit 282x of FIG. 28). The position of the parsed segmentation information element (LayerFlag) within the relevant segmentation information is determined based on the segmentation information output by the previous layer. As can be seen in FIGS. 28 and 29, there can be various different ways of associating and propagating the position information. The present disclosure is not limited to any particular implementation form.

[0317] For example, the amount of segmentation information elements parsed from the bitstream is determined based on the segmentation information output by the preceding layer. In particular, if a region has already been covered by segmentation information from a previous layer, the region does not need to be covered again on a subsequent layer. It is noted that this design provides an efficient parsing approach. Each position of the resulting reconstructed feature map data corresponding to a position of the resulting reconstructed segmentation information is associated only with segmentation information belonging to a single layer (out of N processing layers). This means that there is no duplication. However, the present disclosure is not limited to such an approach. It is considered that the segmentation information is duplicated, even though it may lead to maintaining some redundancy.

[0318] As already shown in Figure 27, in some embodiments, the parsed segmentation information element is represented by a set of binary flags. The ordering (syntax) of the flags in the bitstream may convey an association between the flags and the layer to which they belong. The ordering (sequence) may be given by a predefined order of processing at the encoder and, correspondingly, at the decoder. These are illustrated, for example, in Figures 16 and 28.

[0319] In some exemplary embodiments, for example, the embodiment described above with reference to the seventh variant, obtaining the decoding data for picture or video processing includes determining at least one of the following parameters based on the segmentation information: The segmentation information may determine the analysis of additional information such as coding parameters, which may include, as well as the motion information, intra-picture or inter-picture prediction mode, picture reference index, single-reference or multiple-reference prediction (including bi-prediction), presence or absence prediction residual information, quantization step size, motion information prediction type, motion vector length, motion vector resolution, motion vector prediction index, motion vector difference size, motion vector difference resolution, motion interpolation filter, in-loop filter parameters, and / or post-filter parameters. In other words, the segmentation information, when processed by the segmentation information processing layer, may specify from which processing layer of the coding parameters the coding parameters may be obtained. For example, in the approach of the encoder described above in FIG. 22 or FIG. 23, the reconstruction (coding) parameters may be received from the bitstream instead of (or in addition to) the motion information (LayerMv). Such reconstruction (coding) parameters blk_rec_params may be parsed in the decoder in the same way as illustrated in Figures 28 and 29 for motion information.

[0320] Generally, the segmentation information is used for analyzing and inputting feature map elements (either the motion information or the above-mentioned reconstruction parameters or sample-related data). The method may further include obtaining a set of feature map elements from the bitstream, and inputting the set of feature map elements into a feature map processing layer of the plurality of layers, respectively, based on the segmentation information processed by the segmentation information processing layer. Furthermore, the method further includes obtaining decoded data for picture or video processing based on the feature map processed by the plurality of cascade layers. In particular, in some embodiments, at least one of the plurality of cascade layers is a segmentation information processing layer and a feature map processing layer. As described above, the network may be designed with a separated segmentation information processing layer and a feature map processing layer, or with a combined layer having both functions. In some implementations, each layer of the plurality of layers is either a segmentation information processing layer or a feature map processing layer.

[0321] The above-mentioned methods may be embodied as a computer program product stored in a non-transitory medium, which, when executed on one or more processors, causes the processor to perform any of the steps of the methods. Similarly, a device for decoding an image or video is provided, including a processing circuit configured to perform any of the method steps of the above-described methods. The functional structure of the device also provided by the present disclosure may correspond to the functions provided by the above-mentioned embodiments and steps. For example, a device for decoding data for picture or video processing from a bitstream is provided, the device including: an acquisition unit configured to acquire two or more sets of segmentation information elements from the bitstream; an input unit configured to input each of the two or more sets of segmentation information elements into two or more segmentation information processing layers of a plurality of cascaded layers, respectively; a processing unit configured to process each set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data acquisition unit configured to acquire the decoded data for picture or video processing based on the segmentation information processed in the plurality of cascaded layers. These units and further units may perform all the functions of the methods described above.

[0322] Overview of Some Embodiments Encoding using feature or segmentation information According to one aspect of the present disclosure, there is provided a method for encoding data for image or video processing into a bitstream, the method including: processing the data, the processing including generating feature maps in a plurality of cascaded layers, each feature map having a respective resolution, where at least two of the generated feature maps have different resolutions from each other; selecting from the plurality of layers a layer different from a layer generating a feature map with a lowest resolution; and generating a bitstream including inserting information related to the selected layer into the bitstream.

[0323] Such a method may provide improved efficiency of such encoding by allowing data from different layers to be encoded, thus allowing different resolution features or other types of layer-related information to be included in the bitstream.

[0324] According to one aspect of the present disclosure, a device for encoding data for image or video processing into a bitstream includes a processing unit configured to process the data, where the processing includes generating feature maps of different resolutions in a plurality of cascaded layers, each feature map including a respective resolution, a selection unit configured to select a layer from the plurality of layers that is different from the layer generating the lowest resolution feature map, and a generation unit configured to generate the bitstream, where the selection unit includes inserting an indication of data related to the selected layer into the bitstream. The processing unit, the selection unit, and the generation unit may be implemented by one or more processing circuits, such as processors, or any combination of software and hardware.

[0325] Such a device may allow data from different layers to be decoded and used for reconstruction, thus providing improved efficiency of such decoding, since it allows the use of features of different resolution or other types of layer-related information.

[0326] In an exemplary implementation, the processing further includes downsampling by one or more of the cascaded layers. The application of downsampling allows, on the one hand, a reduction in processing complexity, and, on the other hand, can also reduce the data provided in the bitstream. Furthermore, layers processing different resolutions can thus focus on features of different scales. Thus, a network processing pictures (still or video) can operate efficiently.

[0327] For example, one or more of the downsampling layers may include average pooling or max pooling for downsampling. Average pooling and max pooling operations are part of several frameworks, which provide efficient means for downsampling with low complexity.

[0328] In another example, convolution is used in downsampling. Convolution may provide some more advanced methods of downsampling with kernels that may be appropriately selected for a particular application or may even be trainable. This allows for a learnable downsampling process that allows for finding a more appropriate latent representation of the motion information, while maintaining the advantage of representing and transferring information at different spatial resolutions, increasing adaptability.

[0329] In an example implementation, the information related to a selected layer includes elements of a feature map for that layer.

[0330] By providing features with different resolutions, the scalability of the encoding / decoding is increased and the bitstream so generated can provide more flexibility to meet optimization criteria such as rate, distortion, complexity, etc., ultimately providing the potential for increased coding efficiency.

[0331] In any of the above examples, for example, the information associated with the selected layer may include information indicating from which layer and / or from which portion of the feature map of that layer the feature map elements of that layer were selected.

[0332] Signaling segmentation information may provide efficient coding of feature maps from different layers, such that each area of ​​the original (coded) feature map (data) may be covered by information from only one layer. Although this is not a limitation of the invention, the invention may also provide overlap between layers of certain regions in the encoded feature map (data) in some cases.

[0333] The above-mentioned method, in an exemplary implementation, includes a step of obtaining data to be encoded, where processing of the data to be encoded includes processing by each layer j of a plurality of N cascaded layers, obtaining the data to be encoded as a layer input when j=1, and otherwise obtaining a feature map processed by the (j-1)th layer as a layer input; processing the obtained layer input, where the processing includes downsampling; and outputting the downsampled feature map.

[0334] Correspondingly, in an exemplary implementation, the above-mentioned apparatus has a processing unit configured to obtain data to be encoded and perform processing of the data to be encoded, where the processing of the data to be encoded includes processing by each layer j of a plurality of N cascaded layers, where the processing includes obtaining the data to be encoded as a layer input when j=1, and otherwise obtaining a feature map processed by the (j-1)th layer as a layer input, processing the obtained layer input, where the processing includes downsampling, and outputting the downsampled feature map.

[0335] The method according to any of the previous examples, in some embodiments, is to select information for insertion into the bitstream, the information being related to a first region in the feature map processed by layer j>1, the first region corresponding to a region in the feature map encoded in a layer smaller than j that contains a plurality of elements or in the initial data, and to exclude from the selection a region corresponding to the first region from selections in the feature map processed by layer k (where k is an integer greater than or equal to 1 and k<j).

[0336] The apparatus according to any of the foregoing examples, in some embodiments, is to select information for insertion into the bitstream, the information being related to a first region in the feature map processed by layer j>1, the first region corresponding to a region in the feature map encoded in a layer smaller than j that contains a plurality of elements or in the initial data, and to exclude from the selection a region corresponding to the first region from selections in the feature map processed by layer k (where k is an integer greater than or equal to 1 and k<j), and further includes a processing circuit configured to perform the foregoing.

[0337] Such a selection in a layer does not cover the area of the original feature map covered by other layers and can be particularly efficient with respect to coding overhead.

[0338] In any of the above examples, for example, the data to be encoded includes image information and / or prediction residual information and / or prediction information.

[0339] Alternatively, the information related to the selected layer includes prediction information.

[0340] In any of the above examples, for example, the data related to the selected layer includes an indication of the position of the feature map elements within the feature map of the selected layer.

[0341] Such a display enables appropriate association of feature map elements of different resolutions with the input data region.

[0342] In any of the above examples, for example, the locations of selected and non-selected feature map elements may be indicated by a number of binary flags based on the location of the flags in the bitstream.

[0343] Binary flags provide a particularly efficient way of coding the segmentation information.

[0344] According to one embodiment, in the above-mentioned method or apparatus, the processing by layer j of the plurality of N cascaded layers includes: determining a first cost resulting from reconstructing a portion of the reconstructed picture using the feature map elements output by the jth layer; determining a second cost resulting from reconstructing the portion of the picture using the feature map elements output by the (j-1)th layer; and if the first cost is higher than the second cost, selecting the (j-1)th layer and selecting information relating to the portion in the (j-1)th layer.

[0345] Providing an optimization that includes distortion provides an efficient means for achieving the desired quality.

[0346] For example, the first cost and the second cost include the amount of data and / or the distortion. Optimization by considering the rate (the amount of data generated by the encoder) and the distortion of the reconstructed picture allows for flexible meeting of the requirements of different applications or users.

[0347] Alternatively or additionally, the data to be coded is a motion vector field. The methods described above are readily applicable to compress motion vector fields such as dense optical flow or subsampled optical flow. Application of these methods may provide efficient coding (in terms of rate and distortion or other criteria) of the motion vectors, allowing further reduction in the bitstream size of the coded picture or video data.

[0348] In some embodiments, the prediction information includes a reference index and / or a prediction mode. In addition to or instead of the motion vector field, further information regarding prediction may be processed. The reference index and the prediction mode may be correlated with the content of the picture, similar to the motion vector field, and thus the coding of feature map elements with different resolutions may improve efficiency.

[0349] For example, the amount of data may include the amount of data required to transmit data related to the selected layer. In this manner, the overhead generated by providing information related to a layer other than the output layer may be taken into account during optimization.

[0350] Additionally or alternatively, the distortion is calculated by comparing the reconstructed picture with a target picture. Such an end-to-end quality comparison ensures that the distortion in the reconstructed image is properly taken into account. Thus, the optimization can select a coding approach in an efficient way and can meet the quality requirements imposed by the application or user in a more accurate way.

[0351] In any of the above examples, for example, the processing includes additional convolutional layers between cascaded layers having different resolutions.

[0352] The provision of such additional layers in a cascaded layer network makes it possible to introduce additional processing, such as various types of filtering, to improve the quality or efficiency of the coding.

[0353] According to an example implementation, a processing circuit of a method or apparatus based on the aforementioned embodiment includes, in the downsampling by layer, downsampling an input feature map using a first filter to obtain a first feature map and downsampling the input feature map using a second filter to obtain a second feature map; determining a third cost resulting from reconstructing a portion of the reconstructed picture using the first feature map; determining a fourth cost resulting from reconstructing a portion of the reconstructed picture using the second feature map; and in the selecting, selecting the first feature map if the third cost is less than the fourth cost.

[0354] The application of different downsampling filters can help to adapt to different characteristics of the content.

[0355] For example, the shape of the first filter and the second filter may be a square, a horizontally oriented rectangle, or a vertically oriented rectangle.

[0356] These filters are still of simple shape but can provide additional improvements in adapting to object boundaries.

[0357] The method steps or steps performed by the processing circuitry of the apparatus may further include obtaining a mask, the mask consisting of flags, the mask representing an arbitrary filter shape, and one of the first and second filters having the arbitrary filter shape.

[0358] This provides the flexibility to design filters of any shape.

[0359] The method steps or steps performed by the processing circuitry of the apparatus may further include processing data relating to a same picture segmented into blocks having different block sizes and shapes in different layers, where selecting includes selecting a layer based on a cost calculated for a given set of coding modes.

[0360] In some example implementations, the processing includes determining, for at least one layer, costs for different sets of coding modes and selecting one of the sets of coding modes based on the determined costs.

[0361] The application of optimizations to coding modes may allow for efficient rate-distortion optimization and thus improved coding efficiency.

[0362] For example, the indication of data relating to a selected layer may include a selected set of coding modes.

[0363] According to one aspect of the present disclosure, a computer program stored on a non-transitory medium includes code that, when executed on one or more processors, performs the steps of any of the methods presented above.

[0364] According to one aspect of the present disclosure, there is provided a device for encoding images or videos, the device including a processing circuit configured to perform a method according to any of the examples presented above.

[0365] Any of the above mentioned devices can be embodied on an integrated chip. The present invention can be implemented in hardware (HW) and / or software (SW). Furthermore, HW-based implementations can be combined with SW-based implementations.

[0366] It should be noted that the present disclosure is not limited to any particular framework. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, image generation, and recognition systems.

[0367] For clarity, any one of the above-described embodiments may be combined with any one or more of the other above-described embodiments to create new embodiments within the scope of the present disclosure.

[0368] Decoding using feature map elements According to one embodiment, there is provided a method for decoding data for picture or video processing from a bitstream, the method including: obtaining, from the bitstream, two or more sets of feature map elements, each set of feature map elements relating to a feature map; inputting each of the two or more sets of feature map elements respectively to two or more feature map processing layers of a plurality of cascade layers; and obtaining the decoded data for picture or video processing as a result of processing by the plurality of cascade layers.

[0369] Such a method may provide improved efficiency because it allows data from different layers to be used in decoding, thus enabling features or other kinds of layer-related information to be parsed from the bitstream.

[0370] For example, a feature map may be processed in each of two or more feature map processing layers, and the feature maps processed in each of the two or more feature map processing layers may differ in resolution.

[0371] In some embodiments, the processing of the feature maps in two or more feature map processing layers includes upsampling.

[0372] The application of upsampling allows, on the one hand, a reduction in the processing complexity (since the first layer has a low resolution) and, on the other hand, it can also reduce the data provided in the bitstream and analyzed in the decoder. Furthermore, layers processing different resolutions can thus focus on features of different scales. Thus, networks processing pictures (still or video) can operate efficiently.

[0373] In an example implementation, the method further includes obtaining, from the bitstream, segmentation information related to the two or more layers, wherein obtaining the feature map elements from the bitstream is based on the segmentation information, and inputting the sets of feature map elements to the two or more feature map processing layers, respectively, is based on the segmentation information.

[0374] Using segmentation information can provide efficient decoding of feature maps from different layers, so that each area of ​​the (reconstructed) original resolution can be covered by information from only one layer. Although this is not a limitation of the present invention, the present invention can also provide overlapping between layers of certain areas in the feature map (data) in some cases. For example, the multiple cascade layers further have multiple segmentation information processing layers, and the method further includes processing the segmentation information in the multiple segmentation information processing layers. Such an approach provides the possibility of controlling the analysis of feature elements from different layers.

[0375] In some embodiments, the processing of the segmentation information in at least one of the multiple segmentation information processing layers includes upsampling. The hierarchical structure of the segmentation information allows for a small amount of side information to be inserted into the bitstream, thus improving efficiency and / or processing time.

[0376] For example, the upsampling of segmentation information and / or the upsampling of feature maps includes nearest neighbor upsampling, which has low computational complexity and can be easily implemented. Moreover, it is efficient, especially for logical representations such as flags.

[0377] In some embodiments and examples, the upsampling of the segmentation information and / or the upsampling of the feature maps includes transposed convolution. The use of convolution can help reduce blocking artifacts and can enable a trainable solution where the upsampling filter is selectable.

[0378] In an exemplary implementation, obtaining the feature map elements from the bitstream is based on processed segmentation information processed by at least one of a plurality of segmentation information processing layers.

[0379] In an exemplary implementation, inputting each of the two or more sets of feature map elements into two or more feature map processing layers, respectively, is based on processed segmentation information processed by at least one of the multiple segmentation information processing layers.

[0380] According to one embodiment, the obtained segmentation information is represented by a set of syntax elements, a position of an element in the set of syntax elements indicating which feature map element position the syntax element is associated with, and the processing of the feature map includes, for each of the syntax elements, parsing from the bitstream the feature map element on the position indicated by the position of the syntax element in the bitstream when the syntax element has a first value, and bypassing parsing from the bitstream the feature map element on the position indicated by the position of the syntax element in the bitstream otherwise.

[0381] Such a relationship between the segmentation information and the feature map information enables both efficient coding of frequency information and analysis in a hierarchical structure considering different resolutions.

[0382] For example, the processing of the feature map by each layer 1 < j < N of a plurality of N feature map processing layers includes analyzing the segmentation information element of the j-th feature map processing layer from the bitstream, obtaining the feature map processed by the preceding feature map processing layer, analyzing the feature map element from the bitstream, and associating the analyzed feature map element with the obtained feature map. The position of the feature map element in the processed feature map is indicated by the analyzed segmentation information element and the segmentation information processed by the preceding segmentation information processing layer.

[0383] In particular, when the syntax element has a first value, analyzing the element of the feature map from the bitstream, and when the syntax element has a second value or the segmentation information processed by the preceding segmentation information processing layer has a first value, bypassing the analysis of the element of the feature map from the bitstream.

[0384] For example, the syntax element analyzed from the bitstream representing the segmentation information is a binary flag. In particular, the processed segmentation information is represented by a set of binary flags.

[0385] Providing binary flags enables efficient coding. On the decoder side, the processing of logical flags can be performed with low complexity.

[0386] According to an exemplary implementation, upsampling of the segmentation information in each segmentation information processing layer j further includes, for each p-th position in the obtained feature map indicated by the input segmentation information, determining as the upsampled segmentation information an indication of a feature map position that falls within the same area in the reconstructed picture as the p-th position.

[0387] For example, data for picture or video processing includes motion vector fields. Since dense optical flow or motion vector fields with a resolution similar to that of a picture are desirable for modeling motion, the hierarchical structure of the present invention is easily applicable and efficient for reconstructing such motion information. With layer processing and signaling, a good tradeoff between rate and distortion can be achieved.

[0388] For example, the data for picture or video processing includes picture data and / or prediction residual data and / or prediction information data. The present disclosure may be used for a variety of different parameters. However, the picture data and / or prediction residual data and / or prediction information data may still have some redundancy in the spatial domain, and the layered approach described herein may provide efficient decoding from a bitstream using different resolutions.

[0389] In some embodiments and examples, a filter is used in upsampling the feature map, and the shape of the filter is one of a square, a horizontal rectangle, and a vertical rectangle.

[0390] Applying different upsampling filters can help adapt to different characteristics of the content, for example, a filter is used in upsampling the feature map, and inputting information from the bitstream further includes obtaining information indicative of a filter shape and / or filter coefficients from the bitstream.

[0391] Correspondingly, the decoder can provide better reconstruction quality based on information from the encoder conveyed in the bitstream.

[0392] For example, the information indicating the filter shape may indicate a mask made up of flags, where a flag having a third value indicates a non-zero filter coefficient, and a flag having a fourth value different from the third value indicates a zero filter coefficient, where the mask represents the filter shape. This provides flexibility for designing filters of any shape.

[0393] For example, the multiple cascaded layers include convolutional layers with no upsampling between layers having different resolutions.

[0394] The provision of such additional layers in a cascaded layer network makes it possible to introduce additional processing, such as various types of filtering, to improve the quality or efficiency of the coding.

[0395] According to one embodiment, there is provided a computer program product stored on a non-transitory medium which, when executed on one or more processors, performs a method according to any of the methods described above.

[0396] According to one embodiment, there is provided a device for decoding images or videos, comprising a processing circuit configured to perform a method according to any of the above embodiments and examples.

[0397] According to one aspect, a device for decoding data for picture or video processing from a bitstream is provided, the device including: an acquisition unit configured to acquire two or more sets of feature map elements from the bitstream, each set of feature map elements relating to a feature map; an input unit configured to input each of the two or more sets of feature map elements respectively to two or more feature map processing layers of a plurality of cascaded layers; and a decoded data acquisition unit configured to acquire the decoded data for picture or video processing as a result of processing by the plurality of cascaded layers.

[0398] Any of the above mentioned devices can be embodied on an integrated chip. The present invention can be implemented in hardware (HW) and / or software (SW). Furthermore, HW-based implementations can be combined with SW-based implementations.

[0399] It should be noted that the present disclosure is not limited to any particular framework. Furthermore, the present disclosure is not limited to image or video compression, but may also be applied to object detection, image generation, and recognition systems.

[0400] Some exemplary implementations in hardware and software A corresponding system in which the above-mentioned encoder-decoder processing chain can be deployed is shown in FIG. 35. FIG. 35 is a schematic block diagram illustrating an example coding system, e.g., a video, image, audio, and / or other coding system (or short coding system), that may utilize the techniques of the present application. The video encoder 20 (or short encoder 20) and the video decoder 30 (or short decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform techniques according to various examples described in the present application. For example, the video coding and decoding may employ neural networks, or generally processing networks such as those described in the above embodiments and examples.

[0401] As shown in FIG. 35, coding system 10 includes a source device 12 configured to provide encoded picture data 21 to, for example, a destination device 14 for decoding encoded picture data 13.

[0402] The source device 12 includes an encoder 20 , and may further, i.e. optionally, include a picture source 16 , a pre-processor (or pre-processing unit) 18 , such as a picture pre-processor 18 , and a communication interface or unit 22 .

[0403] Picture source 16 may include or be any kind of picture capture device, e.g., a camera for capturing real-world pictures, and / or any kind of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any kind of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may be any kind of memory or storage that stores any of the aforementioned pictures.

[0404] As distinct from the pre-processor 18 and the processing performed by the pre-processing unit 18 , the picture or picture data 17 may also be referred to as the original picture or original picture data 17 .

[0405] The pre-processor 18 is configured to receive the (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component. It is noted that a neural network may be employed for the pre-processing.

[0406] Video encoder 20 is configured to receive pre-processed picture data 19 and to provide encoded picture data 21 .

[0407] The communications interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via the communications channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0408] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may further, i.e. optionally, include a communications interface or unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0409] The communications interface 28 of the destination device 14 is configured to receive the encoded picture data 21 (or any further processed version thereof), for example directly from the source device 12 or from any other source, for example a storage device, for example a coded picture data storage device, and to provide the encoded picture data 21 to a decoder 30.

[0410] Communications interface 22 and communications interface 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communications link between source device 12 and destination device 14, such as a direct wired or wireless connection, or via any type of network, such as a wired or wireless network or any combination thereof, or any type of private and public network, or any type of combination thereof.

[0411] The communications interface 22 may be configured, for example, to package the encoded picture data 21 in a suitable format, e.g., packets, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or network.

[0412] Communications interface 28, which forms a counterpart of communications interface 22, may for example be configured to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0413] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 relative to communication channel 13 in Figure 35, or as bidirectional communication interfaces, e.g., to send and receive messages, e.g., to set up a connection, acknowledge and exchange a communication link and / or any other information related to a data transmission, e.g., an encoded picture data transmission. Decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded pictures 31 (e.g., employing a neural network as described in the embodiments and examples above).

[0414] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also referred to as reconstructed picture data), e.g., the decoded picture 31, to obtain post-processed picture data 33, e.g., a post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, e.g., color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to, e.g., prepare the decoded picture data 31 for display by, e.g., a display device 34.

[0415] A display device 34 of destination device 14 is configured to receive the post-processed picture data 33, for example to display the picture to a user or viewer. The display device 34 may be or include any type of display for presenting the reconstructed picture, for example an integrated or external display or monitor. The display may have, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0416] 35 illustrates source device 12 and destination device 14 as separate devices, an embodiment of the devices may also include both source device 12 or corresponding functionality and destination device 14 or corresponding functionality, or both. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0417] As will be apparent to a person skilled in the art based on the description, the presence and (exact) division of functions or functions of different units within the source device 12 and / or destination device 14 as shown in FIG. 35 may vary depending on the actual device and application.

[0418] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30) or both the encoder 20 and the decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated to video coding, or any combination thereof. The encoder 20 may be implemented via processing circuitry 46 to embody various modules, including neural networks. The decoder 30 may be implemented via processing circuitry 46 to embody various modules as described in the above embodiments and examples. The processing circuitry may be configured to perform various operations, such as those described below. If the technology is implemented in part in software, the device may store instructions for the software in a suitable non-transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the technology of the present disclosure. Either the video encoder 20 and the video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG. 36.

[0419] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or fixed device, e.g., a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content delivery server), a broadcast receiver device, a broadcast transmitter device, and the like, and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0420] In some cases, the video coding system 10 shown in FIG. 35 is only an example, and the techniques of the present application may apply to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between encoding and decoding devices. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but simply encode data in memory and / or retrieve data from memory and decode it.

[0421] 37 is a schematic diagram of a video coding device 3700 according to one embodiment of the present disclosure. The video coding device 3700 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 3700 may be a decoder, such as the video decoder 30 of FIG. 35, or an encoder, such as the video encoder 20 of FIG. 35.

[0422] The video coding device 3700 includes an ingress port 3710 (or input port 3710) and a receiver unit (Rx) 3720 for receiving data, a processor, logic unit, or central processing unit (CPU) 3730 for processing the data, a transmitter unit (Tx) 3740 and an egress port 3750 (or output port 3750) for transmitting the data, and a memory 3760 for storing the data. The video coding device 3700 may also include optical-electrical (OE) and electrical-optical (EO) components coupled to the ingress port 3710, the receiver unit 3720, the transmitter unit 3740, and the egress port 3750 for egress or ingress of optical or electrical signals.

[0423] The processor 3730 is implemented by hardware and software. The processor 3730 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGA, ASIC, and DSP. The processor 3730 communicates with the ingress port 3710, the receiver unit 3720, the transmitter unit 3740, the egress port 3750, and the memory 3760. The processor 3730 includes a coding module 3770. The coding module 3770 implements the disclosed embodiments described above. For example, the coding module 3770 implements, processes, prepares, or provides various coding operations. Thus, the inclusion of the coding module 3770 provides a substantial improvement to the functionality of the video coding device 3700, resulting in the transformation of the video coding device 3700 into different states. Alternatively, the coding module 3770 is implemented as instructions stored in the memory 3760 and executed by the processor 3730.

[0424] The memory 3760 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device to store programs when such programs are selected for execution and to store instructions and data read during program execution. The memory 3760 may be, for example, volatile and / or non-volatile, and may be read only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0425] FIG. 38 is a simplified block diagram of an apparatus 3800 that may be used as either or both of the source device 12 and destination device 14 from FIG. 35, according to an exemplary embodiment.

[0426] The processor 3802 in the device 3800 may be a central processing unit. Alternatively, the processor 3802 may be any other type of device, or multiple devices, now existing or later developed, capable of manipulating or processing information. Although the disclosed implementations may be practiced with a single processor as shown, e.g., processor 3802, advantages in speed and efficiency may be achieved using two or more processors.

[0427] The memory 3804 in the apparatus 1100 may be a read-only memory (ROM) device or a random access memory (RAM) device in an implementation. Any other suitable type of storage device may be used as the memory 3804. The memory 3804 may include code and data 3806 that is accessed by the processor 3802 using a bus 3812. The memory 3804 may further include an operating system 3808 and application programs 3810, which include at least one program that enables the processor 3802 to perform the methods described herein. For example, the application programs 3810 may include applications 1-N, which further include a picture coding (encoding or decoding) application that performs the methods described herein.

[0428] The apparatus 3800 may also include one or more output devices, such as a display 3818. The display 3818, in one example, may be a touch-sensitive display that combines a display with a touch-sensitive element that is operable to sense touch input. The display 3818 may be coupled to the processor 3802 via the bus 3812.

[0429] Although shown here as a single bus, the bus 3812 of the device 3800 may be comprised of multiple buses. Additionally, the secondary storage may be directly coupled to other components of the device 3800 or may be accessed over a network, and may include a single integrated unit, such as a memory card, or multiple units, such as multiple memory cards. Thus, the device 3800 may be implemented in a wide variety of configurations.

[0430] In summary, the present disclosure relates to a method and apparatus for encoding data (for still or video processing into a bitstream). In particular, the data is processed by a network including multiple cascaded layers. In this process, a feature map is generated for each layer. The feature maps processed (output) by at least two different layers have different resolutions. In this process, a layer from the cascaded layers is selected that is different from the layer (e.g., latent space) that generates the lowest resolution feature map. The bitstream includes information related to the selected layer. In this approach, a scalable process is provided that can operate at different resolutions, so that the bitstream can convey information about such different resolutions. Thus, data can be efficiently coded in the bitstream according to a resolution that can vary according to the content of the coded picture data.

[0431] The present disclosure further relates to a method and apparatus for decoding data (for still or video processing into a bitstream). In particular, two or more sets of feature map elements are obtained from a bitstream. Each set of feature map elements is associated with a feature map. Each of the two or more sets of feature map elements is then input to two or more feature map processing layers of a plurality of cascaded layers, respectively. Then, decoded data for picture or video processing is obtained as a result of processing by the plurality of cascaded layers. Thus, data can be decoded from the bitstream in an efficient manner in a hierarchical structure.

[0432] The present disclosure further relates to a method and apparatus for decoding data (for still or video processing into a bitstream). Two or more sets of segmentation information elements are obtained from the bitstream. Then, each of the two or more sets of segmentation information elements is input to two or more segmentation information processing layers of a plurality of cascaded layers, respectively. In each of the two or more segmentation information processing layers, a respective set of segmentation information is processed. Decoded data for picture or video processing is obtained based on the segmentation information processed by the plurality of cascaded layers. Thus, data can be decoded from the bitstream in an efficient manner in a hierarchical structure. [Explanation of symbols]

[0433] 10. Video Coding System 12 Source Device 13 Encoded Picture Data 13 Communication Channels 14 Destination Device 16 Picture Source 17 Pictures 18 Preprocessor / Preprocessing Unit 19 Pictures 20 Video Encoder 21 Encoded Picture Data 21 Bitstream 22 Communication interface or communication unit 28 Communication Interface or Communication Unit 30 Video Decoder 31 Decoded Pictures 32 Post-processor / Post-processing unit 33 Post-processing picture data 34 Display Devices 46 Processing Circuit 101 Encoder 102 Quantizer 103 HyperEncoder / Decoder 104 Decoder 105 Arithmetic Encoder 106 Arithmetic Decoder 107 Modules 108 Modules 109 Modules 110 Module 121 Encoder 122 Quantizer 125 Arithmetic Coding Module 123 HyperEncoder 147 Hyper Decoder 201 Input Interface 203 Block 204 Residual Calculation Unit 206 Conversion Processing Unit 208 Quantization Units 210 Inverse Quantization Unit 212 Inverse Transformation Processing Unit 214 Reconstruction Unit 220 Loop Filter Unit 230 Decoded Picture Buffer (DPB) 244 Inter Prediction Units 254 prediction units 260 Mode Selection Unit 262 Split Unit 270 Entropy Coding Unit 272 Output Interface 304 Entropy Decoding Unit 309 Quantization Coefficients 310 Inverse Quantization Unit 312 Inverse Transformation Processing Unit 313 Residual Block 314 Reconstruction Unit 315 Reconstructed Blocks 320 Loop Filtering Unit 322 Quantizer 331 Decoded Picture 360 mode application unit 365 predicted blocks 401 Downsampling Layer 402 Downsampling Layer 403 Downsampling Layer 404 Downsampling Layer 405 Downsampling Layer 406 Downsampling Layer 413 Output Signal 414 Input images 415 Components 407 Upsampling Layer 408 upsampling layers 409 Upsampling Layer 410 Upsampling Layer 411 Upsampling Layer 412 upsampling layers 420 More Layers 430 Convolutional Layer 610 Dense Optical Flow 611 LayerMv Tensor 613 Cost Accounting Units 614 Layer Information Selection Unit 621 LayerMV Tensor 622 results 623 Cost Accounting Units 624 Layer Information Selection Unit 625 Pooling Operations 631 LayerMv 633 Cost Accounting Units 634 Layer Information Selection Unit 635 MinCost Pooling 705 Indicator 710 Layer 1 Cost Calculation Units 720 Minimum Cost Pooling 730 Layer Information Selection Module 810 Processing 820 Selection 830 Generation Step 900 Network 901 Input 901 Video Processing 911 Cascade Layer 912 Cascade Layer 913 Cascade Layer 920 Signal Selection Logic 930 Bitstream 940 Signal Supply Logic 951 upsampling layers 952 upsampling layers 953 upsampling layers 1010 Array 1020 Array 1050 Feature Maps 1060 Feature Maps 1100 Signal selection circuit (logic) 1120 elements 1110 Dense Optical Flow 1110 Feature Map 1120 Information 1130 Segmentation Information 1140 Motion Segmentation Net 1150 Bitstream 1210 Optical flow estimation module (unit) 1215 Optical Flow 1220 Motion Specification (or Segmentation) Module 1250 bitstream 1260 Movement Information 1270 Motion Generation Unit 1275 Motion Vector Field 1280 Motion Compensation Unit 1310 Network 1320 Signal Selection Logic 1350 Bitstream 1360 Movement Generation Unit 1400 cost calculation (or estimated) units 1405 Reference Picture 1408 Target Picture 1415 Motion Field Upsampling 1420 Motion Compensation 1430 Distortion 1440 Rate Estimation Module 1450 bit normalization 1460 Cost Calculation Module 1470 Downsampling 1480 Cost Tensor 1500 Cost Estimate Units 1501 Downsampling 1505 Reference Picture 1510 Motion Compensation 1515 Downsampling 1518 Downsampling 1530 Distortion Evaluation 1540 Estimated Rate 1600 Signal Selection Unit 1700 Signal Selection Logic 1810 Motion Segmentation Unit 1860 Movement Generation Network (Module) 1869 Movement Generation Unit 1910 Motion Segmentation Network (Module) 1960 Movement Generation Unit 2010 Motion Segmentation Unit 2060 Movement Generation Network (Unit) 2201 Block Size 2202 Block Size 2203 Block Size 2211 Cost Accounting Unit 2212 Cost Accounting Unit 2213 Cost Accounting Unit 2222 Minimum Cost Pooling Operation 2223 MinCost Pooling 2231 Layer Information Selection Unit 2232 Layer Information Selection Unit 2233 Layer Information Selection Unit 230x block size 2300 cost calculation units 2310 units 2320 Reconstruction Unit 2330 Distortion Calculation Unit 2340 Downsampling 2360 bit estimation units 2410 Cost Accounting Unit 2420 Minimum Cost Pooling 2430 Layer Information Selection Unit 2510 Quadtree splitting 2520 Binary tree splitting 2530 Asymmetric Binary Tree Partitioning 2540 ternary tree splitting 2610 Feature Map Elements 2611 Flag 2620 Feature Map 2621 Flag 2622 Flag 2623 Feature Map Elements 2624 Flag 2630 Feature Map 2800 Signal Supply Logic 2801 Upsampling 2811 Tensor Binding Logic 2812 Tensor Binding Logic 2813 Tensor Binding Logic 2821 Syntax Interpretation Unit 2822 Syntax Interpretation Unit 2823 Syntax Interpretation Unit 2900 Signal Supply Logic 2901 Consolidated 2902 Consolidated 2910 Initialization Unit 2911 Tensor Connection Block 2912 Tensor Connection Block 2913 Tensor Connection Block 2918 Initialization Unit 2919 Initialization Unit 2921 Syntax Interpretation Block 2922 Syntax Interpretation Block 2923 Syntax Interpretation Block 2990 Optical Flow Reconstruction 3000 Convolution Filter Units 3030 Bitstream 3040 Signal Supply Logic 3100 Upsampling Filter Unit 3700 Video Coding Device 3710 Inlet Port 3720 Receiver Unit (Rx) 3730 Central Processing Unit (CPU) 3740 Transmitter Unit (Tx) 3750 Exit Port 3760 Memory 3770 Coding Module 3800 equipment 3802 processor 3804 Memory 3806 Data 3808 Operating System 3810 Application Program 3812 Bus 3818 Display

Claims

1. 1. A method for decoding data for picture or video processing from a bitstream, the method comprising: obtaining two or more sets of segmentation information elements from the bitstream; inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers of a plurality of cascaded layers, respectively; and processing a respective set of segmentation information in each of the two or more segmentation information processing layers; The method, wherein obtaining decoded data for picture or video processing is based on the segmentation information processed by the multiple cascade layers.

2. The method of claim 1 , wherein the step of obtaining the set of segmentation information elements is based on segmentation information processed by at least one segmentation information processing layer of the plurality of cascaded layers.

3. The method of claim 1 or 2, wherein the step of inputting the set of segmentation information elements is based on the processed segmentation information output by at least one of the plurality of cascaded layers.

4. The method according to claim 1 , wherein the segmentation information processed in each of the two or more segmentation information processing layers differs in resolution.

5. The method of claim 4 , wherein the step of processing the segmentation information in the two or more segmentation information processing layers comprises a step of upsampling.

6. The method of claim 5 , wherein the step of upsampling the segmentation information comprises nearest neighbor upsampling.

7. The method of claim 5 or 6, wherein the step of upsampling the segmentation information comprises a transposed convolution.

8. For each segmentation information processing layer j of the plurality of N segmentation information processing layers among the plurality of cascade layers, - the step of inputting comprises the steps of inputting initial segmentation information from the bitstream if j=1, and otherwise inputting segmentation information processed by the (j-1)th segmentation information processing layer; - outputting the processed segmentation information.

9. The step of processing the input segmentation information by each layer j<N of the plurality of N segmentation information processing layers includes:

9. The method of claim 8, further comprising the steps of parsing a segmentation information element from the bitstream and associating the parsed segmentation information element with the segmentation information output by a previous layer, wherein a position of the parsed segmentation information element within the associated segmentation information is determined based on the segmentation information output by the previous layer.

10. The method of claim 9 , wherein an amount of segmentation information elements parsed from the bitstream is determined based on segmentation information output by the previous layer.

11. The method of claim 9 or 10, wherein the parsed segmentation information elements are represented by a set of binary flags.

12. The step of obtaining decoded data for picture or video processing includes, based on the segmentation information, - intra- or inter-picture prediction mode, - picture reference index, - single reference or multiple reference prediction (including bi-prediction); - presence or absence prediction residual information, - quantization step size, - Motion information prediction type, - the length of the motion vector, - motion vector resolution, - motion vector prediction index, - motion vector difference size, - motion vector differential resolution, - motion interpolation filters, - in-loop filter parameters, - post-filter parameters, The method according to claim 1 , further comprising determining at least one of:

13. obtaining a set of feature map elements from the bitstream, and inputting the set of feature map elements into a feature map processing layer among a plurality of layers based on the segmentation information processed by a segmentation information processing layer; The method of claim 1 , further comprising: obtaining the decoded data for picture or video processing based on the feature maps processed by the multiple cascade layers.

14. The method of claim 13 , wherein at least one of the plurality of cascaded layers is a segmentation information processing layer and a feature map processing layer.

15. The method of claim 13 , wherein each layer of the plurality of layers is either a segmentation information processing layer or a feature map processing layer.

16. A computer program product stored on a non-transitory medium, which when executed on one or more processors performs the method of any one of claims 1 to 15.

17. A device for decoding images or videos comprising a processing circuit configured to perform the method of any one of claims 1 to 15.

18. 1. A device for decoding data for picture or video processing from a bitstream, the device comprising: an acquisition unit configured to acquire two or more sets of segmentation information elements from the bitstream; an input unit configured to input each of the two or more sets of segmentation information elements to two or more segmentation information processing layers of a plurality of cascaded layers, respectively; a processing unit configured to process the respective set of segmentation information in each of the two or more segmentation information processing layers; a decoding data obtaining unit configured to obtain decoding data for picture or video processing based on the segmentation information processed in the multiple cascade layers.

Images