Signaling decoding using partition information

Abstract

The present application also relates to a method and apparatus for decoding data (for still image or video processing) from a bitstream. Two or more sets of partition information elements are obtained from the bitstream. Then, each of the two or more sets of partition information elements is input into two or more partition information processing layers in a plurality of cascaded layers, respectively. In each of the two or more partition information processing layers, the respective set of partition information is processed. Decoded data for image or video processing is obtained according to the partition information processed by the plurality of cascaded layers. Thus, the data can be decoded from the bitstream in an efficient manner in a hierarchical structure.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application generally relate to the field of decoding data for image or video processing from a bitstream using multiple processing layers. In particular, some embodiments relate to methods and apparatuses for such decoding. BACKGROUND

[0002] For decades, hybrid image and video codecs have been used to compress image and video data. In such codecs, the signal is typically encoded block by block by predicting a block and by coding the difference between the original block and its prediction. In particular, such coding can include a transform, quantization, and generating a bitstream, typically including some entropy coding. Typically, the three components of the hybrid coding method - transform, quantization, and entropy coding - are optimized separately. Modern video compression standards, such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC), also use transform representation to code the residual signal after prediction.

[0003] Recently, machine learning has been applied to image and video coding. Generally, machine learning can be applied to image and video coding in various different ways. For example, some end-to-end optimized image or video coding schemes have been discussed. In addition, machine learning has been used to determine or optimize certain parts of the end-to-end coding, such as the selection of prediction parameters or compression, etc. A commonality of these applications is that some feature map data is produced, which will be transmitted between the encoder and the decoder. An efficient structure of the bitstream can greatly help to reduce the number of bits to encode the image / video source signal.

[0004] Neural networks typically include two or more layers. Feature maps are the output of the layers. In a neural network divided between devices (e.g., between an encoder and a decoder, between a device and a cloud, or between different devices), the feature maps at the output of the division location (e.g., the first device) are compressed and transmitted to the remaining layers of the neural network (e.g., to the second device).

[0005] There can be a need to further improve encoding and decoding using trained network architectures. SUMMARY

[0006] Some embodiments of the present application provide methods and apparatuses for decoding images in an efficient manner and adapting some scalability to the required parameters and content.

[0007] The above and other objects are achieved by the subject-matter claimed in the independent claims. Further implementation forms are evident from the dependent claims, the description and the attached drawings.

[0008] According to an aspect, there is provided a method for decoding data for image or video processing from a bitstream, wherein the method comprises: obtaining two or more sets of partition information elements from the bitstream; inputting each of the two or more sets of partition information elements into two or more partition information processing layers of a plurality of cascaded layers, respectively; processing the respective set of partition information in each of the two or more partition information processing layers, wherein the decoded data for image or video processing is obtained according to the partition information processed in the plurality of cascaded layers.

[0009] This method can improve efficiency as it can decode data in various partitions that can be configured based on layers in a hierarchical structure. Providing partitions can take into account characteristics of the decoded data.

[0010] For example, the obtaining the set of partition information elements is according to partition information processed by at least one of the partition information processing layers of the plurality of cascaded layers.

[0011] In some example embodiments, the inputting the set of partition information elements is according to processed partition information output by at least one of the cascaded layers of the plurality of cascaded layers.

[0012] Cascaded partition information processing can efficiently parse the partition information.

[0013] For example, resolutions of the partition information processed in the two or more partition information processing layers, respectively, are different.

[0014] In some embodiments and examples, the processing of the partition information in the two or more partition information processing layers comprises upsampling.

[0015] The hierarchical structure of partition information can provide a small amount of side information to be inserted into the bitstream, thereby improving efficiency and / or shortening processing time.

[0016] In particular, the upsampling of the partition information comprises nearest-neighbor upsampling. Nearest-neighbor upsampling has a low computational complexity and can be easily implemented. Nevertheless, nearest-neighbor upsampling is still very efficient, especially for logical indications such as flags. For example, the upsampling of the partition information comprises transposed convolution. Performing upsampling can improve the upsampling quality. Furthermore, such a convolutional upsampling layer can be provided trainable or configurable at the decoding side, such that the convolutional kernel can be controlled by an indication parsed from the bitstream or otherwise derived.

[0017] In an example implementation, for each partition information processing layer j of the plurality N of partition information processing layers in the plurality of cascaded layers, the input comprises: if j = 1, the initial partition information is input from the bitstream, otherwise, the partition information processed by the (j - 1)th partition information processing layer is input; and the output comprises the processed partition information.

[0018] For example, the processing of each j < N of the plurality N of partition information processing layers on the input partition information further comprises: parsing partition information elements from the bitstream, and associating the parsed partition information elements with the partition information output by the previous layer, wherein the position of the parsed partition information elements in the associated partition information is determined according to the partition information output by the previous layer. In particular, the number of partition information elements parsed from the bitstream is determined according to the partition information output by the previous layer. For example, the parsed partition information elements are represented by a set of binarized flags.

[0019] Such a hierarchical structure provides processing that can be performed in parallel, and can be easily run on GPU / NPU and can take advantage of parallelism. A fully trainable scheme that transmits gradients can use it in an end-to-end trainable video coding technology scheme.

[0020] In some example embodiments and examples, obtaining decoded data for image or video processing comprises determining at least one of the following according to the partition information: intra or inter prediction mode; picture reference index; single or multi-reference prediction (including bi-prediction); presence or absence of prediction residual information; quantization step size; motion information prediction type; motion vector length; motion vector resolution; motion vector prediction index; motion vector difference size; motion vector difference resolution; motion interpolation filter; in-loop filter parameter; post-filter parameter. The decoding of the present disclosure is generally applicable to any type of data related to image or video coding.

[0021] The method provided by the above embodiments or examples can further include: obtaining a feature map element set from the bitstream, and inputting the feature map element set into a feature map processing layer in the plurality of cascaded layers according to segmentation information processed by the segmentation information processing layer; and obtaining decoded data for image or video processing according to the feature map processed by the plurality of cascaded layers.

[0022] In particular, at least one of the plurality of cascaded layers is a segmentation information processing layer and a feature map processing layer. In other embodiments, each layer in the plurality of layers is a segmentation information processing layer or a feature map processing layer.

[0023] The separated layer functions provide a concise design and functional separation. However, the present application can also work if a layer implements both functions.

[0024] According to an aspect, there is provided a computer program product stored in a non-transitory medium, which when executed on one or more processors, performs the method according to any of the above examples and embodiments.

[0025] According to an aspect, there is provided a device for decoding an image or video, comprising processing circuitry configured to perform the method according to any of the above examples and embodiments.

[0026] According to an aspect, there is provided a device for decoding data for image or video processing from a bitstream, wherein the device comprises: an obtaining unit configured to obtain two or more sets of segmentation information elements from the bitstream; an input unit configured to input each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers; a processing unit configured to process each set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data obtaining unit configured to obtain the decoded data for image or video processing according to the segmentation information processed in the plurality of cascaded layers.

[0027] Any of the above apparatuses can be implemented on an integrated chip. The present application can be implemented in hardware (HW) and / or software (SW). Furthermore, a hardware-based implementation can be combined with a software-based implementation.

[0028] It should be noted that the present application is not limited to a specific framework. Furthermore, the present application is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0029] For the sake of clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create a new embodiment within the scope of the present application. Attached Figure Description

[0030] The embodiments of the present invention are described in detail below with reference to the accompanying drawings, wherein:

[0031] Figure 1 A schematic diagram of a channel processed by layers of a neural network;

[0032] Figure 2 A schematic diagram of an autoencoder type for neural networks;

[0033] Figure 3A This is a schematic diagram of an exemplary network architecture that includes an encoder and a decoder with a priori model.

[0034] Figure 3B A schematic diagram of a general network architecture including a priori model at the encoding end;

[0035] Figure 3C This is a schematic diagram of a general network architecture for the decoding end, including a priori model.

[0036] Figure 4 A schematic diagram of an exemplary network architecture including a priori models for both the encoder and decoder ends;

[0037] Figure 5A This is a block diagram of an end-to-end video compression framework based on neural networks.

[0038] Figure 5B A block diagram illustrating some exemplary details of neural network applications for sports field compression;

[0039] Figure 5C A block diagram showing some exemplary details of neural network applications for motion compensation;

[0040] Figure 6 A schematic diagram of the layers of U-net;

[0041] Figure 7A A block diagram of an exemplary hybrid encoder;

[0042] Figure 7B A block diagram of an exemplary hybrid decoder;

[0043] Figure 8 A flowchart of an exemplary method for encoding data for image / video processing (e.g., encoding);

[0044] Figure 9 A block diagram of a network structure that transmits information from layers of different resolutions in a bitstream;

[0045] Figure 10A This is a schematic diagram of max pooling;

[0046] Figure 10B is a schematic diagram for average pooling;

[0047] Figure 11 is a schematic diagram for processing of an exemplary encoder side pair of feature maps and segmentation information;

[0048] Figure 12 is a block diagram of a generalized processing of motion information feature maps at the encoder and decoder side;

[0049] Figure 13 is a block diagram of a network structure for processing of motion vector related information from layers of different resolution transported in the bitstream;

[0050] Figure 14 is a block diagram of an exemplary cost calculation unit with higher cost tensor resolution;

[0051] Figure 15 is a block diagram of an exemplary cost calculation unit with lower cost tensor resolution;

[0052] Figure 16 is a block diagram illustrating the functional structure of the signal selection logic;

[0053] Figure 17 is a block diagram illustrating the functional structure of the signal selection logic with one or more cost calculation units providing several coding options;

[0054] Figure 18 is a block diagram of a network structure with convolutional down- and up- sampling layers from layers of different resolution transported in the bitstream;

[0055] Figure 19 is a block diagram of a structure with additional layers from layers of different resolution transported in the bitstream;

[0056] Figure 20 is a block diagram of a structure with layers capable of down- or up- sampling filter selection from layers of different resolution transported in the bitstream;

[0057] Figure 21 is a block diagram of a network structure capable of convolutional filter selection from layers of different resolution transported in the bitstream;

[0058] Figure 22 is a block diagram illustrating the functional structure of a network based RDO decision unit for selecting a coding mode;

[0059] Figure 23 is a block diagram illustrating an exemplary cost calculation unit that can be used in a network based RDO decision unit for selecting a coding mode;

[0060] Figure 24Block diagram of an exemplary cost computation unit that can be used in a network-based RDO decision unit for selecting a coding mode that supports multiple options;

[0061] Figure 25 Diagram of possible block partitions or filter shapes;

[0062] Figure 26 Diagram of derivation of partition information;

[0063] Figure 27 Diagram of processing of partition information by the decoding side;

[0064] Figure 28 Block diagram of exemplary signal feeding logic for reconstructing dense optical flow;

[0065] Figure 29 Block diagram of exemplary signal feeding logic for reconstructing dense optical flow;

[0066] Figure 30 Block diagram of a convolutional filter bank;

[0067] Figure 31 Block diagram of an up-sampling filter bank;

[0068] Figure 32A Diagram of up-sampling processing using nearest-neighbor replication at the decoding side;

[0069] Figure 32B Diagram of up-sampling processing using convolutional processing at the decoding side;

[0070] Figure 33 Flowchart of an exemplary method for decoding data such as feature map information used when decoding an image or video;

[0071] Figure 34 Flowchart of an exemplary method for encoding data such as partition information used when encoding an image or video;

[0072] Figure 35 Block diagram of an example of a video coding system for implementing embodiments of the present invention;

[0073] Figure 36 Block diagram of another example of a video coding system for implementing embodiments of the present invention;

[0074] Figure 37 Block diagram of an example of an encoding or decoding device;

[0075] Figure 38 Block diagram of another example of an encoding or decoding device. DETAILED DESCRIPTION

[0076] In the following description, reference is made to the accompanying drawings which form a part hereof, and which are shown by way of illustration of specific aspects or can be used to use the specific aspects of the application. It is understood that embodiments of the application can be used in other aspects and encompass modifications not depicted in the drawings. Therefore, the following detailed description is not to be taken in a limiting sense, as the scope of the application is defined by the appended claims.

[0077] It is understood that the disclosure in connection with described methods also holds true for corresponding devices or systems for performing the methods, and vice versa. For example, if one or a plurality of specific method steps are described, a corresponding device can include one or a plurality of units, e.g. functional units, to perform the described one or plurality of method steps, even if such one or more units are not explicitly described or illustrated in figures. On the other hand, for example, if a specific apparatus is described based on one or a plurality of units, e.g. functional units, a corresponding method can include one step to perform the functionality of the one or plurality of units, even if such one or more steps are not explicitly described or illustrated in figures. Further, it is understood that the features of the various exemplary embodiments and / or aspects described herein can be combined with each other, unless specifically noted otherwise.

[0078] It is an aim of some embodiments to improve the quality of encoding and decoding image or video data and / or to reduce the amount of data required to represent encoded image or video data. Some embodiments provide methods for efficiently selecting information that an encoder indicates to a decoder. In the following, a technical overview of some of the technical terms and frameworks that can be employed by some uses of embodiments of the application is provided.

[0079] Artificial neural networks

[0080] An artificial neural network (ANN) or connectionist system is a computational system that is loosely inspired by biological neural networks that constitute animal brains. These systems "learn" to perform tasks by example, typically without using task-specific rules for programming. For example, in image recognition, these systems can learn to identify images containing cats, i.e. by analyzing example images that are manually labeled as "cat" or "no cat" and using the results to identify cats in other images. These systems do not know beforehand that cats have fur, tails, whiskers, and cat faces, etc. Instead, they develop recognition features from the examples they are trained on.

[0081] ANNs are based on a set of connected units or nodes called artificial neurons, which model neurons in the biological brain in a loose way. Each connection, like a synapse in the biological brain, can pass a signal to other neurons. An artificial neuron receives signals, then processes this signal, and can send a signal to neurons connected to it.

[0082] In an ANN implementation, the "signal" at a connection is a real number, and the output of each neuron is computed by some non-linear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as learning proceeds. The weights increase or decrease the strength of the signal at a connection. Neurons can have a threshold that must be passed before the signal is sent to the next layer. Typically, neurons are grouped into layers that process the input data. Different layers can perform different transformations on their inputs. The signal can travel from the first layer (input layer) to the last layer (output layer) after multiple passes through the layers.

[0083] The original goal of ANN methods was to solve problems in the same way that a human brain would. Over time, attention shifted to getting a machine to perform specific tasks, leading to a divergence from biology. ANNs have been used in a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even in activities traditionally considered to be the province of humans, such as painting.

[0084] The name "convolutional neural network" (CNN) indicates that the network employs a mathematical operation called convolution. Convolution is a specialized linear operation. Convolutional networks are neural networks that use convolutions in at least one of their layers instead of the general matrix multiplication used by other layers.

[0085] Figure 1 The general concept of processing by a neural network (e.g., a CNN) is illustrated schematically. A convolutional neural network is composed of an input layer and an output layer, as well as multiple hidden layers. The input layer is the layer that provides the input (e.g., a portion of an image) to be processed. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve with a multiplication or other dot product. The result of a layer is one or more feature maps (f.maps) sometimes also called channels. Subsampling can be involved in some or all of the layers. Thus, the feature maps can get smaller, as Figure 1 Figure 1 Figure 1 ​​The activation function in CNNs is usually a rectified linear unit (ReLU) layer, followed by additional convolutions, such as pooling layers, fully connected layers, and normalization layers, called hidden layers because their inputs and outputs are masked by activation functions and final convolutions. While these layers are colloquially called convolutions, this is just by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This has important implications for the indexing in the matrices, as it affects the way the weights are determined at a particular index point.

[0086] When a CNN is programmed to process images, as shown in Figure 1 The input is a tensor of shape (number of images) x (image width) x (image height) x (image depth). Then, after passing through the convolutional layers, the images are abstracted into feature maps, with shape (number of images) x (feature map width) x (feature map height) x (number of feature map channels). A convolutional layer in a neural network should have the following properties. A convolution kernel defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature maps.

[0087] In the past, traditional multilayer perceptron (MLP) models were used for image recognition. However, due to the full connectivity between nodes, they were affected by high dimensionality and could not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed at scale with full connectivity. Furthermore, this network architecture does not take into account the spatial structure of the data, treating input pixels that are far apart the same way as those that are close. This ignores the reference locality in image data, both computationally and semantically. Therefore, the full connectivity of neurons is wasteful for the purpose of image recognition and other goals dominated by spatially local input patterns.

[0088] Convolutional neural networks are biologically inspired variants of multilayer perceptrons, specifically designed to mimic the behavior of the visual cortex. These models alleviate the challenges posed by the MLP architecture by exploiting the strong spatial locality correlations present in natural images. The convolutional layer is the core building block of CNNs. The parameters of this layer consist of a set of learnable filters (the aforementioned kernels) with a small receptive field, but with a large stride covering a large portion of the input volume. During forward propagation, each filter is convolved across the width and height of the input volume, computing the dot product between the filter entries and the input at each position and producing a two-dimensional activation map for that filter. Thus, the network learns filters that activate when certain types of features are present in the input at a certain spatial position.

[0089] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Thus, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a specified filter. Feature map and activation have the same meaning. In some papers, the feature map is called activation map because it is a map of activations corresponding to different parts of the image, and also a feature map because it is also a map of finding certain features in the image. High activation means finding a certain function.

[0090] Another important concept of CNNs is pooling, a form of non-linear down-sampling. There are several non-linear functions to implement pooling, with max-pooling being the most common. The input image is divided into a set of non-overlapping rectangles, and the maximum value is output for each such sub-region.

[0091] Intuitively, the exact position of a feature is less important than its coarse position relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to progressively reduce the spatial size of the representations, to reduce the number of parameters in the network, memory and computational load, and also to control overfitting. It is common to insert a pooling layer after a convolutional layer in a CNN architecture. Pooling operations provide another form of transformation invariance.

[0092] Pooling layers operate independently on each depth slice of the input and rescale the size spatially. The most common form is the pooling layer, which applies a filter of size 2x2 with a stride of 2 in each depth slice of the input, discarding 75% of the activations along both width and height by 2. In this case, each max operation exceeds 4 numbers. The depth dimension remains unchanged. In addition to max-pooling, other functions can be used by the pooling unit, such as average-pooling or l2-norm pooling. Average-pooling was often used in the past, but has been used less frequently recently compared to max-pooling, which in fact often works better. Due to the large reduction in representation size, there has been a recent trend to use smaller filters or to completely discard the pooling layer. Region of Interest (RoI) pooling, also known as ROI pooling, is a variant of max-pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of the convolutional neural network for object detection based on the fast R-CNN architecture.

[0093] ReLU is short for rectified linear unit, which applies a non-saturating activation function. It effectively removes negative values from the activation map by setting them to 0. It increases the non-linear nature of the decision function and the overall network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the hyperbolic tangent and sigmoid functions. ReLU is generally preferred over the others because it trains neural networks several times faster without much impact on the generalization accuracy.

[0094] After several convolutional and max-pooling layers, high-level reasoning in a neural network is accomplished by fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as in regular (non-convolutional) artificial neural networks. Thus, these activations can be computed as an affine transformation, a matrix multiplication followed by a bias offset (vector addition of a learned or fixed bias term).

[0095] A "loss layer" (including the computation of a loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, typically the last layer of a neural network. Various loss functions can be used that are appropriate for different tasks. The softmax loss is used for predicting a single class out of K mutually exclusive classes. The sigmoid cross-entropy loss is used for predicting K independent probability values in [0, 1]. The Euclidean loss is used for regression to real-valued labels.

[0096] In summary, Figure 1 Data flow in a typical convolutional neural network is illustrated. First, an input image passes through a convolutional layer and is abstracted to a feature map that includes several channels, corresponding to multiple filters in a set of learnable filters for that layer. The feature map is then subsampled using a pooling layer or the like, which reduces the dimensionality of each channel in the feature map. Data then reaches another convolutional layer, which can have a different number of output channels. As mentioned above, the number of input and output channels is a hyperparameter of the layer. To establish connections for the network, these parameters need to be synchronized between two connected layers, such that the number of input channels for the current layer should equal the number of output channels of the previous layer. For the first layer that processes input data such as an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation.

[0097] Autoencoders and unsupervised learning

[0098] An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic is shown in Figure 2The goal of an autoencoder is to learn a representation (encoding) of a set of data by training a network to ignore signal "noise", typically for dimensionality reduction. Along with the simplification side, the reconstruction side is learned, where the autoencoder attempts to generate a representation as close as possible to its original input from the simplified encoding, hence the name. In the simplest case, given a hidden layer, the encoder stage of an autoencoder takes an input x and maps it to h

[0099] h = σ(Wx + b).

[0100] This image h is often called the code, latent variable, or latent representation. Here, σ is an element-wise activation function, e.g., sigmoid function or rectified linear unit. W is a weight matrix and b is a bias vector. The weights and biases are typically initialized randomly and then updated iteratively during training by backpropagation. After that, the decoder stage of the autoencoder maps h to a reconstruction x' of the same shape as x:

[0101] x' = σ'(W' h' + b')

[0102] where the decoder σ', W' and b' can be independent of the corresponding σ, W and b of the encoder.

[0103] Variational autoencoder models make strong assumptions about the distribution of the latent variable. These models use variational methods for latent representation learning, resulting in an additional loss component and a specific estimator for the training algorithm, called the Stochastic Gradient Variational Bayes (SGVB) estimator. Assume that the data is generated by a directed graphical model p θ (x|h) and the encoder is learning an approximation q θ (h|x) of the posterior p φ (h|x), where φ and θ represent the parameters of the encoder (recognition model) and the decoder (generative model), respectively. The probability distribution of the latent vector of a VAE is typically closer to matching the probability distribution of the training data than a standard autoencoder. The goal of a VAE has the following form:

[0104]

[0105] Here, D KL represents the KL divergence. The prior of the latent variable is often set to a centered isotropic multivariate Gaussian Typically, the shapes of the variational and likelihood distributions are chosen such that they are factored Gaussians:

[0106]

[0107]

[0108] where p(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0109] Recent advances in the field of artificial neural networks, in particular convolutional neural networks, have made researchers interested in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression has been proposed, using networks based on variational autoencoders.

[0110] Data compression is therefore considered a fundamental and well-studied problem in engineering, typically in order to design a code with minimal entropy for a given discrete dataset. This technical solution relies heavily on the knowledge of the probability structure of the data, and therefore the problem is closely related to the modeling of probabilistic sources. However, since all practical codes must have finite entropy, continuous-valued data (for example, vectors of image pixel intensities) must be quantized to a finite set of discrete values, which introduces errors.

[0111] In this case, the problem of lossy compression, two contradictory costs must be balanced: the entropy of the discrete representation (rate) and the error caused by quantization (distortion). Different compression applications, such as data storage or transmission over a channel of limited capacity, require different balances between rate and distortion.

[0112] The joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable. Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using a lossless entropy code. Due to the central role of the transformation, this scheme is called transform coding.

[0113] For example, JPEG uses a discrete cosine transform on blocks of pixels, JPEG 2000 uses a multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method (transform, quantization, and entropy coding) are optimized separately (usually by hand-tuning parameters). Modern video compression standards like HEVC, VVC, and EVC also use transform representation for coding the residual signal after prediction. Several transforms are used for this purpose, like the discrete cosine transform (DCT) and the discrete sine transform (DST), and a low frequency non-separable manually optimized transform (LFNST).

[0114] Variational image compression

[0115] The Variable Auto-Encoder (VAE) framework can be considered as a non-linear transform coding model. The transform process can be mainly divided into four parts. This is exemplified in Figure 3A Fig. 1.

[0116] The transform process can be mainly divided into four parts: Figure 3A The VAE framework is exemplified in Figure 3A Fig. 1. The encoder 101 maps the input image x into a latent representation (denoted by y) by the function y = f(x). In the following, this latent representation can also be referred to as a part of the “latent space” or a point therein. The function f() is a transform function that converts the input signal x into a representation y that can be further compressed. The quantizer 102 transforms the latent representation y into a quantized latent representation where the (discrete) values are Q denotes the quantizer function. The entropy model, or hyper-encoder / decoder (also referred to as hyper-prior) 103 estimates the distribution of the quantized latent representation to obtain the minimum rate achievable by lossless entropy source coding.

[0117] The latent space can be understood as a representation of compressed data, where similar data points are closer in the latent space. The latent space is very useful for learning data characteristics and finding a simpler representation of data for analysis. The quantized latent representation and the side information of the hyper-prior 3 are included in the bitstream 2 (binarized). Furthermore, a decoder 104 is provided that transforms the quantized latent representation into a reconstructed image signal is an estimate of the input image x. It is desired that x be as close as possible to i.e. the reconstruction quality is as high as possible. However, The higher the similarity between x and x is, the larger the amount of side information needed for the transmission is. Side information includes Figure 3A bitstream 1 and bitstream 2 shown in Fig. 1, which are generated by the encoder and transmitted to the decoder. Generally, the larger the amount of side information is, the higher the reconstruction quality is. However, a large amount of side information means a low compression ratio. Therefore, Figure 3A One purpose of the system described in Fig. 1 is to balance the reconstruction quality and the amount of side information transmitted in the bitstream.

[0118] In Figure 3A component AE 105 is an arithmetic encoding module that converts quantized latent representations and side information into a binarized representation bitstream 1. For example, and Examples of x and x can include integers or floating point numbers. One purpose of the arithmetic encoding module is to convert the sample values into a binarized string of numbers (then, the binarized numbers are included in the bitstream, which can include other parts corresponding to the encoded image or other side information) by a binarization process.

[0119] Arithmetic decoding (AD) 106 is a process that reverses the binarization process, where the binarized numbers are converted back to sample values. The arithmetic decoding is provided by an arithmetic decoding module 106.

[0120] It is noted that the present invention is not limited to this particular framework. Furthermore, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0121] In Figure 3A there are two subnets that are cascaded with each other. In this context, a subnet is a logical division between parts of the whole network. For example, in Figure 3A the modules 101, 102, 104, 105 and 106 are referred to as the “encoder / decoder” subnet. The “encoder / decoder” subnet is responsible for encoding (generating) and decoding (parsing) the first bitstream “bitstream 1”. Figure 3A The second network in Fig. 1 comprises modules 103, 108, 109, 110 and 107, and is referred to as the “super-encoder / decoder” subnet. The second subnet is responsible for generating the second bitstream “bitstream 2”. The purpose of these two subnets is different.

[0122] The first subnet is responsible for:

[0123] • Transform 101 an input image x into its latent representation y (this is easier to compress x),

[0124] • Quantize 102 the latent representation y into a quantized latent representation

[0125] • The arithmetic encoding module 105 compresses the quantized latent representation using AE to obtain a bitstream "bitstream 1".

[0126] • The arithmetic decoding module 106 parses bitstream 1 by AD,

[0127] • Reconstruct 104 the reconstructed image using the parsed data

[0128] The purpose of the second subnetwork is to obtain statistical properties of the "bitstream 1" samples (e.g. mean, variance and correlation between bitstream 1 samples) so that the first subnetwork is more efficient in compressing bitstream 1. The second subnetwork generates a second bitstream "bitstream 2" which includes said information (e.g. mean, variance and correlation between bitstream 1 samples).

[0129] The second network includes an encoding part which includes transforming 103 the quantized latent representation into side information z, quantizing the side information z into quantized side information and encoding 109 (e.g. binarization) said quantized side information into bitstream 2. In the present example, the binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes an arithmetic decoding (AD) 110 which transforms an input bitstream 2 into a decoded quantized side information may be the same as because the arithmetic encoding and arithmetic decoding operations are lossless compression methods. Then, the decoded quantized side information is

[0130] transformed 107 into a decoded side information representing statistical properties of (e.g. mean of samples of or variance of sample values, etc.). Then, the decoded latent representation is provided to the above-mentioned arithmetic encoder 105 and arithmetic decoder 106 to control the probability model.

[0131] Figure 3AAn example of a variational auto encoder (VAE) is described, the details of which can differ in different implementations. For example, in a particular implementation, there can be other components to more efficiently obtain the statistical properties of the samples of the code stream 1. In this implementation, there can be a context modeler, the purpose of which is to extract relevant information of the code stream 1. The statistical information provided by the second subnetwork can be used by the arithmetic encoder (AE) 105 and arithmetic decoder (AD) 106 components.

[0132] Figure 3A The encoder and the decoder are depicted in a single figure. The skilled person will appreciate that the encoder and the decoder can and often are embedded in mutually different devices.

[0133] Figure 3B The encoder is described, Figure 3C The decoder component of the VAE framework is described. According to some embodiments, the encoder receives an image as input. The input image can comprise one or more channels, e.g. color channels or other types of channels, e.g. depth channels or motion information channels, etc. The output (as Figure 3B depicted) of the encoder is a code stream 1 and a code stream 2. The code stream 1 is the output of the first subnetwork of the encoder, the code stream 2 is the output of the second subnetwork of the encoder.

[0134] Similarly, in Figure 3C , two code streams (code stream 1 and code stream 2) are received as input and a reconstructed (decoded) image As mentioned above, the VAE can be divided into different logical units performing different operations. This is illustrated in Figure 3B and Figure 3C , for example, Figure 3B Components participating in the encoding of a signal, e.g. a video, are depicted and provide encoding information. This encoding information is then received by components, e.g. a decoder, in Figure 3C for encoding. It is noted that the functions of the components of the encoder and decoder denoted by the numbers 12x and 14x can correspond to the components denoted by the numbers 10x mentioned above in Figure 3A .

[0135] In particular, as shown in Figure 3B , the encoder comprises an encoder 121 which transforms an input x into a signal y which is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic encoding module 125 and a hyper-encoder 123. The hyper-encoder 123 provides the code stream 2 described above to a hyper-decoder 147 which in turn provides information to the arithmetic encoding module 105 (125).

[0136] The output of the arithmetic encoding module is a bitstream 1. Bitstream 1 and bitstream 2 are the output of the signal encoding, which is then provided (transmitted) to the decoding process. Although unit 101 (121) is referred to as an “encoder”, the entire subnetwork described in Figure 3B may be referred to as an “encoder”. The encoding process generally refers to a unit (module) that converts an input into an encoded (e.g., compressed) output. From Figure 3B It can be seen that unit 121 can actually be considered as the core of the entire subnetwork, as it performs the conversion of input x into y, which is a compressed version of x. The compression in encoder 121 may, for example, be implemented by applying a neural network or any processing network that generally has one or more layers. In such a network, the compression can be performed by a cascading of processes that include down-sampling, which reduces the size and / or the number of channels of the input. Thus, for example, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0137] The remaining parts in the figure (quantization unit, super-encoder, super-decoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 121 by lossy compression. AE 125 can perform binarization together with the super-encoder 123 and the super-decoder 127 used to configure AE 125, and the quantized signal can be further compressed by lossless compression. Thus, the entire subnetwork in Figure 3B may also be referred to as an “encoder”.

[0138] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into binarized numbers (bits). For example, in a VAE framework, an encoder that performs a non-linear transformation maps the input image x into y, where y has a smaller width and height than x. Since y has a smaller width and height, it is smaller in size, and the (size) dimensionality of the signal is reduced, thus, it is easier to compress the signal y. It is noted that, generally, the encoder does not necessarily need to reduce the size in both (or generally all) dimensions. Rather, some example implementations can provide an encoder that reduces the size in only one (or generally a subset) of the dimensions.

[0139] In the arXiv e-printed version of J. Balle, L. Valero Lapara, and Esimoncelli (2015), “Density Modeling of Images Using a Generalized Normalization Transformation,” presented at the 4th International Conference on Learning Representations (hereinafter referred to as “Balle”) in 2016, the authors proposed an end-to-end optimization framework for image compression models based on nonlinear transformations. The authors optimized for Mean Squared Error (MSE) but used a more flexible transformation constructed from linear convolutions and nonlinear concatenations. Specifically, they employed a joint nonlinearity of generalized divisive normalization (GDN), inspired by neuron models in biological visual systems and proven effective for Gaussianizing image density. This concatenated transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which facilitates the parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0140] Such an example of the VAE framework is as follows: Figure 4 As shown, it utilizes six downsampling layers, labeled 401 to 406. The network architecture includes a hyperprior model. (Left side (g)) a g s The image autoencoder architecture is shown on the right (h). a h s This corresponds to the implementation of a priori autoencoders. Factorized prior models use the same architecture for analysis and synthesis transformations. a and g s Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder inputs the input image x into g. a This produces a response y (latent representation) with spatially varying standard deviation. The encoding g... a It includes multiple convolutional layers, with subsampling and generalized divisive normalization (GDN) as the activation function.

[0141] Feed the response to h a In the first step, the standard deviation distribution of z is summarized. Then, it is quantized, compressed, and transmitted as side information z. The encoder then uses the quantized vector... To estimate i.e. the spatial distribution of the standard deviation, used to obtain the probability values (or frequency values) for the arithmetic coding (AE) and to use it to compress and transmit the quantized image representation (or latent representation). The decoder first recovers from the compressed signal s obtains This provides it with the correct probability estimates to successfully recover from the compressed signal is fed into g s to obtain the reconstructed image.

[0142] The layers comprising down-sampling are indicated in the layer description with a downward arrow. The layer description "Conv Nx5x5 / 2↓" means that this layer is a convolutional layer with N channels and a kernel size of 5x5. As mentioned above, 2↓ means that a 2-fold down-sampling is performed in this layer. A 2-fold down-sampling results in a reduction of one dimension of the input signal by a factor of two at the output. In Figure 4 , 2↓ means that both the width and the height of the input image are reduced by a factor of two. Since there are 6 down-sampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, then the width and height of the output signal z^413 are equal to w / 64 and h / 64, respectively. The modules denoted by AE and AD are an arithmetic encoder and an arithmetic decoder, which will be explained with reference to Figure 3A to Figure 3C . The arithmetic encoder and decoder are a specific implementation of entropy coding. The AE and AD can be replaced by other entropy coding means. In information theory, entropy coding is a lossless data compression scheme for converting the values of symbols into a binary representation, which is a reversible process. Furthermore, "Q" in the figure corresponds to the quantization operation also mentioned above with respect to Figure 4 and further explained in the section "Quantization" above. Furthermore, the quantization operation and the respective quantization unit being part of the component 413 or 415 does not necessarily exist and / or can be replaced by another unit.

[0143] In Figure 4 , it is also shown that the decoder comprises up-sampling layers 407 to 412. Another layer 420 is provided in the processing order of the input between the up-sampling layers 411 and 410, which is implemented as a convolutional layer, but does not provide an up-sampling of the received input. It is also shown that a respective convolutional layer 430 is used for the decoder. Such a layer can be provided in the NN for performing an operation on the input which does not change the size of the input but changes specific features. However, it is not necessary to provide such a layer.

[0144] When seen from the processing order of the bitstream 2 through the decoder, the up-sampling layers run in reverse order, i.e. from up-sampling layer 412 to up-sampling layer 407. Each up-sampling layer is here shown to provide an up-sampling with a factor of 2, denoted by. Of course, it is not necessary that all up-sampling layers have the same up-sampling factor, and other up-sampling factors can be used as well, like 3, 4, 8, etc. Layers 407 to 412 are implemented as convolutional layers (conv). In particular, since they can provide the inverse operation of the encoder on the input, the up-sampling layers can apply an inverse convolution operation on the received input, such that its size is increased by a factor corresponding to the up-sampling factor. However, the present invention is generally not limited to inverse convolution, and the up-sampling can be performed in any other way, e.g. by bilinear interpolation between two adjacent samples, or by nearest-neighbor sample replication, etc.

[0145] In the first subnetwork, after some convolutional layers (401 to 403) on the encoding side, a generalized divisive normalization (GDN) is applied, and on the decoding side, an inverse GDN (IGDN) is applied. In the second subnetwork, a ReLu activation function is applied. It is noted that the present invention is not limited to this implementation, and generally, other activation functions can be used instead of GDN or ReLu.

[0146] End-to-end image or video compression

[0147] DNN-based image compression methods can exploit large-scale end-to-end training and highly non-linear transformations, which are not used in traditional methods. However, direct application of these techniques to build end-to-end learning systems for video compression is not common. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0148] One simple solution is to represent the motion information using learning-based optical flow. However, the goal of current learning-based optical flow methods is to generate the flow field as accurately as possible. Precise optical flow is not always the best choice for specific video tasks. Moreover, compared with the motion information in traditional compression systems, the data volume of optical flow is significantly increased, and directly applying existing compression methods to compress the optical flow values will significantly increase the number of bits required to store the motion information. Second, it is not clear how to construct a DNN-based video compression system by minimizing the rate-distortion-based purpose of residual and motion information. The purpose of rate-distortion optimization (RDO) is to achieve higher quality (i.e., less distortion) of the reconstructed frame when given the number of compressed bits (or bit rate). RDO is very important for video compression performance. In order to take advantage of the power of end-to-end training of learning-based compression systems, an RDO strategy is needed to optimize the entire system.

[0149] In Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao, “DVC: An End-to-end Deep Video Compression Framework,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0150] Such an encoder is shown in Figure 5A Specifically, Figure 5A The overall structure of the end-to-end trainable video compression framework is shown. To compress the motion information, a CNN is specified to transform the optical flow into a corresponding representation that is better suited for compression. Specifically, an autoencoder-style network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 5B The network architecture is similar to the g Figure 4 a / g s ​Some similarities. Specifically, the optical flow is fed into a series of convolutional operations and nonlinear transformations including GDN and IGDN. The output channel number of the convolution (deconvolution) is 128, but the last deconvolution layer is equal to 2. Given the optical flow size M x N x 2, the MV encoder will generate a motion representation with size M / 16 x N / 16 x 128. Then, the motion representation is quantized, entropy coded and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0151] Figure 5C The structure of the motion compensation part is shown. Here, the previously reconstructed frame x t-1 and the reconstructed motion information, the warping unit generates a warped frame (typically, with the help of an interpolation filter, e.g., bilinear interpolation filter). Then, a separate CNN with three inputs generates the predicted image. The architecture of the motion compensation CNN is also shown as Figure 5C

[0152] The residual information between the original frame and the predicted frame is encoded by the residual encoder network. A highly nonlinear neural network is used to transform the residual into the corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better exploit the power of nonlinear transformation to achieve higher compression efficiency.

[0153] As can be seen from the above overview, considering different parts of the video framework, including motion estimation, motion compensation and residual coding, CNN-based architectures can be applied to image and video compression. Entropy coding is a popular method for data compression, widely adopted by the industry, and is also applicable to feature map compression for human perception or computer vision tasks.

[0154] Machine video coding

[0155] Machine video coding (VCM) is another popular direction in computer science today. The main idea behind this approach is to transmit the encoded representation of image or video information so that it can be further processed by computer vision (CV) algorithms, such as object segmentation, detection and recognition. Compared with traditional image and video coding for human perception, the quality feature is the performance of computer vision tasks, e.g., object detection accuracy, rather than the reconstruction quality.

[0156] ​A recent study proposes a new deployment paradigm, called collaborative intelligence, which partitions a deep model between the mobile and the cloud. Extensive experiments under various hardware configurations and wireless connectivity modes show that the optimal operating point in terms of energy consumption and / or computation latency involves partitioning the model, usually at some point deep in the network. The common schemes nowadays, where the model is either completely in the cloud or completely in the mobile, are rarely, if ever, optimal. The concept of collaborative intelligence is also extended to model training. In this case, the data flow is bidirectional: from the cloud to the mobile during the backward pass in training, from the mobile to the cloud during the forward pass in training, and for inference.

[0157] In the context of recent deep models for object detection, lossy compression of deep feature data is studied based on HEVC intra coding. As the level of compression increases, the detection performance degrades, and compression-enhanced training is proposed to minimize this loss by generating a model that is more robust to the quantization noise in the feature values. However, this is still a suboptimal technical solution because the codec used is very complex and optimized for natural scene compression rather than deep feature compression.

[0158] The trade-off between compression efficiency and recognition accuracy is studied through a method that uses the popular YOLOv2 network for the object detection task, addressing the problem of deep feature compression for collaborative intelligence. Here, the term "deep feature" has the same meaning as "feature map." The word "deep" comes from the idea of collaborative intelligence where the output feature maps of certain hidden (deep) layers are captured and transmitted to the cloud for inference. This seems more efficient than sending compressed natural image data to the cloud and performing object detection using the reconstructed image.

[0159] Efficient compression of feature maps is beneficial for image and video compression and reconstruction, both for human perception and for machine vision. The drawbacks regarding the state-of-the-art autoencoder-based compression methods also apply to machine vision tasks.

[0160] Artificial neural network with skip connections

[0161] A residual neural network (ResNet) is a type of artificial neural network (ANN) that builds on the known structure of the cerebral cortex pyramidal cells. Residual neural networks achieve this goal by utilizing skip connections or shortcuts that skip certain layers. A typical ResNet model is implemented with double or triple skips that include nonlinear (ReLU) and intermediate batch normalization. Additional weight matrices can be used to learn the skip weights; these models are referred to as HighwayNets. Models with multiple parallel skips are referred to as DenseNets. In the context of a residual neural network, a non-residual network can be described as a vanilla network.

[0162] One purpose of the skip layer is to avoid the vanishing gradient problem by reusing the activation of the previous layer until the adjacent layer learns its weights. During training, the weights are adjusted to mute the upstream layers and amplify the previously skipped layers. In the simplest case, only the weights of the adjacent layer connections are adjusted without adjusting the explicit weights of the upstream layers. The effect is best when a single nonlinear layer is crossed, or when the intermediate layers are all linear. If not, an explicit weight matrix should be learned for the skipped connection (HighwayNets should be used).

[0163] This skipping simplifies the network, thereby reducing the number of layers used during the initial training phase. This speeds up learning by reducing the impact of vanishing gradients, as fewer layers are to be propagated. Then, the network gradually restores the skipped layers as it learns the feature space. At the end of training, when all layers are expanded, it will remain closer to the manifold, thereby learning faster. A neural network without the residual part can explore more of the feature space. This makes it more susceptible to perturbations that cause it to leave the manifold and requires additional training data to recover.

[0164] Longer skip connections are introduced in U-Net, as shown in Figure 6 The U-Net architecture is derived from the "fully convolutional network" first proposed by Long and Shelhamer. The main idea is to complement the usual contracting network by successive layers that each increase the resolution of the output. These layers thus improve the resolution of the output. In addition, successive convolutional layers can learn an accurate output based on this set of information.

[0165] A key modification in U-Net is the inclusion of numerous feature channels in the upsampling portion, allowing the network to propagate contextual information to higher-resolution layers. Consequently, the expansion path is more or less symmetrical to the contraction path, resulting in a U-shaped architecture. The network uses only the effective portion of each convolution, without any fully connected layers. To predict pixels in image boundary regions, missing context is extrapolated by mirroring the input image. This tiling strategy is crucial for applying the network to large images, as resolution would be limited by GPU memory without it.

[0166] Introducing skip connections can better capture features at different spatial resolutions, and this has been successfully applied to computer vision tasks such as object detection and segmentation. However, such skip connections in image or video compression are not a trivial task, because information from the encoding end needs to be transmitted in the communication channel, and direct connections between layers would require the transmission of large amounts of data.

[0167] Traditional hybrid video codec

[0168] Neural network frameworks can also be used in combination or in traditional hybrid encoding and decoding, as illustrated later. A very brief overview of exemplary hybrid encoding and decoding is given below.

[0169] Figure 7A This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 7A In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 7A The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0170] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, for example, an image in an image sequence forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be decoded (especially in video decoding when distinguishing the current image from other images, such as those in the same video sequence, i.e., including previously encoded and / or decoded images in the video sequence of the current image).

[0171] A (digital) image is, or can be, a two-dimensional array or matrix of samples with intensity values. Samples in the array can also be called pixels (a short form of image element). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the image size and / or resolution. To represent color, three color components are typically used; that is, the image can be represented as or comprise three sample arrays. In RGB format or color space, the image comprises corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented in a luminance and chrominance format or in a color space, for example, YCbCr, comprising a luminance component represented by Y (sometimes also L) and two chrominance components represented by Cb and Cr. The luminance component Y represents luminance or grayscale intensity (e.g., as in a grayscale image), and the two chrominance components Cb and Cr represent chrominance or color information components. Accordingly, an image in YCbCr format comprises a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An RGB format image can be converted to YCbCr format and vice versa; this process is also known as color transformation or conversion. If the image is black and white, it can consist only of a luminance sample array. Accordingly, for example, an image can be a black and white luminance sample array or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0172] Embodiments of the video encoder 20 may include an image segmentation unit ( Figure 7A (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be referred to as root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) (H.265 / HEVC and VVC). Image segmentation units can be used to segment each image into corresponding blocks by applying the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or by varying the block size between images, subsets of images, or groups of images. The abbreviation AVC stands for Advanced Video Coding.

[0173] In further embodiments, the video encoder can be configured to receive directly a block 203 of the picture 17, e.g. one, several or all blocks constituting the picture 17. The picture block 203 can also be referred to as a current picture block or a picture block to be coded.

[0174] As with the picture 17, the picture block 203 likewise is or can be considered to be a two-dimensional array or matrix of samples consisting of intensity values (sample values), but the size of the picture block 203 is smaller than that of the picture 17. That is, for example, the block 203 can comprise one array of samples (e.g. a luminance array in case of a black and white picture 17, or a luminance or chrominance array in case of a color picture), or three arrays of samples (e.g. a luminance array and two chrominance arrays in case of a color picture 17), or any other number and / or type of arrays depending on the applied color format. The number of samples in the horizontal and vertical direction (or axis) of the block 203 defines the size of the block 203. Thus, the block can be an M x N (M columns x N rows) array of samples, or an M x N array of transform coefficients, etc.

[0175] Figure 7A The illustrated embodiment of the video encoder 20 can be configured to encode the picture 17 block-wise, e.g. to perform encoding and prediction for each block 203.

[0176] Figure 7A The illustrated embodiment of the video encoder 20 can also be configured to segment and / or encode the picture using slices (also referred to as video slices), wherein the picture can be segmented or encoded using one or more slices (typically non-overlapping), and each slice can comprise one or more blocks (e.g. CTUs).

[0177] Figure 7A The illustrated embodiment of the video encoder 20 can also be configured to segment and / or encode the picture using tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles), wherein the picture can be segmented or encoded using one or more tile groups (typically non-overlapping), each tile group can comprise one or more blocks (e.g. CTUs) or one or more tiles, etc., wherein each tile can be rectangular or the like shape, can comprise one or more blocks (e.g. CTUs), e.g. complete or partial blocks.

[0178] Figure 7B An example of a video decoder 30 for implementing the techniques of the present application is shown. The video decoder 30 is configured to receive encoded picture data 21 (e.g. an encoded bitstream 21), e.g. encoded by the encoder 20, to obtain a decoded picture 331. The encoded picture data or bitstream comprises information for decoding the encoded picture data, e.g. data representing picture blocks of encoded video slices (and / or tile groups or tiles) and associated syntax elements.

[0179] The entropy decoding unit 304 is configured to parse the bitstream 21 (or generally coded picture data 21) and, for example, entropy-decode the coded picture data 21 to obtain quantized coefficients 309 and / or decoded syntax elements (not shown in FIG. 3), such as inter prediction parameters (e.g., reference picture indices and motion vectors), intra prediction parameters (e.g., intra prediction modes or indices), transform parameters, quantization parameters, loop filter parameters, and / or any or all of the other syntax elements. The entropy decoding unit 304 can be configured to apply decoding algorithms or schemes corresponding to the encoding schemes described for the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can be further configured to provide inter prediction parameters, intra prediction parameters, and / or other syntax elements to the mode application unit 360, as well as to provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice level and / or the video block level. Additionally or as an alternative to slices and corresponding syntax elements, tile groups and / or tiles and corresponding syntax elements can be received or used.

[0180] The reconstruction unit 314 (e.g., an adder or summer 314) can be configured to add the reconstructed residual block 313 to the prediction block 365 to obtain a reconstructed block 315 in the sample domain, e.g., by adding the sample values of the reconstructed residual block 313 to the sample values of the prediction block 365.

[0181] Figure 7B Embodiments of the video decoder 30 as shown in FIG. 3 can be configured to use slices (also referred to as video slices) for partitioning and / or decoding pictures, wherein a picture can be partitioned or decoded using one or more slices (typically non-overlapping), and each slice can comprise one or more blocks (e.g., CTUs).

[0182] Figure 7B Embodiments of the video decoder 30 as shown in FIG. 3 can be configured to use tile groups (also referred to as video tile groups) and / or tiles (also referred to as video tiles) for partitioning and / or decoding pictures, wherein a picture can be partitioned or decoded using one or more tile groups (typically non-overlapping), each tile group can comprise one or more blocks (e.g., CTUs) or one or more tiles, etc., wherein each tile can be rectangular or the like shape, can comprise one or more blocks (e.g., CTUs), such as complete or partial blocks.

[0183] Other variants of video decoder 30 can be used to decode encoded image data 21. For example, decoder 30 can produce an output video stream without loop filter unit 320. For example, a non-transform based decoder 30 can directly dequantize the residual signal for certain blocks or frames without inverse transform processing unit 312. In another implementation, inverse quantization unit 310 and inverse transform processing unit 312 can be combined into one unit in video decoder 30.

[0184] It should be appreciated that in encoder 20 and decoder 30, the processing results of the current step can be further processed before output to the next step. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing results of interpolation filtering, motion vector derivation or loop filtering can be further operated, such as clip or shift operation.

[0185] Improving coding efficiency

[0186] As mentioned above, image and video compression methods based on variational autoencoder methods lack spatial means for adaptive processing and object segmentation to capture real object boundaries. Therefore, the content adaptivity is limited. In addition, for certain types of video information, such as motion information or residual information, sparse representation and coding are ideal to keep the indication overhead at a reasonable level.

[0187] Therefore, some embodiments of the present application introduce segmentation information coding and feature map coding from different spatial resolution layers of autoencoders to achieve content adaptivity and sparse signal representation and transmission.

[0188] In some exemplary implementations, connections are introduced between the encoder and decoder layers except the lower resolution layers (latent space) which are transmitted in the bitstream. In some exemplary implementations, only a part of the feature maps of different resolution layers are provided in the bitstream to save bandwidth. For example, signal selection and signal feeding logic is introduced to select, transmit and use the part of the feature maps from different resolution layers. At the receiving end, tensor combination logic is introduced to combine the output of the last resolution layer with the information received from the bitstream corresponding to the current resolution layer.

[0189] In the following, some detailed embodiments and examples related to the encoding end and the decoding end are provided.

[0190] Encoding method and device

[0191] According to an embodiment, a method of encoding data for image or video processing into a bitstream is provided. The method comprises the steps of processing the data, including generating feature maps in a plurality of cascaded layers, each feature map comprising a respective resolution, wherein the resolutions of at least two of the generated feature maps are different from each other.

[0192] That is, the resolutions of two or more cascaded layers can be different from each other. Here, the resolution of a layer refers to the resolution of the feature map processed by the layer. In an exemplary implementation, it refers to the resolution of the feature map output by the layer. A feature map comprising a resolution means that at least a portion of the feature map has the resolution. In some implementations, the entire feature map can have the same resolution. For example, the resolution of a feature map can be given by the number of feature map elements in the feature map. However, it can also be more specifically defined by the number of feature map elements in one or more dimensions (e.g., x, y; or, or in addition, the number of channels can be taken into account).

[0193] The term layer here refers to a processing layer. It does not necessarily have to be a layer with trainable or trained parameters (weights), as in some of the neural network layers mentioned above. Instead, a layer can represent a specific processing of a layer input to obtain a layer output. In some embodiments, a layer can be trained or trainable. Training here refers to machine learning or deep learning.

[0194] When referring to cascaded layers, it means that the layers have a certain predefined order (sequence), and that the input to the first layer (in the given order) is sequentially processed by the first layer and the other layers according to the given order. That is, the output of the jth layer is the input of the (j+1)th layer, j being an integer from 1 to the total number of cascaded layers. In one particular and non-limiting example, for all possible values of j, the (j+1)th layer comprises (or has) the same or lower resolution than the jth layer. That is, the resolution of the layers does not increase (e.g., at the encoding end) with the order of the cascade (processing sequence). However, it is noted that the present invention is not limited to this particular cascade of layers. In some embodiments, the layers of the cascade can also include layers that increase the resolution. In any case, there can be layers that do not change the resolution.

[0195] For example, a lower resolution of a feature map can mean fewer feature elements per feature map. For example, a higher resolution of a feature map can mean more feature elements per feature map.

[0196] The method further comprises the steps of selecting a layer from the plurality of layers that is different from the layer that generates the lowest resolution feature map, and generating the bitstream comprises inserting information related to the selected layer into the bitstream.

[0197] That is, in addition to (or alternatively) outputting the processed results of all layers in the cascade to the bitstream, information is provided to another (selected) layer. There can be one or more selected layers. The information related to the selected layer can be any type of information, e.g., the output of the layer or some segmentation information of the layer (described below), or other information related to the feature maps of the layer processing and / or the processing performed by the layer. That is, in some examples, the information can be elements of the feature maps and / or the positions of the elements within the feature maps (within the layer).

[0198] The input to the cascade processing is data for image or video processing. For example, such data can be related to prediction coding, e.g., inter or intra prediction. It can be motion vectors or other parameters of prediction, like prediction modes or reference pictures or directions, or other coding parts than prediction, like transforms, filtering, entropy coding or quantization. The bitstream generation can include any conversion of values to bits including fixed codewords, variable length codes or arithmetic encoding (binarization).

[0199] Here, the image can be a still image or a video image. An image refers to one or more samples, e.g., captured by a camera or generated by computer graphics or the like. The image can include samples representing luminance levels in gray scale or can have multiple channels including one or more of a luminance channel, a chrominance channel, a depth channel or other channels. The image or video coding can be any of hybrid coding as described above (e.g., similar to HEVC or VVC or the like) or autoencoder.

[0200] Figure 8 A flowchart of the above described method. Thus, the method comprises a step 810 of processing input data. In a selection step 820, a portion is selected from the processed data and the data portion is included in a bitstream in a generation step 830. Not all data generated in the processing step has to be included in the bitstream.

[0201] According to an example implementation, the processing further comprises down-sampling by one or more of the cascade layers. Figure 9 An example network 900 implementing (in operation) such processing is shown in Fig.

[0202] In particular, Figure 9 Input data for image or video processing 901 is shown entering the network 900. The input data for image or video processing can be any type of data for such processing, e.g., samples of an image (picture) or video directly as described above, prediction modes, motion vectors or the like. Figure 9The processing applied to input 901 is performed by a plurality of processing layers 911-913, each of which reduces the resolution of each array of motion vectors. That is, cascaded layers 911-913 are down-sampling layers. Note that when a layer is referred to as a down-sampling layer, it performs down-sampling. There are embodiments in which down-sampling layers 911-913 perform down-sampling as the only task; and there are embodiments in which down-sampling layers 911-913 do not perform down-sampling as the only task. Rather, a down-sampling layer can also typically perform other types of processing.

[0203] As can be seen in Figure 9 , in addition to the data input and output of the processing, down-sampling layers 911-913 have additional selection outputs that result in signal selection logic 920. Note that the term "logic" here refers to any circuitry (here, signal selection) that implements the function. Signal selection logic 920 selects information from the selection outputs of any of the layers to include in bitstream 930. In Figure 9 the example shown, each layer 911-913 down-samples the layer input. However, layers can also be added between the down-sampling layers that do not apply down-sampling. For example, these layers can process the input through filtering or other operations.

[0204] In Figure 9 the example shown, signal selection logic 920 selects information from the outputs of layers 911-913 to include in the bitstream. The goal of this selection can be to select information from the multiple feature maps output by the different layers that is relevant to the reconstructed image or video. That is, the down-sampling layers and signal selection logic can be implemented as part of an encoder (image or video encoder). For example, the encoder can be Figure 3A encoder 101 shown in Figure 3B , encoder 121, MV encoder net Figure 5A part of end-to-end compression in Figure 5B , MV encoder, or Figure 7A some portion of the encoder (e.g., loop filter 220 or a portion of mode selection unit 260 or prediction units 244, 254, etc.

[0205] Figure 9 A decoding-side portion (which can be referred to as an expansion path) is also included, comprising signal feeding logic 940 and up-sampling layers 951-953. The input to the encoding-side is bitstream 930. For example, output 911 is the reconstructed input 901. The decoding-side is described later below.

[0206] For example, down-sampling can be done through max-pooling, average-pooling, or any other operation that results in down-sampling. Another example of such an operation includes a convolution operation. Figure 10AAn example of max pooling is shown. In this example, every four elements of array 1010 (adjacent 2x2 square) are grouped and used to determine one element in array 1020. Arrays 1020 and 1010 can correspond to feature maps in some embodiments of the present invention. However, the arrays can also correspond to portions of feature maps of the present embodiments. The fields (elements) in arrays 1020 and 1010 can correspond to elements of a feature map. In this image, feature map 1020 is determined from down-sampled feature map 1010. The numbers in the fields of arrays 1010 and 1020 are merely exemplary. For example, the fields can also include motion vectors instead of numbers. In Figure 10A In the max pooling example shown, the four fields in the upper left corner of array 1010 are grouped and the maximum of their values is selected. This group of values determines the upper left field of array 1020 by assigning the maximum value to the field. That is, the maximum of the four upper left values of array 1010 is inserted into the upper left field of array 1020.

[0207] Alternatively, in some implementations, min pooling can be used. In min pooling, the field with the minimum value is selected instead of the field with the maximum value. However, these down-sampling techniques are merely exemplary and various down-sampling strategies can be used in different embodiments. Some implementations can use different down-sampling techniques at different layers, in different regions within a feature map, and / or for different types of input data.

[0208] In some implementations, down-sampling is performed using average pooling. In average pooling, the average of a group of feature map elements is computed and associated with a corresponding field in the feature map of the down-sampled feature map.

[0209] Figure 10B An example of average pooling is shown. In this example, the feature map elements in the upper left corner of feature map 1050 are averaged and the upper left element of feature map 1060 takes on this average value. Figure 10B The three groups of upper right, lower right, and lower left in show the same situation.

[0210] In another embodiment, convolution operations are used for downsampling in some or all layers. In convolution, a filter kernel is applied to groups or blocks of elements in the input feature map. The kernel itself can be an array of elements with the same size as the input element block, where each element of the kernel stores the weights of the filter operation. In downsampling, the sum of the elements in the input block is computed, with each element weighted by the corresponding value taken from the kernel. If the weights of all elements in the kernel are fixed, such a convolution can correspond to the filter operation described above. For example, a convolution with kernels of the same, fixed weights and kernel size stride corresponds to an average pooling operation. However, the stride of the convolution used in this embodiment can be different from the kernel size, and the weights can be different. In one example, the kernel weights can be such that certain features in the input feature map can be enhanced or distinguished from each feature. Furthermore, the kernel weights can be learnable or learned in advance.

[0211] According to one embodiment, the information associated with the selected layer includes element 1120 of the feature map of that layer. For example, this information may convey feature map information. Typically, the feature map may include any features associated with the moving image.

[0212] Figure 11 Feature map 1110 is shown as an exemplary implementation of dense optical flow of a motion vector with width W and height H. Motion segmentation mesh 1140 includes three downsampling layers (e.g., corresponding to...). Figure 9 The downsampling layers 911 to 913 and the signal selection circuit (logic) 1100 (e.g., corresponding to signal selection logic 920) are included. Figure 11 Examples of the outputs (L1 to L3) of different layers in the right-hand contraction path are shown.

[0213] In this example, the output of each layer (L1 to L3) is a feature map with progressively decreasing resolution. The input to L1 is a dense optical flow 1110. In this example, one element of the feature map output from L1 is determined from sixteen (4×4) elements of the dense optical flow 1110. Each square in the L1 output ( Figure 11 The lower right corner corresponds to the motion vector obtained by downsampling (downspl4) from the 16 motion vectors of the dense optical flow. Such downsampling can be, for example, average pooling or another operation, as described above. In this exemplary implementation, information 1120 includes only a portion of the feature map L1 of that layer. Layer L1 is selected, and the portion corresponding to the four motion vectors (feature map elements) associated with the selected layer is sent within the selected information 1120.

[0214] The output L1 of the first layer is then input to the second layer (downspl2). The output L2 feature map elements of the second layer are determined from the four elements of L1. However, in other examples, each element of a feature map with a lower resolution can also be determined from a group of any other number of elements of a feature map with the next higher resolution. For example, the number of elements in a group that determines one element in the next layer can also be any power of 2. In the present example, the output L2 feature map corresponds to three motion vectors, which are also included in the selected information 1120, so the second layer is also a selected layer. The third layer (downspl2) downsamples the output L2 of the second layer by a factor of 2 in each of the two dimensions. Accordingly, one feature map element of the third layer output L3 is derived from the four elements of L2. In the feature map L3, no element is transmitted, i.e. in the present example, the third layer is not a selected layer.

[0215] The signal selection module 1100 of the motion segmentation network 1140 selects the above-mentioned motion vectors (elements of the feature maps from the first and second layer outputs) and provides them to the bitstream 1150. The specification can be a simple binarization, which can but does not necessarily include entropy coding.

[0216] The groups of elements can be arranged in a square shape, as shown in the example of Figure 11 However, the groups can also be arranged in any other shape, for example a rectangular shape, where the long side of the rectangular shape can be arranged in the horizontal or vertical direction. The shapes are just examples. In one implementation, arbitrary shapes can be used. The shape can also be indicated in the bitstream 1150. The indication can be achieved by a flag map indicating which feature elements belong to the shape and which do not. Or, the indication can be done using a more abstract description of the shape.

[0217] In the present example implementation, the feature map elements are grouped such that each element belongs to exactly one group of elements that determines one element of the next layer feature map. I.e. the feature map element groups are non-overlapping, only one group contributes to a feature map element of a higher (later in the cascading processing order) layer. However, it is conceivable that one element of a layer can be used for multiple elements of the next layer. I.e. when generating a new layer output, e.g. layer output L2, from a layer output L1 with a higher resolution, a filter operation can be used in the processing 810.

[0218] In the present embodiment, the selection 820 (e.g. by means of the signal selection 1100) selects elements from the plurality of output feature maps (L1 to L3) to be included in the bitstream. By the selection, the amount of data required to transmit the selected data is small, while at the same time the amount of information relevant for decoding is kept as large as possible. For example, rate-distortion optimization or other optimization can be employed.

[0219] The above example shows a three-layer processing. Generally, the method is not limited to this. Any number of processing layer(s) can be used. That is, according to a more general example, the method comprises obtaining data to be encoded. This can be a dense stream of motion vectors 1110, as shown above. However, the invention is not limited to this, and instead of or in addition to motion vectors, other data can be processed, such as prediction modes, prediction directions, filtering parameters, even spatial image information (samples) or depth information, etc.

[0220] The processing 810 of the data to be encoded comprises, in this example, a processing by each j-th layer of a plurality of N cascaded layers.

[0221] The processing of the j-th layer comprises:

[0222] - if j = 1, obtaining the data to be encoded as layer input, otherwise obtaining the feature maps of the (j - 1)-th layer processing as layer input (i.e. if the i-th layer is the current processing layer, the (j - 1)-th layer is the previous layer);

[0223] - processing the obtained layer input, the processing comprising down-sampling;

[0224] - outputting the down-sampled feature maps.

[0225] In the present example, the layer for j = 1 is the layer with the highest resolution among the N processing layers. It is to be noted that the input of this layer can be a dense optical flow (which can also be considered as a feature map in general). Thus, in some specific embodiments, the layer for j = 1 can be the input layer. However, this is not necessarily the case, as it is conceivable that there are some pre-processing layers before the N processing layers. A typical feature of an encoder is that the earlier processing layers have higher resolution than the later processing layers (shrinkage path). At the decoding side, this is a reversed process accordingly. It is conceivable that some processing layers do not change the resolution, or even enhance the resolution - the invention is still applicable.

[0226] In the above example, the bitstream 1150 carries the selected information 1120. For example, the information can be motion vectors or any other feature. That is, the bitstream 1150 carries feature map elements from at least one output layer that is not a processing network (encoding side processing network) output layer. In Figure 11In the example shown, only a part of the selected feature map is signaled in the bitstream. This part has one or more feature elements. In order to enable the decoder to determine which part of the feature map is transmitted, rules for the determination can be defined. In some embodiments, partition information can be signaled in the bitstream 1150 to configure which parts of the feature map are transmitted. These exemplary embodiments will be described below. However, it is to be noted that the above embodiments are only exemplary and such additional signaling is generally not needed because rules for deriving the information can exist and depend on other known or indicated parameters.

[0227] In exemplary embodiments related to the partition information, the information related to the selected layer comprises (in addition or alternatively to the selected information 1120) information 1130 indicating from which layer and / or from which part of the feature map of the layer the elements of the feature map of the layer are selected.

[0228] In Figure 11 In the example shown, the partition information is illustrated by a binarized flag. For example, on the right side, each lower resolution feature map or feature map part is assigned a value of 0 or 1. For example, L3 is assigned a value of zero (0) because it is not selected and no motion vector (feature element) signal is indicated for L3. The feature map L2 has four parts. The layer processing L2 is the selected layer. The feature map elements (motion vectors) of three of the four parts are indicated and accordingly the flag is set to 1. The remaining one part of the feature map L2 does not include a motion vector and therefore the flag is set to 0 because the motion vector corresponding to this part is represented by the L1 feature map. Since L1 is the first layer, the remaining motion vector is implicitly provided in this layer. It is to be noted that the binarized flag here takes a first value (e.g. 1) when the corresponding feature map part is a partial selected information and a second value (e.g. 0) when the corresponding feature map part is not a partial selected information. Since it is a binarized flag, it can only take one of the two values.

[0229] Such partition information can be provided in the bitstream. Figure 11The left side of Figure 11 shows the processing of the segmentation information 1130. It is noted that the segmentation information 1130 can also be processed by the layers of the motion segmentation network 1140. It can be processed in the same layers as the feature maps or in separate layers. The segmentation information 1130 can also be interpreted as follows. One superpixel of the layer with the lowest resolution covers a 16x16 cell of the feature map resulting from the downsampling of the dense optical flow 1110 by downspl4. Since the flag assigned to the superpixel covering the 16x16 cell is set to 0, it means that the feature map element, here a motion vector, is not indicated for this layer (this layer is not selected). Thus, the feature map element can indicate in the area corresponding to the 16x16 cell of the next layer, which is represented by four superpixels of equal size, each covering a cell of 8x8 feature elements. Each of these four superpixels is associated with a flag. For those superpixels associated with a flag having a value of 1, the feature map element (motion vector) is indicated. For the superpixels with a flag set to 0, the motion vector signal is not indicated. The motion vector not indicated is indicated for the layer of the superpixel covering a cell of 4x4 elements.

[0230] More generally, the method for encoding data for image / video decoding can also comprise selecting (segmenting) information to insert in the bitstream. This information concerns a first region (superpixel) in the feature map processed by the layer j>1. The first region corresponds to a region in the feature map or initial data to be encoded in a layer smaller than j comprising a plurality of elements. The method also comprises the step of not selecting, when selecting the feature map processed by the k layers, a region corresponding to said first region, where k is an integer equal to or greater than 1 and k Figure 11 In the example of Figure 11, the initial data segmented is the LI data. However, this correspondence can also refer to the dense optical flow 1110.

[0231] By way of example, the initial data segmented is the LI data. However, this correspondence can also refer to the dense optical flow 1110. Figure 11The particular arrangement in the concatenation of layers ensures that each feature element of the initial feature map (e.g. LI) is only included in a super-pixel of only one of the N layers. This arrangement provides the advantage of an efficient coding of the feature map and the segmentation information. The concatenation of layer processing framework corresponds to a neural network processing framework, which can segment data in this way and provide data of individual segmentations at different resolutions. In particular, the advantage of downsampling in some layers can include a reduction of the amount of data required to indicate the initial feature map representation. Specifically, in the example of indicating motion vectors, due to the downsampling, groups of similar motion vectors can be indicated by one common motion vector. However, in order to achieve good inter prediction, the prediction error caused by the grouping of motion vectors should be small. This can mean that, for different regions of the image, different levels of grouping of motion vectors can be the optimal level to achieve the required prediction quality, while at the same time, a small amount of data is required to indicate the motion vector signal. This can be achieved using multiple layers with different resolutions.

[0232] In embodiments where the feature map elements are motion vectors, for the purpose of downsampling, the length and direction of the motion vectors can be averaged and the averaged motion vector is associated with the corresponding feature map element of the downsampled feature map. In a usual average, all elements of the group of elements corresponding to one element in the downsampled feature map have the same weight. This corresponds to applying a filter with equal weights to the group or block of elements to calculate the downsampled feature map element. However, in other implementations, such a filter can have different weights for different elements in the layer input. In other implementations, the median of the respective group of elements can be calculated instead of calculating an average of the group or block of elements for the downsampling.

[0233] In Figure 11 In the example of FIG. 6, the downsampling filter operation uses a square filter with an input element size of 2x2 and calculates a filter output that is mapped to one element in the downsampled feature map according to the selected filter operation. The filter operation uses a stride that is equal to the edge length of the square filter or 2 steps. That is, between two filter operations, the filter will move a step size that is equal to the filter size. Thus, in the downsampling, the downsampled elements are calculated from non-overlapping blocks in the layer where the downsampling filter is applied.

[0234] However, in some further conceivable embodiments, the step size can be different from the edge length of the filter. For example, the step size can be smaller than the length of the filter edge. Thus, the filter blocks used to determine the elements in the downsampled layer can overlap, i.e. one element in the feature map to be downsampled contributes to the calculation of multiple elements in the downsampled feature map.

[0235] In general, the data related to a selected layer includes an indication of the position of the feature map elements in the feature map of the selected layer. Here, similar to theFigure 11 The concept of a selected layer's feature map (feature maps L1 to L3) refers to the output from the selected layer, i.e. the feature map processed by the selected layer.

[0236] For example, the positions of selected and non-selected feature map elements are indicated by a plurality of binarization flags, the indication being based on the positions of the flags in the code stream. In the description above with reference to Figure 11 The binarization flags are included in the code stream 1150 as partition information 1130. In order for the decoder to be able to parse and correctly interpret the partition information, the assignment between the flags in the feature map processed by the layer and the layer and / or region should be defined. This can be done by defining a binarization order of the flags, which is known by both the encoder and the decoder.

[0237] The above examples have provided for encoding data of an image / video as motion vectors. However, the present invention is not limited to such embodiments. In one embodiment, the data to be encoded comprises image information and / or prediction residual information and / or prediction information. The image information here refers to the sample values of the original image (or the image to be encoded). The sample values can be samples of one or more color or other channels.

[0238] The information related to the selected layer is not necessarily a motion vector or a motion vector of a super-pixel. Additionally or alternatively, in some embodiments, the information comprises prediction information. The prediction information can comprise a reference index and / or a prediction mode. For example, the reference index can indicate which particular image from a reference picture set is applied for inter prediction. The index can be relative to the current picture in which the current block to be predicted is located. For example, the prediction mode can indicate whether a single or multiple reference frames and / or a combination of different predictions, e.g. inter-intra joint prediction, etc., are used.

[0239] However, when the data to be encoded is a motion vector field, an efficient motion vector field coding and reconstruction can be achieved. Figure 12 A respective general block scheme of a device that can perform encoding and decoding of motion fields is shown. At the encoding side, the motion information is obtained using some motion estimation or optical flow estimation module (unit) 1210. The input to the motion vector (optical flow) estimation is the current picture and one or more reference pictures (stored in a reference picture buffer). At the decoding side, the input to the motion vector (optical flow) estimation is the current picture and one or more reference pictures (stored in a reference picture buffer). Figure 12In the above embodiments, the image is referred to as "frame", a term sometimes used for images of a video. The optical flow estimation unit 1210 outputs optical flow 1215. In different implementations, the motion estimation unit can output motion information already having different spatial resolutions, e.g. for some NxN blocks or for each pixel of the original resolution, which can be referred to as dense optical flow. The motion vector information is intended to be transmitted (embedded into the bitstream 1250) to the decoding side and used for motion compensation. In order to obtain a motion compensated region, each pixel of the region should have a defined motion vector. Transmitting motion vector information for each pixel of the original resolution can be too costly. In order to reduce the indication overhead, a motion norming (or partitioning) module 1220 is used. A corresponding module 1270 at the decoding side performs the motion generation (dense) task to reconstruct the motion vector field 1275. The motion norming (or partitioning) module 1220 outputs motion information (e.g. motion vectors, and / or possibly reference images) and partitioning information. This information is added (encoded) into the bitstream.

[0240] In the present embodiment, the motion partitioning unit 1220 and the motion generation unit 1270 only contain down-sampling layers dwnspl and corresponding up-sampling layers up spl, as shown in Figure 13 The nearest-neighbor method can be used for down-sampling and up-sampling; average pooling can be used for down-sampling. Feature map data from layers of different spatial resolutions are selected by the encoder and transmitted in the bitstream as selected information 1120 together with partitioning information 1130 indicating how the decoder interprets and utilizes the selected information 1120. The motion partitioning (sparsification) net 1220 is shown in Figure 13 as a network 1310. Thus, the dense optical flow 1215 is inserted into the motion partitioning (sparsification) net 1310. The network 1310 comprises three down-sampling layers and a signal selection logic 1320 selecting the information to be included in the bitstream 1350. This functionality is similar to Figure 9 (as described in more generality).

[0241] In the above embodiments, the indication of information related to layers different from the output layer improves the scalability of the system. Such information can be information related to hidden layers. In the following, embodiments and examples are given which make use of the provided scalability and flexibility. Namely, some methods are provided on how to select layers and how to present information.

[0242] Some embodiments herein illustrate an image or video compression system using an autoencoder architecture comprising one or more dimension (or spatial resolution) reduction steps (implemented by layers merging down-sampling operations) in the encoding part. Together with the simplified (encoding) side, the reconstruction (decoding) side is learned, where the autoencoder tries to generate from the simplified encoding a representation as close as possible to its original input, which usually implies one or several resolution increase steps (implemented by layers comprising up-sampling operations) in the decoding side.

[0243] Here and after, under the action of an encoder, the encoding part of the autoencoder generates a latent signal representation included in a bitstream. For example, such an encoder is the above mentioned encoder 101 or 121. Under the action of a decoder, the generating part of the autoencoder perceives the latent signal representation obtained from the bitstream. For example, such a decoder is the above mentioned decoder 104 or 144.

[0244] As referred to above, the encoder selects a part (or parts) of the feature map information (selected information 1120) from different spatial resolution layers according to the signal selection logic 1100 and transmits the selected information 1120 in the bitstream 1150. The split information 1130 indicates from which layer and which part of the feature map of the corresponding layer the selected information is taken. Figure 11

[0245] According to one embodiment, the processing of the jth layer of the plurality of N concatenated layers comprises:

[0246] - determining a first cost resulting from the reconstruction of a part of a reconstructed image by using feature map elements output by said jth layer;

[0247] - determining a second cost resulting from the reconstruction of said part of said reconstructed image by using feature map elements output by said (j-1)th layer;

[0248] - if said first cost is higher than said second cost, selecting said (j-1)th layer and selecting information related to said part in said (j-1)th layer.

[0249] The selection of which layer to select can be performed according to a distortion or according to a distortion function. For example, for motion vector field coding, the reconstructed image (or image part) can be a motion compensated image (or image part).

[0250] ​To select the selected information in the present example implementation, the encoder includes a cost computation unit (module) that estimates the cost of transmitting the motion information from the particular resolution layer at the particular location. The cost is computed as the distortion resulting from the motion compensation with the selected motion vector, combined with an estimate of the number of bits required to transmit the motion information, multiplied by a Lagrange multiplier. That is, according to one embodiment, rate-distortion optimization (RDO) is performed.

[0251] That is, in some embodiments, the first cost and the second cost include an amount of data and / or distortion. For example, the amount of data includes an amount of data required to transmit data related to the selected layer. It can be motion information or other information. It can also be or include overhead resulting from residual coding. The distortion is computed by comparing the reconstructed image with the target image (the original image to be encoded or a portion of the original image to be encoded). Note that RDO is just one possibility. The present application is not limited to this approach. Moreover, complexity or other factors can be included in the cost function.

[0252] Figure 14 The first part of the cost computation is shown. Specifically, the cost computation (or estimation) unit 1400 obtains the optical flow L1 downsampled by the downsampling layer of the motion segmentation unit 1140. Then, the cost computation unit 1400 upsamples the optical flow 1415 to its original resolution, e.g., in this case, by upsampling by 4 in each of the two directions (x and y). Then, motion compensation 1420 is performed using the upsampled motion vectors output from 1410 and the reference image 1405 to obtain the motion of the compensated frame (image) or portion of the motion compensated frame (image) 1420. Then, the distortion 1430 is computed by comparing the motion compensated image (portion) 1420 with the target image 1408. For example, the target image 1408 can be the image to be encoded (the original image). In some example implementations, the comparison can be performed by computing the mean squared error (MSE) or the sum of absolute difference (SAD) between the target image 1408 and the motion compensated image 1420. However, other types of measures / metrics can also be used instead of or in addition, e.g., more advanced metrics for subjective perception, e.g., MS-SSIM or VMAF. The computed distortion 1430 is then provided to the cost computation module 1460.

[0253] Further, the rate estimation module 1440 computes an estimate of the amount of bits per motion vector. The rate estimation can include not only the bits used to signal the motion vector, but also the bits used to signal the segmentation information (in some embodiments). For example, the number of bits thus obtained can be normalized 1450 per pixel (feature map element). The resulting rate (amount of bits) is provided to the cost computation module 1460. To obtain the rate (amount of bits) estimate, for example, the motion information coding module is used (e.g., by performing the coding and recording the resulting amount of bits) or, in some simplified implementations, the length of the motion vector of its x or y component is used as a rough estimate. Another estimation technique can be applied. To account for the segmentation information, it can be evaluated by the segmentation information coding module (e.g., by generating and coding the segmentation information and counting the resulting number of bits) or, in a simpler implementation, by adding bits to the total amount of bits.

[0254] In the present example, the next step of the cost computation is the cost computation 1460, followed by a downsampling 1470 by 4 to reach the resolution of the corresponding downsampled layer of the motion segmentation unit 1100. Each point (image sample value) can only transmit one motion vector. Thus, the resulting cost tensor can have a corresponding size (dimension). Thus, the bit estimate can be normalized by the square of the downsampling filter shape (e.g., 4x4).

[0255] Then, using the Lagrange multiplier, the cost estimation unit 1460 computes the cost using the following formula:

[0256] Cost = D + λ * R, or

[0257] Cost = R + β * D

[0258] where D denotes the distortion (computed by 1430), R denotes the bit estimate (the rate estimate output by 1440 or 1450), and λ and β are the Lagrange multipliers. The downsampling 1470 outputs the cost tensor 1480. The Lagrange multipliers, as well as λ and β, can be obtained empirically, as known in the art.

[0259] This results in a tensor 1480 with a cost estimate for each location in the feature map (in this case, WxH locations for dense optical flow). It is noted that using sequential average pooling and upsampling with the nearest neighbor method results in an average motion vector in an N x N (e.g., 4x4) region, where N x N is the average pooling filter shape and the scaling factor of the upsampling operation. During upsampling with the nearest neighbor method, the value of the lower resolution layer is copied (repeated) in all points of the higher resolution layer corresponding to the filter shape. This corresponds to a translational motion model.

[0260] Various implementations of the cost selection unit are possible. For example, Figure 15 Another exemplary implementation is shown. In this example, with... Figure 14 The difference is that the motion vector field obtained after downsampling the dense optical flow by 4 in each of the x and y dimensions 1501 is not upsampled 1415. Instead, the motion vector field is directly provided to motion compensation 1510 and rate estimation 1540. Conversely, before motion compensation 1510 and distortion assessment 1530, the reference image 1505 and the target image 1505 can be downsampled 1515, 1518 to the corresponding resolution. This eliminates the need for performing... Figure 14 The initial motion field is upsampled to the original resolution by 1415, and no steps are performed. Figure 14 The final cost downsampling step 1470. This also makes bit normalization 1450 unnecessary. This implementation requires less memory to store tensors during processing, but may provide less accurate results. It should be noted that, to speed up or reduce the complexity of RDO, one can imagine downsampling dense optical flow as well as the reference and target images, even more than L1. However, the accuracy of such RDO may be further reduced.

[0261] Cost estimation units (1400, 1500) are applied to each downsampling layer of motion segmentation units (1220, 1310) to obtain costs with different motion vector averaging levels (different spatial resolutions). As a next step, signal selection logic 1100 uses the cost information from each downsampling layer to select motion information at different spatial resolutions. To enable signal selection logic 1100 to perform pairwise comparisons of costs from sequential (cascaded) downsampling layers, signal selection logic 1100 selects the minimum cost at each spatial location and propagates it to the next (in processing order) downsampling layer. Figure 16 An exemplary architecture of the signal selection unit 1600 is shown.

[0262] The dense optical flow 610 enters three downsampling layers downspl4, downspl2, and downspl2, similar to... Figure 11 The downsampling layer shown in the image. Figure 16 The signal selection logic 1600 in the middle is Figure 11 An exemplary implementation of the signal selection logic 1100 is shown below. Specifically, the LayerMv tensor 611 is a subsampled motion vector field (feature map) that enters the cost calculation unit 613. The LayerMv tensor 611 also enters the first layer information selection unit 614. If a selected motion vector exists on this (first) layer, the layer information selection unit 614 provides the selected motion vector to the bitstream. Its function will be further described below.

[0263] For example, the cost calculation unit 613 calculates the cost as described with reference to Figure 14 The cost calculation unit described for the cost calculation unit 1400. The cost calculation unit outputs a cost tensor, which is then downsampled by 2 to match the resolution of the second layer operation. After processing by the second downsampling layer downspl2, the LayerMV tensor 621 is provided to the next (third layer) layer and to the cost calculation unit 623 of the second layer. The cost calculation unit 623 operates in a similar way as the cost calculation unit 1400. As will be clear to the skilled person, the upsampling / downsampling by 4 in the example described with reference to Figure 14 differs from the example described with reference to

[0264] In order to make a pairwise comparison of the cost tensors from the cost calculation units 613 and 623, the cost tensor from the previous (first) downsampling layer has been downsampled (by 2) to the current resolution layer (second layer). Then, a pooling operation 625 is performed between the two cost tensors. That is, the pooling operation 625 keeps the lower cost in the cost tensor of each element. From the element index of the pooling operation result, the selection of the layer with the lower cost is captured. For example, if in one particular tensor element, the cost of the first tensor is lower than the cost of the corresponding element of the second tensor, the index is equal to 0, otherwise the index is equal to 1.

[0265] In order to ensure gradient propagation for training purposes, a soft arg max can be used to obtain the pooled index with gradient. If gradient propagation is not required, a regular pooling with index can be used. As a result 622 of the pooling operation 625, the index representing whether the motion vector from the current resolution layer or the previous resolution layer (LayerFlag tensor) and the corresponding downsampling layer of the motion segmentation unit (LayerMv tensor) is transmitted to the layer information selection unit 624 of the current (here: second) layer. The best pooled cost tensor is propagated to the next downsampling level (downspl2) and the operation is then repeated for the third layer.

[0266] In particular, the output LayerMv 621 of the second layer is further downsampled (downspl2) by the third layer, and the resulting motion vector field LayerMv 631 is provided to the cost computation unit 633 of the third layer. The computed cost tensor is compared element-wise 635 with the downsampled cost tensor propagated from the second layer and provided by the MinCost pooling unit 625. After the MinCost pooling 635 process, the index indicating whether the motion vector from the current (third) resolution layer or the previous (second) resolution layer was selected (LayerFlag tensor) and the index of the corresponding downsampled layer's motion vector from the motion partitioning unit (LayerMv tensor) are transmitted to the layer information selection unit 634 of the current (here: third) layer. In the present example, only three layers are shown as an example. However, in general, there can be more than three layers, wherein the other layers and signal selection logic of these layers have similar functionality as shown for the second and third layers.

[0267] To collect the pooling information from each spatial resolution layer, the next process is performed in reverse order - from lower to higher resolution layers using the layer information selection units 634, 624 and 614. First, a TakeFromPrev tensor of the same size as the lowest resolution layer (here: third layer) is initialized 601 to 0. Then, the same operation is repeated for the layers of different resolutions as follows. At positions where the value of the tensor (NOT TakeFromPrev) is equal to 1, the value of the LayerFlag tensor (in the current layer) is selected as partitioning information and transmitted in the bitstream. The (NOT TakeFromPrev) tensor is the element-wise negation of the TakeFromPrev tensor. Thus, in the third layer (here: the last layer), the (NOT TakeFromPrev) tensor sets all values to 1 (negation of the set 0 value 601). Thus, the partitioning information 1130 (LayerFlag) of the last (here: third) layer is always transmitted.

[0268] The TakeFromCurrent tensor is obtained using the logical operation TakeFromCurrent = (NOT TakeFromPrev) AND LayerFlag. The flags of this tensor TakeFromCurrent indicate whether the motion vector information is selected to be transmitted in the bitstream at the current resolution layer. The layer information selection unit (634, 624, 614) selects the motion vector information from the corresponding downsampled layer of the motion partitioning unit by taking the value of the LayerMv tensor, where the value of the TakeFromCurrent tensor is equal to 1. This information is transmitted in the bitstream as selected information 1120.

[0269] For the third (first in reverse order) processing layer corresponding to the lowest resolution, all flags will be transmitted because TakeFromPrev is initialized to 0, and then all values of (NOT TakeFromPrev) are equal to 1. For the last processing layer corresponding to the highest resolution layer, the LayerFlag flag does not need to be transmitted. It is assumed that for all positions that do not select motion information from the previous layer, these positions should select from the current or next (highest resolution) layer.

[0270] It is noted that the cost calculation as shown in Figure 16 is a parallelizable scheme that can run on a GPU / NPU. The scheme is also trainable because it is transmitting gradients and can be used in an end-to-end trainable video coding technology scheme.

[0271] It is noted that the reverse order processing is similar to the processing performed by the decoder when parsing the split information and the motion vector information, as shown when discussing the decoder functionality below.

[0272] Another exemplary implementation of the signal selection logic 1700 is shown in Figure 17 Compared to Figure 16 , the block diagram of Figure 17 introduces multiple coding options for the same resolution layer. This is illustrated by the options 1 to N in the first layer cost calculation unit 710. It is noted that typically one or more or all layers can include more options. That is, any of the cost calculation units 613, 623, 633 can provide more options. For example, these options can be one or more or all of the following: different reference pictures for motion estimation / compensation, uni-prediction, bi-prediction or multi-hypothesis prediction, different prediction methods (e.g., inter prediction or intra prediction), direct coding without prediction, multi-hypothesis prediction, presence or absence of residual information, quantization level of residuals, etc. In the cost calculation unit 710, a cost is calculated for each coding option. Then, the best option is selected using the minimum cost pooling 720. An indicator (e.g., index) 705 of the best selected option is transmitted to the layer information selection module 730, which then transmits the indicator BestOpt in the bitstream if the corresponding point of the current layer will be selected to transmit information. In the given example, options are shown only for the first layer, but it should be understood that similar option selection logic can be applied to other layers of different resolutions, or to all layers.

[0273] The above-described approach is also applicable to the splitting and transmission of logical information, such as flags or switches that control the image reconstruction process, as well as information that is to remain unchanged after decoding and identical to the encoding side. That is, instead of Figure 16The motion vector field (dense optical flow) processed in the exemplary implementation can be encoded in a similar manner, including segmentation, for any one or more other parameters. Parameters can be one or more of the following: indicators indicating different reference images, etc., used for motion estimation / compensation; unidirectional, bidirectional, or multi-hypothesis prediction indicators; different prediction methods, such as inter-frame or intra-frame prediction; direct decoding indicators without prediction; multi-hypothesis prediction; the presence or absence of residual information; the quantization level of the residual; parameters of the in-loop filter, etc.

[0274] Further modifications to the above embodiments and examples

[0275] According to the first modification, the downsampling layer of the motion segmentation unit 1310 and / or the upsampling layer of the motion generation unit 1360 include convolution operations. This in Figure 18 As shown in the image. Figure 18 As shown, with Figure 13 In contrast, the downsampling layer "dwnspl" and the upsampling layer "upspl" are replaced by the downsampling convolutional layer "conv↓" in the motion segmentation unit 1810, and by the upsampling convolutional layer "conv↑" in the motion generation unit 1860, respectively. Some advantages of the convolutional rescaling (downsampling, upsampling) layers are that they enable learnable downsampling and upsampling processes. For example, in cases involving motion-intensive information, it helps to find the optimal upsampling transform and thus reduces block artifacts caused by motion compensation using block-averaged motion vector information, as described in the embodiments and examples above. The same applies to texture restoration processes, such as for generating raw image intensity values ​​or prediction residuals processed by cascaded layers.

[0276] The above text Figure 18 In the example, all downsampling and upsampling layers are convolutional layers. However, this invention is not typically limited to this implementation. Generally, within the segmentation units (1310, 1810) and / or the generation units (1360, 1860), subsets (one or more) of the downsampling and corresponding upsampling operations can be implemented as convolutions.

[0277] Please recall that although the examples described herein are for dense optical flow / motion vector field processing, and thus refer to motion segmentation units (1310, 1810) and / or motion generation units (1360, 1860), the invention is not limited to such data / feature maps. Instead, in any embodiments and examples herein, any decoding parameter or even texture, such as samples of an image or prediction residuals (prediction errors), can be processed. Or, in addition to motion vector fields, any decoding parameter or even texture can be processed.

[0278] It is noted that, for example, an encoder with motion information averaged in down-sampling can be used in combination with a decoder comprising a convolutional up-sampling layer. Also, the purpose of an encoder with convolutional layers to find a better latent representation can be combined with a motion generating network (decoder) implementing a nearest-neighbor based up-sampling layer. There can be other combinations. That is, the up-sampling layer and the down-sampling layer are not necessarily of the same type.

[0279] According to a second modification, which can be combined with any of the above embodiments and examples (as well as the first modification), the processing of the network comprises one or more additional convolutional layers between the cascade of layers of different resolutions described above. For example, the motion segmentation unit 1310 and / or the motion generation unit 1360 further comprises one or more intermediate convolutional layers between some or all of the down-sampling layers and the up-sampling layers. This is illustrated in Figure 19 Figure 19 An exemplary implementation of such a motion segmentation network (module) 1910 and a motion generation network (module) 1860 is shown. It is noted that the terms "module" and "unit" are used interchangeably here to denote a functional unit. In this particular embodiment, the units 1910 and 1960 are more specifically network structures with a plurality of cascade layers.

[0280] For example, compared to the motion segmentation unit 1310, the motion segmentation unit 1910 has an additional convolutional layer "conv" before each down-sampling layer "conv↓" (which can also be other types of down-sampling). Also, compared to the motion generation unit 1360, the motion generation unit 1960 has an additional convolutional layer "conv" before each up-sampling layer "conv" (which can also be other types of up-sampling).

[0281] This can further reduce blocking artifacts caused by motion information sparsification and increase the generalization effect of finding a better latent representation. Similar to what was mentioned above with respect to the first modification, the encoders and decoders from the above different embodiments / modifications can be combined in one compression system. For example, there can be only an encoder with additional layers between the down-sampling layers and a decoder without such additional layers, or vice versa. Alternatively or in addition, there can be different numbers and positions of such additional layers at the encoder and the decoder.

[0282] According to a third modification, as shown in Figure 19 a direct connection of the input and output signals is provided. It is noted that the second and third modifications are independent, even though they are shown in the same figure here. They can be applied together or separately to the embodiments and examples described before as well as the other modifications. The direct connection is indicated by a dashed line.

[0283] ​In addition to the bottleneck information of the autoencoder's latent representation (output of the lowest resolution layer), in some embodiments, information from higher resolution layers is added to the bitstream. To optimize the signaling overhead, only part of the information from the different resolution layers is inserted into the bitstream, controlled by a signal selection logic. On the receiving (decoder) side, a corresponding signal feeding logic feeds the information from the bitstream to the layers of different spatial resolution, as described in detail below. Furthermore, information from the input signal before the down-sampling layers can be added to the bitstream, through which further variability and flexibility can be added. For example, the coding can be aligned to real object boundaries and segmentations with higher spatial resolution and adjusted according to the characteristics of the specific sequence.

[0284] According to a fourth modification, the shape of the down-sampling and up-sampling filters can be other than square, e.g. rectangular, with horizontal or vertical orientation, asymmetric shape or other more arbitrary shape by employing a mask operation. This helps to further increase the variability of the segmentation process to better capture real object boundaries. This modification is shown in Figure 20

[0285] Generally, when a layer is down-sampled, what is applied is that the input feature map is down-sampled using a first filter to obtain a first feature map and the input feature map is down-sampled using a second filter to obtain a second feature map. The cost calculation comprises determining a third cost resulting from reconstructing a part of the reconstructed image by using the first feature map and determining a fourth cost resulting from reconstructing a part of the reconstructed image by using the second feature map. Then, at the selection, if the third cost is smaller than the fourth cost, the first feature map is selected and if the third cost is larger than the fourth cost, the second feature map is selected. In this example, a selection from two filters is made. However, the present invention is not limited to two filters but can select in a similar way from a predefined number of filters, e.g. by estimating the cost of all selectable filters and minimizing the cost by selecting the filter.

[0286] The shape of the first filter and the second filter can be any one of a square, a horizontally oriented rectangle and a vertically oriented rectangle. However, the present invention is not limited to these shapes. Generally, any arbitrary filter shape can be designed. The filters can also comprise filters which can be defined with any desired shape. Such a shape can be represented by taking a mask, wherein the mask consists of flags, the mask representing an arbitrary filter shape and one of the first filter and the second filter (generally any selectable filter from the filter set) has the arbitrary filter shape.​

[0287] In an example implementation, to provide variability, the encoder further comprises a pooling between the cost tensors obtained by means of filters having mutually different shapes. An index of the selected filter shape is indicated in the bitstream as (part of) the partition information, similar to the motion vectors described above. For example, for selecting between a rectangular shape in the horizontal and vertical direction, a corresponding flag can be indicated in the bitstream. For example, with reference to Figure 17 The described method of selecting multiple encoding options can be used to select different filter shapes in the same resolution layer.

[0288] According to a fifth modification, a motion model from a predefined set of different motion models can be selected in the same resolution layer. In the previous embodiments, the specific case of down- and / or up-sampling filters has been described. In this case, the motion information can be averaged over the squares representing a translational motion model. In the fifth modification, other motion models can be employed in addition to the translational motion model. Such other motion models can include one or more of:

[0289] - an affine motion model,

[0290] - a higher order motion model, or

[0291] - a CNN layer specifically trained to represent a particular motion model such as scaling, rotation, affine, perspective, etc.

[0292] In an example implementation of the fifth modification, the autoencoder further comprises a set of CNN layers and / or "handcrafted" layers representing motion models other than the translational motion model. Such an autoencoder (and decoder) is shown in Figure 21 In Figure 21 , the autoencoder comprises layers provided at the encoding and decoding end comprising a set of filters representing a "convolution fit set".

[0293] For example, at each spatial layer, the encoder selects the appropriate filter from the set of filters corresponding to a particular motion model and inserts an indicator into the bitstream. At the receiving end, the signal feeding logic interprets the indicator and performs the convolution at the particular layer using the corresponding filter from the set.

[0294] The example methods described above use motion information, in particular motion vectors, as an example input for encoding. Again, it is noted that these methods are also applicable to the compression of different types of image or video information, such as direct image sample values, prediction residual information, intra- and inter-prediction parameters, etc.

[0295] According to a sixth modification, the RDO illustrated above with reference to Figure 16 or Figure 17 can be applied to a conventional block-based codec.

[0296] Traditional video coding methods, e.g. state-of-the-art video coding standards such as AVC, HEVC, VVC or EVC, use a block-based coding concept according to which the picture is recursively divided into blocks of square or rectangular shape. For these blocks, signal reconstruction parameters are estimated or evaluated at the encoder side and transmitted to the decoder in the bitstream. Typically, the goal of the encoder is to find the best set of reconstruction parameters for the blocks of the picture in terms of rate-distortion cost, trying to maximize the reconstruction quality (i.e. to minimize the distortion of the original picture) and to minimize the amount of bits required to transmit the reconstruction process parameters. The task of parameter selection (or coding mode decision) is a complex and resource demanding task and a major source of encoder complexity. For example, in real-time applications, the encoder can sacrifice the quality of the mode decision which in turn affects the quality of the reconstructed signal. Optimizing the mode decision process has always been a desirable technical improvement.

[0297] One of the coding mode decisions is the decision whether to split the current block (or Coding Unit, CU) into multiple blocks according to a partitioning method.

[0298] According to a sixth modification, the motion partitioning unit 1310 (or 1810) described above is adapted to make the partitioning mode decision based on a minimization of a cost (e.g. a rate-distortion optimization criterion). Figure 22 An example of such an optimization is shown.

[0299] As Figure 22 shown, the block partitioning structure is used to represent information of different spatial resolutions instead of down-sampled layers. For each block of a given size NxN (considering square blocks) of the picture or part of the picture, the cost computation unit computes a distortion tensor and further down-samples the resolution by a factor of 16 (to match the original resolution). In Figure 22In the given example, the first block size is 16x16 (e.g. down-sampling is performed by an average pooling operation) to obtain a tensor, each element of which represents the average distortion in each 16x16 block. In the first layer, the image is split into 16x16 blocks 2201 at the initial highest resolution. In the second layer, the resolution is reduced such that the block size 2202 in the image is 32x32 (corresponding to four blocks connected from the previous layer). In the third layer, the resolution is again reduced such that the block size 2203 is 64x64 (corresponding to four blocks connected from the previous layer). It is noted that in this case, connecting four blocks from the previous layer can be seen as a sub-sampling of the block-related information. This is because in the first layer, block-related information is provided for each 16x16 block, whereas in the second layer, block-related information is provided only for 32x32 blocks, i.e. four times less parameters are provided. Similarly, in the third layer, block-related information is provided only for 64x64 blocks, i.e. 16 times less parameters than in the first layer and four times less parameters than in the second layer.

[0300] In this context, block-related information is any information that is coded per block, e.g. prediction mode; prediction mode specific information, e.g. motion vector, prediction direction, reference picture, etc.; filtering parameters; quantization parameters; transform parameters or other settings that can be changed at block (coding unit) level.

[0301] Then, the cost computation units 2211, 2212 and 2213 of the respective first, second and third layers compute the cost from the block reconstruction parameters for the respective block sizes 2201, 2202 and 2203 and from the input image of size WxH.

[0302] The output cost tensor is obtained as the average distortion in each block and combined with the bit estimate value needed to transmit the coding parameters of the N x N (e.g. 16x16 for the first layer) block using a Lagrange multiplier. Figure 23 An exemplary structure of a cost computation unit 2300 (which can correspond to each or any of the cost computation units 2211, 2212 and 2213) for a block N x N is shown.

[0303] Figure 23An exemplary block diagram of a cost computation unit 2300 for a general block size of NxN, 230x, is shown. The cost computation unit 2300 takes 2310 the block reconstruction parameters (block dependent parameters) associated with a block of size NxN. This taking can correspond to taking the parameters (parameter values) from a memory or the like. For example, the block dependent parameters can be a specific prediction mode, e.g. an inter prediction mode. In unit 2310, the block reconstruction parameters are taken, in reconstruction unit 2320, these parameters are used to reconstruct a part of the image (in this example, all blocks are reconstructed using the inter prediction mode). A distortion computation unit 2330 computes the distortion of the reconstructed part of the image by comparing it to the corresponding part of the target image, which can be the original image to be encoded. Since the distortion can be computed for each sample, in order to get the distortion on a block basis, a down-sampling 2340 of the distortion can be performed (one value per NxN block). In the lower branch, the rate or number of bits needed to code the image is estimated 2360. In particular, the bit estimation unit 2360 can estimate the number of bits to be indicated per block of size NxN. For example, the number of bits per block needed for the inter prediction mode can be computed. With the estimated distortion and the bit amount (or rate), the cost can be computed 2350, e.g. according to the Lagrangian optimization mentioned above. The output is one cost tensor.

[0304] It is noted that throughout the description, the term "tensor" can be a matrix if only a 2D image of the samples is observed, e.g. a grayscale image. However, the image can have multiple channels, like color or depth channels, and thus the output can have more dimensions as well. The feature maps can also exceed 2 or 3 dimensions in general.

[0305] For the first level (with a 16x16 block granularity) and the next level of quad-tree partitioning into blocks of size 32x32 samples, the same cost evaluation procedure is performed. To decide whether it is better to use one 32x32 block as reconstruction parameter (block dependent parameter) or four 16x16 blocks, the cost tensor evaluated for the 16x16 blocks is down-sampled by a factor of 2 (see Figure 22). Then, a MinCost pooling operation 2222 provides the best decision for each 32x32 block. The index of the pooled cost is passed to the layer information selection unit 2232 for transmission in the bitstream as split_flags. The reconstruction parameters blk_rec_params of the best selected block according to the pooling index are also passed to the layer information selection unit 2231. The pooled cost tensor is further passed (down-sampling by 2) to the next quadtree aggregation level of blocks of size 64x64, i.e., to the MinCost pooling 2223. The MinCost pooling 2223 also receives the costs computed in the cost computation unit 2213 for the 64x64 block resolution 2203. It passes the index of the pooled cost to the layer information selection unit 2233 as split_flags for indication in the bitstream. The reconstruction parameters blk_rec_params of the best selected block according to the pooling index are also passed to the layer information selection unit 2233.

[0306] To collect the pooling information from each block aggregation level, the layer information selection units 2233, 2232 and 2231 are used in the manner described above with reference to Figure 16 The processing is performed in the reverse order, from the higher (in this example, the highest) aggregation level (64x64 samples) to the lower (in this example, the lowest) aggregation level (16x16 samples), in the manner described above with reference to

[0307] This results in a bitstream encoding the quadtree partitioning obtained by optimization, as well as the further coding parameters of the encoded values and possible resulting sub-blocks (blocks). The above-described approach can make decisions on the partitioning flags for the block split. To obtain the reconstruction parameters for each block, a conventional approach based on evaluating each or a subset of the possible coding modes can be used.

[0308] Figure 24 An example of a seventh modification is shown. The seventh modification is an evolution of the sixth modification described above with reference to Figure 22 and Figure 23 The seventh modification represents a scheme in which the evaluation of coding modes is incorporated into the design. Specifically, as shown in the figure, the cost computation unit 710 can evaluate N options. It is noted that the term "N" is here a placeholder for some integer. The "N" denoting the number of options is not necessarily the same as the "N" in "N x N" denoting the general block size. In the cost computation unit 710, for the same level of sub-block partitioning, e.g., for blocks of size 16x16 samples (as in the first layer), the encoder iterates over all possible (or a limited set of) coding modes for each block.

[0309] One needs to consider N options to code each 16x16 block, denoted as blk_rec_params 0, blk_rec_params 1,... blk_rec_params N. For example, the parameter combination blk_rec_params k (k is an integer from 0 to N) can be a combination of certain prediction modes (e.g. inter or intra prediction), certain transforms (e.g. DCT or KLT), certain filter orders or filter coefficient sets (among predefined filters), etc. In some implementations, if only one parameter is optimized, blk_rec_params k can be the value k of the single parameter. It is understood by those skilled in the art that any one or more parameters can be optimized by examining their usage cost.

[0310] For each given block reconstruction parameter set (blk_rec_params k), the cost computation unit 2410 computes a tensor representing the cost of each block. Then, using a MinCost pooling 2420, the best coding mode for each block is selected and transmitted to the layer information selection unit 2430. The best-pooled cost tensor is further downsampled by a factor of 2 and transmitted to the next quadtree aggregation level (in this example, the second layer corresponds to aggregation with block size 32x32). Then, the split (partition) decision is made in the same way as the sixth modification described above. In Figure 24 In the sixth modification, options 0...N are only computed in the first layer (aggregation level 16x16). However, the present application is not limited to this approach. On the contrary, the evaluation of options 0...N can be performed at each aggregation level.

[0311] For example, at the next quadtree aggregation level (32x32, 64x64), the encoder evaluates (by computing the cost in the corresponding cost unit) and pools (by the corresponding MinCost pooling unit) the best coding mode for each block (not depicted in the figure for ease of understanding) compared to the previous aggregation level. The decision about the best mode and the corresponding reconstruction parameter set is provided to the layer information selection unit (e.g. the layer information selection unit 2430 shown for the first layer). To collect the pooling information from each block aggregation level, the layer information selection unit is used to perform the processing in the reverse order from higher aggregation levels (64x64) to lower aggregation levels (16x16) in the same way as described in the sixth modification.

[0312] Different block shapes can be used to represent more advanced partitioning methods, e.g. binary tree, ternary tree, asymmetric and geometric partitioning. Figure 25 This partitioning of blocks is illustrated by way of example. Namely, the optimization is not necessarily performed only on different block sizes, it can also be performed for different partitioning types (e.g. by corresponding options). Figure 25 The following examples are shown:

[0313] - quad-tree partitioning 2510: in quad-tree partitioning, a block is split (partitioned) into four blocks of equal size.

[0314] - (symmetric) binary tree partitioning 2520: in symmetric binary tree partitioning, a block is split into two blocks of equal size. The split can be vertical or horizontal. Vertical or horizontal is an additional parameter of the split.

[0315] - (asymmetric) binary tree partitioning 2530: in asymmetric binary tree partitioning, a block is split into two blocks of different size. The size ratio can be fixed (to save overhead of indication) or variable (in which case,

[0316] Some of the ratio options can also be optimized, i.e. configurable.

[0317] - ternary tree partitioning 2540: in ternary tree partitioning, a block is split into three sub-blocks by two vertical or two horizontal lines. Vertical or horizontal is an additional parameter of the split.

[0318] The invention is not limited to these exemplary partitioning modes. A ternary tree partitioning or any other type of partitioning can be used.

[0319] In a seventh modification, a hybrid architecture, which is applicable to popular video coding standards, is supported and empowered by a powerful (neural) network based approach. The technical advantages of the described approach can provide a highly parallelizable GPU / NPU friendly solution, which can help to speed up the computation of the mode decision process. The described approach can enable global image optimization, as multiple blocks are considered at the same decision level, and, thus, include a learnable part to speed up the decision, e.g. to evaluate the bit amount required for the coding of the reconstruction parameters.

[0320] In summary, according to the sixth or seventh modification, the processing of the cascaded layer structure comprises processing data of the same image in different layers in relation to blocks being partitioned, i.e. divided / split, into blocks having different block sizes and / or shapes, respectively. The selection of the layers comprises selecting the layers according to a computation cost of a predetermined set of coding modes.

[0321] That is, different layers can process the image data in different block sizes. Thus, the cascaded layers comprise at least two layers processing different block sizes from each other. When referring to a “block” here, it is meant a unit, i.e. a portion of the image for which coding is performed. The block can also be referred to as a coding unit or processing unit, etc.

[0322] The predetermined set of coding modes corresponds to a combination of coding parameter values. Different block sizes can be evaluated under one set of coding modes, i.e. one or more combinations of values of coding parameters. Alternatively, the evaluation can comprise various combinations of block sizes and partitioning shapes, e.g.,Figure 25 (Combinations of [variables]). However, the invention is not limited thereto, and, as specifically mentioned in the seventh modification, there may be several predetermined sets of decoding modes (combinations of decoding parameter values), which may also include, for example, decoding modes such as intra / inter-frame prediction types for each block, intra-frame prediction modes, residual skipping, residual data, etc.

[0323] For example, the process includes determining the cost (a combination of decoding parameter values) of different sets of decoding modes for at least one layer, and selecting one of the sets of decoding modes based on the determined cost. Figure 24 The diagram illustrates the case where only the first layer performs this selection. However, this is not a limitation. Quickly, each cost calculation unit can have the same structure as the first cost calculation unit 2410, including options 0…N. This is not shown in the diagram for simplicity.

[0324] As mentioned above, this is a GPU-friendly RDO that can be executed by the codec and selects the optimal decoding mode for each block. Figure 24 In this approach, the input image (picture) is identical across all layers. However, the decoding (computational cost) of the image is performed in each layer with a different block size. Besides the block size, other decoding parameters can be selected based on RDO testing and for one or more block sizes.

[0325] Specifically, in these modifications, the indication of data associated with the selected layer includes the selected set of decoding modes (e.g., blk_rec_params).

[0326] In summary, in some embodiments, an encoder may be provided whose structure corresponds to a neural network autoencoder for decoding video or image information. Such an encoder can be used to: analyze input image or video information using a neural network comprising layers of different spatial resolutions; transmit a latent representation in the bitstream corresponding to the output of the lowest resolution layer; and transmit outputs other than the lowest resolution layer in the bitstream.

[0327] decoding

[0328] The encoder described above provides a bitstream that includes feature data and / or segmentation information for a selected layer. Accordingly, the decoder performs multi-level processing on the data received from the bitstream. Furthermore, the selected layer receives additional (direct) input from the bitstream. This input may be some feature data information and / or segmentation information.

[0329] Accordingly, embodiments described below focus on information related to a selected layer as feature data. Other described embodiments focus on information related to a selected layer as segmentation information. Hybrid embodiments of bitstream carrying and layer processing feature data and segmentation information also exist.

[0330] As a simple example, a decoder of a neural network autoencoder can be provided for video or image information coding. The decoder can be configured to read a latent representation corresponding to a lower resolution layer input from a bitstream; obtain layer input information from corresponding information read from the bitstream other than the lower resolution layer; obtain a combined input for the layer from the output of the previous layer and the layer information obtained from the bitstream; feed the combined input to the layer; synthesize an image from the output of the layer.

[0331] Here, the term "lower resolution" refers to a layer that processes a feature map having a lower resolution, e.g., a feature map of a latent space provided from a bitstream. The lower resolution can in fact be the lowest resolution of the network.

[0332] The decoder can also be configured to obtain segmentation information from corresponding information read from the bitstream; obtain a combined input for the layer from the segmentation information. The segmentation information can be a quadtree, a binary tree, or a ternary tree data structure, or a combination thereof. The layer input information can correspond to, for example, motion information, image information, and / or prediction residual information, etc.

[0333] In some examples, a hyperprior neural network is used to decode information corresponding to layer input information obtained from a bitstream. A hyperprior neural network can be used to decode information corresponding to segmentation information obtained from a bitstream.

[0334] The decoder can be readily applied to the decoding of motion vectors, e.g., motion vector fields or optical flow. Some of these motion vectors can be similar or related. For example, in a video showing an object moving on a constant background, there can be two groups of similar motion vectors. The first group of motion vectors can be vectors used to predict pixels of the object, and the second group can be vectors used to predict pixels of the background. Thus, instead of indicating all motion vectors in the encoded data, groups of motion vectors can be indicated to reduce the amount of data representing the encoded video. This can facilitate indicating a representation of a motion vector field that requires a smaller amount of data.

[0335] Figure 9 A bitstream 930 generated by the encoder described above is shown as being received at the decoding end. At the decoding end, the decoder portion of the system 900 includes signal feeding logic 940, which in some embodiments interprets segmentation information obtained from the bitstream 930. From the segmentation information, the signal feeding logic 940 identifies the particular (selected) layer, spatial size (resolution), and location of a feature map portion for which corresponding selected information (also obtained from the bitstream) should be placed.

[0336] It is noted that, in some embodiments, the segmentation information is not necessarily processed by the cascaded network. The segmentation information can be provided independently or derived from other parameters in the bitstream. In other embodiments, the feature data is not necessarily processed in the cascaded network, but the segmentation information is processed in the cascaded network. Thus, the two parts of “decoding using feature information” and “decoding using segmentation information” describe examples of these embodiments, as well as combinations of these embodiments.

[0337] It is noted that, for the two-part embodiments, the above-mentioned encoding-side modifications (first to seventh modifications) are applied to the decoding side accordingly. For ease of understanding, other features of the modifications are not repeated in the two parts. However, they can be applied to the decoding methods of the two parts alternatively or in combination, as clear to the skilled person.

[0338] Decoding using feature information

[0339] In embodiments of the present application, a method of decoding data for image or video processing from a bitstream is provided, as shown in Figure 33 Accordingly, an apparatus of decoding data for image or video processing from a bitstream is provided. The apparatus can comprise processing circuitry for performing the steps of the method.

[0340] The method comprises obtaining 3310 from the bitstream two or more sets of feature map elements, wherein each set of feature map elements is related to a (respective) feature map. The obtaining can be performed by parsing the bitstream. In some example implementations, the bitstream parsing can further comprise entropy decoding. The present application is not limited to any particular way of obtaining data from the bitstream.

[0341] The method further comprises the step of inputting 3320 each of the two or more sets of feature map elements into two or more feature map processing layers of a plurality of cascaded layers, respectively.

[0342] The cascaded layers can form part of a processing network. In the present application, the term “cascaded” means that the output of one layer is later processed by another layer. The cascaded layers do not have to be immediately adjacent (where the output of one cascaded layer directly enters the input of the second cascaded layer). When referring to Figure 9 The data from the bitstream 930 is input into the signal feeding logic 940, which feeds the sets of feature map elements to the appropriate layers (indicated by the arrows) 953, 952 and / or 951. For example, the first set of feature elements is inserted into the first layer 953 (the first in the processing sequence), the second set of feature elements is inserted into the third layer 951. It is not necessary to insert a set in the second layer. The number and position (within the processing sequence) of the layers can vary, the present application is not limited to any particular number and position.

[0343] The method further comprises obtaining 3330 the decoded data for image or video processing as a result of the processing of the plurality of concatenated layers. For example, the first set is the set of latent feature map elements processed by all layers of the network. The second set is an additional set provided to another layer. When referring to Figure 9 the decoded data 911 is obtained after processing the first set by the three layers 953, 952 and 951 (in this order).

[0344] In an example implementation, the feature maps are processed in each of the two or more feature map processing layers, wherein the resolution of the feature maps processed in the two or more feature map processing layers, respectively, is different. For example, the resolution of the first feature map processed by the first layer is different from the resolution of the second feature map processed by the second layer.

[0345] In particular, the processing of the feature maps in the two or more feature map processing layers comprises up-sampling. Figure 9 It is shown that the decoding part comprises a network of three (directly) concatenated up-sampling layers 953, 952 and 951.

[0346] In an example implementation, the decoder comprises only up-sampling layers of different spatial resolution, and a nearest-neighbor approach is used for up-sampling. The nearest-neighbor approach repeats the value of a lower resolution in a higher resolution region corresponding to the given shape. For example, if one lower resolution element corresponds to four higher resolution elements, the value of one element is repeated four times in the higher resolution region. In this case, the term "corresponds" refers to describing the same region in the highest resolution data (initial feature map, initial data). This way of up-sampling facilitates the transfer of information from a lower resolution layer to a higher resolution layer without modification, which can be suitable for some kinds of data, e.g. logical flag or indicator information, or information that is intended to be kept the same as obtained at the encoding side, e.g. some modification of convolutional layers. One example of such data is prediction information, e.g. motion information that can include motion vectors estimated at the encoding side, a reference index indicating which specific picture from a reference picture set should be used, a prediction mode indicating whether a single reference frame or multiple reference frames are used, or a combination of different predictions (like inter-intra joint prediction), the presence or absence of residual information, etc.

[0347] However, the invention is not limited to up-sampling performed by the nearest-neighbor approach. Alternatively, the up-sampling can be performed by applying some interpolation or extrapolation, or by applying a convolution, etc. These methods can be particularly suitable for up-sampling data that is expected to have smooth characteristics (e.g. motion vectors or residuals or other sample-dependent data).

[0348] In Figure 9In some implementations, the encoder (e.g., symbols 911-920) and the decoder (e.g., symbols 940-951) have the same number of down-sampling layers and up-sampling layers, respectively, nearest-neighbor method can be used for up-sampling, and average pooling can be used for down-sampling. In some other possible implementations, another pooling method, e.g., max-pooling, can be used.

[0349] As has been exemplified in several encoder embodiments, the data for image or video processing can include a motion vector field. For example, Figure 12 An encoding end and a decoding end are shown. At the decoding end, the bitstream 1250 is parsed, and the motion information 1260 (which can have partitioning information to be discussed later) is obtained from the bitstream. The obtained motion information is then provided to a motion generation network 1270. The motion generation network can increase the resolution of the motion information, i.e., dense motion information. The reconstructed motion vector field (e.g., dense optical flow) 1275 is then provided to a motion compensation unit 1280. The motion compensation unit 1280 uses the reconstructed motion vector field to obtain predicted image / video data from a reference frame, and reconstructs a motion compensated frame from the reference frame (e.g., by adding the decoded residual, as in Figure 5A As shown in, the decoder part of the encoder, or in Figure 7B In, the reconstruction unit 314).

[0350] Figure 13 A decoding end motion generation (dense) network 1360 is also shown. The network 1360 includes a signal feeding logic 1370, which functions similarly to the signal feeding logic 940 of Figure 9 , and includes three up-sampling (processing) layers. The main difference from the embodiments described above with reference to Figure 9 is that, in Figure 13 , the network 1360 is dedicated to motion vector information processing, and outputs a motion vector field.

[0351] As mentioned above, according to one embodiment, the method further includes obtaining, from the bitstream, partitioning information related to the two or more layers. Then, feature map elements are obtained from the bitstream according to the partitioning information. According to the partitioning information, the set of feature map elements are respectively input into the two or more feature map processing layers. Some detailed examples related to the use of partitioning information in parsing and processing are provided below in the section related to decoding using partitioning information. For example, Figure 28 and Figure 29 Very specific (and merely exemplary) layer processing options are provided.

[0352] In some embodiments, the plurality of cascaded layers further comprises a plurality of segmentation information processing layers. The method further comprises processing the segmentation information in the plurality of segmentation information processing layers. For example, processing the segmentation information in at least one of the plurality of segmentation information processing layers comprises upsampling. In some embodiments, such upsampling of the segmentation information and / or the upsampling of the feature maps comprises nearest-neighbor upsampling. Generally, the upsampling applied to the feature map information and the upsampling applied to the segmentation information can be different. Moreover, the upsampling within the same network can be different, so one network (segmentation information processing or feature map processing) can comprise different types of upsampling layers. For example, in Figure 20 or Figure 21 Such examples are shown in

[0353] According to exemplary implementations, the upsampling of the segmentation information and / or the upsampling of the feature maps comprises a (transposed) convolution. This corresponds to the first modification of the encoder discussed above. Figure 18 A motion generation unit 1869 is shown at the decoding side, comprising a convolution operation "conv " instead of nearest-neighbor upsampling. The motion generation unit can perform a learnable upsampling process, e.g. in case of motion information dense cases, helping to find the best upsampling transform, and can reduce blockiness artifacts caused by motion compensation by using block average motion vector information as described above with reference to the encoder. The same applies to the texture restoration process, e.g. for original image intensity value or prediction residual generation. The motion generation unit 1869 further comprises signal feeding logic, the function of which is the same as the signal feeding logic 940 of Figure 9 and the signal feeding logic 1370 of Figure 13 .

[0354] Figure 30 A block diagram of layer processing of an exemplary decoding side according to the first modification is shown. Specifically, the bitstream 3030 is parsed, and signal feeding logic 3040 (the function of which is the same as the function of the signal feeding logic 940 or 1370) provides selection instructions to a convolution upsampling filter 300. In some embodiments, the convolution filter can be selected from a set of N filters, denoted as filter 1 to N. The filter can be selected according to information denoting the selected filter and parsed from the bitstream. The indication of the selected filter can be provided by the encoder (generated and inserted into the bitstream) according to RDO or the like optimization method. Specifically, the filter can be selected according to Figure 17 or Figure 24RDO (treating the filter size / shape / order as one of the options to optimize, i.e. a coding parameter) is exemplarily illustrated. However, the present application is not limited thereto and, in general, the filter can be derived from other indication parameters (e.g. coding mode, interpolation direction, etc.).

[0355] In summary, the signal feeding logic controls the input of different layers with different filter shapes and selectively bypasses the output to the next layer according to the partition information and motion information obtained from the bitstream. The convolutional filter unit 3000 corresponds to the convolution performed by one layer. Several such convolutional pooling filters can be cascaded, like Figure 18 It is noted that the present application is not limited to variable or trainable filter settings. In general, the convolutional upsampling can also be performed with fixed convolution operations.

[0356] Aspects of the present embodiment can be combined with aspects of other embodiments. For example, an encoder with motion information averaged in the down-sampling layers can be used in combination with a decoder comprising convolutional upsampling layers. The purpose of an encoder with convolutional layers is to find a better latent representation, which can be combined with a motion generation network comprising nearest-neighbor based upsampling layers. There can be other combinations. That is, the implementation of the encoder and the decoder does not have to be symmetric.

[0357] Figure 32A Two examples of reconstruction applying the nearest-neighbor approach are shown. Specifically, example 1 shows the case where the partition information of the lowest resolution layer has a value of the flag set (1). Accordingly, the motion information represents one motion vector. Since the motion vector is already represented in the lowest resolution layer, there are no other motion vectors and no other partition information in the bitstream. The network generates the motion vector field with higher resolution (2x2) and the motion vector field with highest resolution (4x4) from one indicated motion vector by copying the indicated motion vector during nearest-neighbor upsampling. The 4x4 area results in all 16 motion vectors being identical and equal to the indicated motion vector.

[0358] Figure 32B Two examples of reconstruction applying the convolutional layer based approach are shown. Example 1 has the same input as example 1 of Figure 32A Specifically, the partition information of the lowest resolution layer has a value of the flag set (1). Accordingly, the motion information represents one motion vector. However, after applying the convolutional layer (possibly trained), the motion vectors in the higher layers and the highest layer are not identical to the indicated motion vector but rather than just copying one motion vector.

[0359] Similarly, Figure 32AExample 2 in shows the partition information of 0 in the lowest resolution layer, and the partition information 0101 of the next (higher resolution) layer. Accordingly, the two motion vectors for the locations represented by the partition information are indicated in the bitstream as motion information. These are shown in the middle layer. From the bottom layer, it can be seen that the indicated motion vectors are copied, each motion vector is copied four times to include the highest resolution region. The remaining 8 motion vectors of the highest resolution (bottom) layer are indicated in the bitstream. Figure 32B Example 2 of applies convolution instead of nearest neighbor copying. The motion vectors are no longer copied. Figure 32A The transition between the copied motion vectors in is now smoother, reducing block artifacts.

[0360] Similar to the second modification of the encoder discussed above, at the decoding end, the multiple cascaded layers include convolution layers without upsampling between layers with different resolutions. It is noted that the encoder and the decoder are not necessarily symmetrical in this regard: the encoder can have these additional layers while the decoder does not, or vice versa. Of course, the encoder and the decoder can also be designed symmetrically, with additional layers between the corresponding down- and up-sampling layers of the encoder and the decoder.

[0361] Regarding the combination of partition information processing and feature map processing, the feature map elements are obtained from the bitstream in accordance with the processed partition information processed by at least one of the multiple partition information processing layers. The partition layers can parse and interpret the partition information, as described in more detail in the section on decoding using partition information below. It is noted that the embodiments and examples described in this section can be used in combination with the embodiments in this chapter. In particular, reference is made to Figure 26 to Figure 32B The layer processing of the partition information described can also be performed in combination with the feature map processing described herein.

[0362] For example, inputting each of the two or more sets of feature map elements into the two or more feature map processing layers is performed according to the processed partition information processed according to at least one of the plurality of partition information processing layers. The obtained partition information is represented by a set of syntax elements, where the position of an element in the set of syntax elements indicates which feature map element position the syntax element relates to. For example, the set of syntax elements is a part of a bitstream, which can be binarized using fixed codes, entropy codes (e.g. variable length codes or arithmetic codes), any of which can be context adaptive. The present invention is not limited to any particular form of coding of the bitstream if the present invention has a predefined structure known at both the encoding and the decoding side. In this way, parsing and processing the partition information and the feature map information can be performed in association. For example, for each syntax element, the processing of the feature map comprises: (i) when the syntax element has a first value, parsing an element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream, (ii) otherwise (or more generally, when the syntax element has a second value), not parsing an element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream. The syntax elements can be binarized flags, which are orderedly arranged into the bitstream at the encoding side and parsed from the decoder in the correct order by the particular layer structure of the processing network.

[0363] It is noted that options (i) and (ii) can also be provided for syntax elements that are not binarized. In this case, the first value refers to parsing and the second value refers to not parsing. The syntax elements can take some other values than the first and second values. These can also result in parsing or not parsing, or can represent a particular type of parsing, etc. The number of parsed feature map elements can correspond to the number of syntax elements that are equal to the first value.

[0364] According to an example implementation, the processing of the feature map by each of the plurality of (N) feature map processing layers, 1 < j < N, further comprises: parsing a partition information element of the j-th feature map processing layer from the bitstream; obtaining the feature map processed by the preceding feature map processing layer, and parsing a feature map element from the bitstream and associating the parsed feature map element with the obtained feature map, where the position of the feature map element in the processed feature map is indicated by the parsed partition information element, said partition information being processed by the preceding partition information processing layer. For example, the association can be a replacement of the previously processed feature map element, or a combination, e.g. an addition, a subtraction or a multiplication. Some example implementations are provided below. The parsing can depend on the previously processed partition information, which can provide a very compact and efficient syntax.

[0365] For example, the method can comprise: parsing the elements of the feature map from the bitstream when the syntax element has a first value; not parsing the elements of the feature map from the bitstream when the syntax element has a second value or the partition information processed by the previous partition information processing layer has a first value. That is, if the relevant part is parsed at the previous layer, no parsing is performed. For example, the syntax element parsed from the bitstream representing the partition information is a binarization flag. As mentioned above, it is beneficial that the processed partition information is represented by a set of binarization flags. The set of binarization flags is a sequence of binarization flags each having a value of 1 or 0 (corresponding to the first and second values mentioned above).

[0366] In some embodiments, the upsampling of the partition information in each partition information processing layer j further comprises, for each p-th position in the obtained feature map represented by the input partition information, determining, as the upsampled partition information, an indication for a feature map position comprised in the same region in the reconstructed image as the p-th position. This way, a spatial relationship between the reconstructed image (or the reconstructed feature map or the data in general), the positions in the sub-sampled feature map and the corresponding partition flags is provided.

[0367] As mentioned above, and in the encoder embodiments, the data for image or video processing can comprise image data (e.g. image samples) and / or prediction residual data and / or prediction information data. When referring to “residuals” in the present disclosure, it is noted that these residuals can be pixel domain residuals or transform (spectral) coefficients (i.e. transform residuals, residuals represented in a domain different from the sample / pixel domain).

[0368] Similar to the fourth modification described above for the encoding side, according to an exemplary implementation, the filter is used for upsampling of the feature map and the shape of the filter is any one of a square, a horizontal rectangle and a vertical rectangle. It is noted that the shape of the filter can be similar to the patch shape as shown in Fig. 6. Figure 25

[0369] Figure 20 An exemplary decoder side layer processing is shown in Fig. 6. The operation generation network (unit) 2060 comprises signal feeding logic and one or more (here two) upsampling layers using filters (upsampling filters) which can be selected from a predetermined or predefined set of filters. The selection can be performed at the encoding side, e.g. by RDO or other settings, and indicated in the bitstream. At the decoding side, the indication of the filter selection is parsed from the bitstream and applied. Alternatively, the filter can be selected at the decoding side without being indicated by other coding parameters derived from the bitstream. These parameters can be any parameters related to the content, e.g. prediction type, direction, motion information, residual, loop filtering characteristics, etc.

[0370] Figure 31 ​A block diagram of an upsampling filter unit 3100 that supports selecting one filter among N filters 1 to N is shown. The indication of the filter selection can directly include an index of one filter among the N filters. The index can include filter direction, filter order, filter shape and / or coefficients. At the decoding side, signal feeding logic interprets the filter selection flag (e.g. a direction flag to distinguish between a vertical filter and a horizontal filter or other directions) and feeds the feature map values to the layers with the corresponding filter shape set. In Figure 31 In some embodiments, a flexible filter shape can be provided, wherein the information representing the filter shape represents a mask composed of flags, and the mask represents the filter shape in that a flag having a third value represents a non-zero filter coefficient, and a flag having a fourth value different from the third value represents a filter coefficient of 0. That is, as already described for the encoding side, the filter shape can be defined by indicating the positions of the non-zero coefficients. The non-zero coefficients can be derived according to a predefined rule or can be indicated.

[0371] Generally, the upsampling of the feature map uses a filter, and the input information from the bitstream also includes information representing the filter shape and / or filter direction and / or filter coefficients. There can be an implementation with a set of filters to select per layer, or an implementation with one filter per layer and signal feeding logic that determines which layers to select and which layers to bypass according to a filter selection flag (indicator).

[0372] In some embodiments, a flexible filter shape can be provided, wherein the information representing the filter shape represents a mask composed of flags, and the mask represents the filter shape in that a flag having a third value represents a non-zero filter coefficient, and a flag having a fourth value different from the third value represents a filter coefficient of 0. That is, as already described for the encoding side, the filter shape can be defined by indicating the positions of the non-zero coefficients. The non-zero coefficients can be derived according to a predefined rule or can be indicated.

[0373] The decoder embodiments described above can be implemented as a computer program product stored in a non-transitory medium that, when executed on one or more processors, performs the steps of any of the methods described above. Similarly, the decoder embodiments described above can be implemented as a device for decoding an image or video, comprising processing circuitry configured to perform the steps of any of the methods described above. In particular, a device for decoding data for image or video processing from a bitstream can be provided, the device comprising: an obtaining unit configured to obtain, from the bitstream, two or more sets of feature map elements, wherein each set of feature map elements is related to a feature map; an input unit configured to input each set of feature map elements of the two or more sets of feature map elements to two or more feature map processing layers of a plurality of cascaded layers, respectively; and a decoded data obtaining unit configured to obtain the decoded data for image or video processing as a processing result of the plurality of cascaded layers. These units can be implemented in software or hardware or a combination of both, as discussed in more detail below.

[0374] Decoding using segmentation information

[0375] At the receiving end, the decoder of the present embodiment performs parsing and interpretation of the segmentation information. Thus, as shown in Figure 34 A method for decoding data for image or video processing from a bitstream is provided. Accordingly, an apparatus for decoding data for image or video processing from a bitstream is provided. The apparatus can comprise processing circuitry for performing the method steps.

[0376] The method comprises obtaining 3410 two or more sets of segmentation information elements from the bitstream. The obtaining can be performed by parsing the bitstream. In some example implementations, the bitstream parsing can further comprise entropy decoding. The present invention is not limited to any particular way of obtaining data from the bitstream. The method further comprises inputting 3420 each of the two or more sets of segmentation information elements into two or more segmentation information processing layers of a plurality of cascaded layers, respectively. It is noted that the segmentation information processing layers can be the same layers as the feature map processing layers or different layers. That is, one layer can have one or more functionalities.

[0377] Further, in each of the two or more segmentation information processing layers, the method comprises processing the respective set of segmentation information. Obtaining 3430 the decoded data for image or video processing is performed in accordance with the segmentation information processed by the plurality of cascaded layers.

[0378] Figure 26 An example segmentation information for three-layer decoding is shown. The segmentation information can be regarded as a selection (see encoder side description) of the layers for which the feature map elements are parsed or otherwise obtained. The feature map element 2610 is not selected. Thus, the encoder sets the flag 2611 to 0. That is, the feature map element 2610 with the lowest resolution is not included in the bitstream. However, the flag 2611 indicating that the feature map element is not selected is included in the bitstream. For example, if the feature map element is a motion vector, then the motion vector 2610 of the largest block is not selected and not included in the bitstream.

[0379] In the example shown in Figure 26 In the feature map 2620, of the four feature map elements used to determine the feature map elements of the feature map 2610, three feature map elements are selected for indication (indicated by flags 2621, 2622 and 2624), and one feature map element 2623 is not selected. In the example using motion vectors, then three motion vectors are selected from the feature map 2620 and their respective flags are set to 1, and one feature map element is not selected and its respective flag 2623 is set to 0.

[0380] Then, the bitstream can include all four flags 2621 to 2624 and the three selected motion vectors. Generally, the bitstream can include the four flags 2621 to 2624 and the three selected feature map elements. In the feature map 2630, one or more elements of the unselected feature map elements of the feature map 2620 can be selected.

[0381] In this example, when selecting the feature map elements, no elements of the higher resolution feature map are selected. In this example, no feature map elements of the feature map 2630 are selected for determining the feature map elements represented by flags 2621, 2622, and 2624. In one embodiment, none of the flags of these feature map elements are included in the bitstream. Instead, only the flags of the feature map elements of the feature map 2630 are included in the bitstream, which determine the feature map element with flag 2623.

[0382] In the example where the feature map elements are motion vectors, the feature map elements 2621, 2622, and 2624 can each be determined by a group of four motion vectors in the feature map 2630. In determining each group of motion vectors with flags 2621, 2622, and 2624, the motion vectors can have more similarity to each other than the four motion vectors in the unselected (indicated by flag 2623) feature map 2630 that determine the motion vectors (feature map elements) in the feature map 2620.

[0383] Figure 26 The above is described by the features of the bitstream. It is noted that the decoder accordingly decodes (parses) these bitstreams: the decoder determines which information is included (indicated) according to the values of the flags as described above, and accordingly parses / interprets the parsed information.

[0384] In an example implementation, the partition information is organized as shown in Figure 27 For 2D information such as images or videos that are considered as image sequences, the feature maps of some layers can be represented in a two-dimensional space. The partition information includes an indicator (a binarized flag) for a 2D spatial location, indicating whether the feature map value corresponding to the location is presented in the bitstream.

[0385] In Figure 27In the middle, there is a starting layer (Layer 0) for decoding the segmentation information, e.g. the lowest resolution layer, i.e. the latent representation layer. For this starting layer, each 2D position includes a binarized flag. If this flag is equal to 1, the selected information includes the feature map value for this position on this particular layer. On the other hand, if this flag is equal to 0, there is no information for this position on this particular layer. This set of flags (or generally a flag tensor, here a flag matrix) is called TakeFromCurrent. The TakeFromCurrent tensor is upsampled to the next layer resolution, e.g. using the nearest neighbor method. Let us denote this tensor as TakeFromPrev. The flags in this tensor indicate whether the corresponding sample position is filled in the previous layer (here Layer 0).

[0386] As a next step, the signal feed logic reads the flag for the current resolution layer position (LayerFlag). In this example implementation, only positions are indicated that are not filled in the previous layer (not set to 1, no feature map element value filled). Using logical operations, this can be expressed as TakeFromPrev == 0 or!TakeFromPrev == 1, where "!" denotes the logical NOT operation (negation).

[0387] The required amount of flags for this layer can be calculated as the amount of 0 (logical false) elements in the TakeFromPrev tensor, or the amount of values that are 1 (logical true) in the inverse (!TakeFromPrev) tensor. For non-zero elements in the TakeFromPrev tensor, no flags need to be read from the bitstream. This is indicated in the figure by showing “-” at the positions where no reading is required. From an implementation point of view, it can be easier to calculate the sum of elements on the inverse tensor as sum(!TakeFromPrev). The signal feeding logic can use this algorithm to identify how many flags need to be parsed from the bitstream. The reading of the flags is placed at the positions where!TakeFromPrev has a value equal to 1 to obtain the LayerFlag tensor. Then, the TakeFromCurrent tensor for the current resolution layer (here, layer 1) is obtained as a combination of the TakeFromPrev and LayerFlag tensors by keeping the flags at the positions that are read from the bitstream for the current resolution layer and zeroing the values at the positions that were read for the previous resolution layer (positions where LayerFlag is “-”). This can be expressed and implemented using a logical AND operator as follows: TakeFromCurrent =!TakeFromPrev AND LayerFlag. Then, to take into account the positions that were read for the previous resolution layer, the TakeFromCurrent tensor is obtained using TakeFromCurrent = TakeFromCurrent OR TakeFromPrev, using a logical OR operation. It should be understood that the Boolean operations can be implemented using regular mathematical operations, e.g., multiplication for AND and summation for OR. This will provide the benefit of preserving and transferring the gradients, which can be used in an end-to-end training using the above approach.

[0388] Then, the obtained TakeFromCurrent tensor is upsampled to the next resolution layer (here, layer 2) and the above operations are repeated.

[0389] For generality and to simplify the implementation, it is beneficial to unify the processing of all resolution layers without special considerations for the first resolution layer, where all flags are parsed from the bitstream for the first resolution layer. This can be achieved by initializing TakeFromPrev to 0 before processing the first (lower resolution) layer (layer 0) and repeating the above steps for each resolution layer.

[0390] To further reduce the signaling overhead, in some other implementations, the LayerFlags for the last resolution layer (here, layer 3, i.e., layer 2) do not need to be transmitted (included at the encoder and parsed at the decoding side) into the bitstream. That is, for the last resolution layer, the feature map values are selected as the selected information for all positions of the last resolution layer (seeFigure 11 The positions 1120 in the bitstream are not acquired in (all) previous resolution layers (or any previous resolution layers). That is, for the last resolution layer, TakeFromCurrent = ! TakeFromPrev, meaning TakeFromCurrent corresponds to the inverted TakeFromPrev. Similarly, to maintain the generality of the last resolution layer processing, LayerFlag can be initialized to 1 and can be used with the same formula: TakeFromCurrent = ! TakeFromPrev AND LayerFlag.

[0391] In some other possible implementations, the last resolution layer has the same resolution as the original image. If the last resolution layer has no further processing steps, some values ​​of the original tensor are transmitted without compression by the autoencoder.

[0392] Below, for reference Figure 28 An example describing the signal feed logic 2800. In Figure 28 In this process, the decoder's signal feed logic 2800 uses segmentation information (LayerFlag) to obtain and utilize selected information (LayerMv) transmitted in the bitstream. Specifically, at each layer, the bitstream is parsed to obtain the segmentation information (LayerFlag) in the corresponding syntax decoding units 2823, 2822, and 2821, as well as the possible selected information (LayerMv) (in this order). As mentioned above, in order to enable the same operation as other layers in the first layer (syntax decoding 2823), the TakeFromPrev tensor is initialized to 0 in 2820. The TakeFromPrev tensor is propagated in the processing order from the syntax decoding of earlier layers (e.g., from 2823) to the syntax decoding of later layers (e.g., 2822). This propagation includes an upsampling of 2, as referenced above. Figure 27 The explanation given.

[0393] In interpreting the segmentation information (LayerFlag) of each resolution layer, the tensor TakeFromCurrent is obtained (generated). This tensor TakeFromCurrent includes flags that indicate whether feature map information (LayerMv) is present in the code stream for each particular position of the current resolution layer. The decoder reads the values of the feature maps LayerMv from the code stream and places these values in the positions of the TakeFromCurrent tensor whose flags are equal to 1. The total amount of feature map values included in the code stream of the current resolution layer can be calculated as the number of non-zero elements in TakeFromCurrent or as sum(TakeFromCurrent) (sum of all elements of the TakeFromCurrent tensor). In the next step, the tensor in each layer (e.g., in 2812) combines the outputs of the previous resolution layers (e.g., generated by 2813 and upsampled 2801 to match the layer processing 2812 of the following layer) by replacing the values of the feature maps at the positions of the TakeFromCurrent tensor whose values are equal to 1 with the values of the feature maps transmitted in the code stream as selected information (LayerMv). As mentioned above, in order to enable the same operations in the first layer (tensor combination 2813) as in the other layers, the combination tensor is initialized to 0 in 2810. After processing the LayerFlags from all layers and generating the output tensor of the last layer (in 2811), the combination tensor is upsampled by 4 in 2801 to obtain the original size of the dense optical flow, i.e., W x H.

[0394] Figure 28 An exemplary implementation of the example implementation provides a fully parallelizable scheme that can be run on a GPU / NPU and can exploit parallelism. The fully trainable scheme that transmits gradients can be used in an end-to-end trainable video coding technology scheme.

[0395] Figure 29 Another possible and exemplary implementation of the signal feeding logic 2900 is shown. This implementation generates a LayerIdx tensor ( Figure 29LayerIdx tensor, including the indices of the different resolution layers, indicates which layer should be used to fetch the motion information transmitted in the bitstream (included in the encoder, parsed in the decoding side). At each syntax parsing block (2923, 2922, 2923), the LayerIdx tensor is updated by adding the TakeFromCurrent tensor multiplied by the upsampled layer indices numbered from the highest resolution to the lowest resolution. Then, the LayerIdx tensor is upsampled and passed to the next layer in the processing order, e.g. from 2923 to 2922, from 2922 to 2921. To make the processing similar in all layers, the tensor LayerIdx is initialized to 0 in 2920 and passed to the syntax parsing 2923 of the first layer.

[0396] After the last layer (here the third layer), the LayerIdx tensor is upsampled to the original resolution (upsampling 2995 by 4). Thus, each position of LayerIdx includes the index of the layer from which to fetch the motion information. The positions of LayerIdx correspond to the original resolution of the feature map data (here the dense optical flow) at the same resolution, which is 2D (matrix) in this example. Thus, for each position in the reconstructed optical flow, LayerIdx indicates the position from which to fetch the motion information (which layer’s MayerMV).

[0397] The motion information (LayerMv, in Figure 29LayerMvUp) is generated in the following way. At each spatial resolution layer, the tensor combination block (2913, 2912, 2911) combines the LayerMv (parsed from the bitstream by the respective syntax parsing unit 2923, 2922, 2921) with the intermediate tensors and the intermediate TakeFromCurrent boolean tensor according to the method described above, according to the segmentation information (LayerFlag) parsed from the bitstream. The intermediate tensors can be initialized to 0 (see initialization units 2910, 2919, 2918) or to any other value. The initialization value is not important, as eventually, after all steps are completed, according to the method, these values are not selected for the dense optical flow reconstruction 2990. The combined tensor including the motion information (output from each of 2913, 2912, 2911) is upsampled and concatenated (2902, 2901) with the combined tensor of the previous spatial resolution layer. The concatenation is performed along the additional dimension corresponding to the motion information obtained from the layer of different resolution (i.e. the 2D tensor before concatenation 2902 becomes a 3D tensor after concatenation; the 3D tensor before concatenation 2901 remains a 43D tensor after concatenation, but the size of the tensor is increased). Finally, after all upsample steps of LayerIdxUp and LayerMvUp are completed, the reconstructed dense optical flow is obtained by selecting the motion information from LayerMvUp using the values of LayerIdxUp as indices on the axis in LayerMvUp, which is the dimension added during the LayerMvUp concatenation steps. That is, the dimension added in LayerMvUp is the dimension over the number of layers, and LayerIdxUp selects the appropriate layer for each position.

[0398] The above-described specific exemplary implementations do not limit the present application. In general, the segmentation can be performed and indicated in the bitstream in various conceivable ways. In general, the set of segmentation information elements is obtained from the segmentation information processed by at least one segmentation information processing layer of the plurality of concatenated layers. As Figure 28 illustrated, such a layer can include a syntax parsing unit (2823, 2822, 2821) that parses / interprets the meaning (semantics) of the parsed segmentation information LayerFlag.

[0399] More specifically, the set of input segmentation information elements is obtained from the processed segmentation information output by at least one concatenation layer of the plurality of concatenated layers. For example, in Figure 28 this is illustrated by passing the TakeFromPrev tensor between the syntax parsing units (2823, 2822, 2821). As already explained in the description of the encoding side, in some exemplary implementations, the resolution of the segmentation information processed in two or more segmentation information processing layers, respectively, is different.

[0400] Further, the processing of the segmentation information in two or more segmentation information processing layers comprises upsampling, as already referred to Figure 9 、 Figure 13 and other figures illustrate. For example, the upsampling of the segmentation information comprises nearest neighbor upsampling. It is noted that in the present embodiment and in the above embodiments, the application is not limited by the application of nearest neighbor upsampling. The upsampling can comprise interpolation, instead of simply copying the value of a neighboring sample (element). The interpolation can be any known interpolation, for example, linear or polynomial interpolation, for example, cubic upsampling, etc. In relation to copying, it is noted that the copying performed by the nearest neighbor is copying of the element value from the predefined (available) nearest neighbor (e.g., above or left). If the neighbor with the same distance to the position to be filled is available, it can be necessary to copy that neighbor.

[0401] As described above in relation to the first modification, in some example implementations, the upsampling comprises transposed convolution. In addition to or instead of applying convolutional upsampling to the feature map information, convolutional upsampling can also be applied to the segmentation information. It is noted that the type of upsampling performed in relation to the segmentation information is not necessarily the same type of upsampling applied to the feature map elements.

[0402] Generally, for each segmentation information processing layer j of the plurality N of segmentation information processing layers in the plurality of cascaded layers, the input comprises:

[0403] - if j = 1, the initial segmentation information is input from the bitstream (and / or based on initialization, e.g., initialized to 0 in 2820), otherwise the segmentation information processed by the (j - 1)th segmentation information processing layer is input;

[0404] - the processed segmentation information is output.

[0405] This is the segmentation information related to the input layer, not necessarily (still possible) the entire segmentation information from the bitstream. The upsampling segmentation information in the jth layer is the segmentation information upsampling in the jth layer, i.e., output by the jth layer. Generally, the processing of the segmentation layer comprises upsampling (TakeFromPrev) and comprises new elements from the bitstream (LayerFlag).

[0406] For example, the processing of the input segmentation information by each j < N of the plurality N of segmentation information processing layers further comprises parsing a segmentation information element (LayerFlag) from the bitstream, and associating the parsed segmentation information element with the segmentation information output by the previous layer (TakeFromPrev), e.g., in 2820. Figure 28the syntax parsing unit 282x in the picture decoder 280). The position of the parsed segmentation information element (LayerFlag) in the associated segmentation information is determined according to the segmentation information output by the previous layer. As can be seen in Figure 28 and Figure 29 there are various ways to associate and propagate the position information. The present application is not limited to any particular implementation.

[0407] For example, the number of segmentation information elements parsed from the bitstream is determined according to the segmentation information output by the previous layer. Specifically, if some regions have already been covered by the segmentation information of the previous layer, it is not necessary to cover them again on the following layer. It is to be noted that this design provides an efficient parsing method. Each position of the resulting reconstructed feature map data corresponding to the position of the resulting reconstructed segmentation information is only associated with the segmentation information belonging to a single layer (out of N processing layers). Therefore, there is no information duplication. However, the present application is not limited to this method. It is conceivable that the segmentation information is duplicated, even though this can result in some redundant information.

[0408] As has been shown in Figure 27 in some embodiments, the parsed segmentation information elements are represented by a set of binarized flags. The ordering of the flags in the bitstream (syntax) can convey the association between a flag and the layer it belongs to. The order (sequence) can be given by a predefined processing order at the encoding side, and accordingly, also at the decoding side. For example, Figure 16 and Figure 28 are exemplified.

[0409] In some exemplary embodiments, such as the one mentioned above with reference to the seventh modification, obtaining the decoded data for image or video processing comprises determining at least one of the following parameters according to the segmentation information. The segmentation information can determine the parsing of additional information such as coding parameters in a similar way as the motion information, which can include: intra- or inter-prediction mode; picture reference index; single- or multi-reference prediction (including bi-prediction) ; presence or absence of prediction residual information; quantization step size; motion information prediction type; motion vector length; motion vector resolution; motion vector prediction index; motion vector difference size; motion vector difference resolution; motion interpolation filter; in-loop filter parameters; and / or post-filter parameters, etc. That is, the segmentation information can indicate from which processing layer of coding parameters the coding parameters can be obtained when being processed by the segmentation information processing layer. For example, in Figure 22 or Figure 23 the above encoder method, the reconstructed (coding) parameters can be received from the bitstream instead of (or in addition to) from the motion information (LayerMv). Such reconstructed (coding) parameters blk_rec_params can be received at the decoding side in a similar way as Figure 28 andFigure 29 The motion information is parsed in the same way as shown in the middle.

[0410] Generally, the segmentation information is used for feature map element (motion information or any of the above reconstruction parameters or sample related data) parsing and input. The method can further include obtaining a set of feature map elements from the bitstream, and inputting the set of feature map elements into a feature map processing layer in the plurality of cascaded layers respectively according to the segmentation information processed by the segmentation information processing layer. In addition, the method further includes obtaining decoded data for image or video processing according to the feature map processed by the plurality of cascaded layers. Specifically, in some embodiments, at least one of the plurality of cascaded layers is a segmentation information processing layer and a feature map processing layer. As described above, the network can be designed with separate segmentation information processing layers and feature map processing layers, or with combined layers with both functions. In some implementations, each layer in the plurality of layers is either a segmentation information processing layer or a feature map processing layer.

[0411] The above method can be embodied as a computer program product stored in a non-transitory medium, which, when executed on one or more processors, causes the processors to perform the steps of any of the methods. Similarly, a device for decoding an image or video is provided, including processing circuitry for performing the method steps of any of the methods discussed above. The functional structure of the apparatus provided by the present application can correspond to the functions provided by the above embodiments and steps. For example, a device for decoding data for image or video processing from a bitstream is provided, wherein the device includes: an obtaining unit for obtaining two or more sets of segmentation information elements from the bitstream; an input unit for inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers respectively; a processing unit for processing the respective set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data obtaining unit for obtaining the decoded data for image or video processing according to the segmentation information processed in the plurality of cascaded layers. These units and other units can perform all the functions of the above methods.

[0412] Brief overview of some embodiments

[0413] Embodiments regarding encoding using feature information or segmentation information

[0414] According to an aspect of the application, there is provided a method of encoding data for image or video processing into a bitstream. The method comprises processing the data, the processing comprising generating feature maps in a plurality of cascaded layers, each feature map comprising a respective resolution, wherein resolutions of at least two of the generated feature maps are different from each other; selecting a layer from the plurality of layers different from a layer in which a lowest resolution feature map is generated; generating the bitstream, comprising inserting information related to the selected layer into the bitstream.

[0415] This method can improve the efficiency of such encoding, as it can encode data from different layers, thereby including feature maps of different resolutions or other types of layer-related information in the bitstream.

[0416] According to an aspect of the application, there is provided an apparatus for encoding data for image or video processing into a bitstream. The apparatus comprises a processing unit configured to process the data, wherein the processing comprises generating feature maps in a plurality of cascaded layers, each feature map comprising a respective resolution, resolutions of at least two of the generated feature maps being different from each other; a selection unit configured to select a layer from the plurality of layers different from a layer in which a lowest resolution feature map is generated; and a generation unit configured to generate the bitstream, comprising inserting an indication of data related to the selected layer into the bitstream. The processing unit, the selection unit and the generation unit can be implemented by processing circuitry, e.g. one or more processors or any combination of software and hardware.

[0417] This apparatus can improve the efficiency of decoding, as it can decode data from different layers and use it for reconstruction, thereby enabling the use of feature maps of different resolutions or other types of layer-related information.

[0418] In an example implementation, the processing further comprises downsampling by one or more of the cascaded layers. Applying downsampling can on the one hand reduce the complexity of the processing and on the other hand reduce the data provided in the bitstream. Moreover, processing layers of different resolutions can in this way focus on features of different scales. Thus, a network processing an image (still or video) can be run efficiently.

[0419] For example, the one or more downsampling layers comprise average pooling or max pooling for downsampling. Average pooling and max pooling operations are part of several frameworks, which provide an efficient method for low-complexity downsampling.

[0420] In another example, convolution is used for down-sampling. Convolution can provide some more complex down-sampling methods, kernels can be properly selected for specific applications, and kernels can even be trained. This enables the learnable down-sampling process to find a more suitable potential representation of motion information and maintain the advantages of representing and transmitting information of different spatial resolutions, thereby improving adaptability.

[0421] In one example implementation, the information related to the selected layer includes elements of a feature map of the layer.

[0422] By providing features with different resolutions, the scalability of encoding / decoding is improved, and the bitstream thus generated can provide higher flexibility to meet optimization criteria such as rate, distortion, complexity, etc., ultimately improving coding efficiency.

[0423] In any of the above examples, for example, the information related to the selected layer includes information indicating from which layer and / or from which part of the feature map of the layer the elements of the feature map of the layer are selected.

[0424] Signaling the segmentation information can efficiently code the feature maps from different layers so that each region of the original (to be coded) feature map (data) can be included only in the information of one layer. Although this is not a limitation of the present application, in some cases, there can be some repetition of inter-layer information for a particular region in the feature map (data) to be encoded in the present application.

[0425] The above method includes, in an example implementation, a step of obtaining the data to be encoded, wherein the processing of the data to be encoded includes processing by each jth layer of the plurality (N) of cascaded layers, including: if j = 1, obtaining the data to be encoded as a layer input, otherwise obtaining the feature map processed by the (j-1)th layer as a layer input; processing the obtained layer input, the processing including down-sampling; outputting the down-sampled feature map.

[0426] Correspondingly, the above apparatus has, in an example implementation, a processing unit, wherein the processing unit is configured to obtain the data to be encoded and perform processing on the data to be encoded, including: the processing of each jth layer of the plurality (N) of cascaded layers includes: if j = 1, obtaining the data to be encoded as a layer input, otherwise obtaining the feature map processed by the (j-1)th layer as a layer input; processing the obtained layer input, the processing including down-sampling; outputting the down-sampled feature map.

[0427] The method according to any of the preceding examples comprises, in some embodiments, selecting information for insertion into the bitstream, the information relating to a first region in a feature map processed by a layer j > 1, wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer smaller than j comprising a plurality of elements; not selecting a region corresponding to the first region when a feature map processed by k layers is selected, wherein k is an integer equal to or greater than 1 and k < j.

[0428] The apparatus according to any of the preceding examples comprises, in some embodiments, the processing circuitry further configured to select information for insertion into the bitstream, the information relating to a first region in a feature map processed by a layer j > 1, wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer smaller than j comprising a plurality of elements; not selecting a region corresponding to the first region when a feature map processed by k layers is selected, wherein k is an integer equal to or greater than 1 and k < j.

[0429] This selection in certain layers does not include regions of the original feature map included by other layers, which is particularly efficient in terms of coding overhead.

[0430] In any of the preceding examples, the data to be encoded comprises, for example, image information and / or prediction residual information and / or prediction information.

[0431] Alternatively, the information relating to the selected layer comprises prediction information.

[0432] In any of the preceding examples, the data relating to the selected layer comprises, for example, an indication of the position of the feature map element in the feature map of the selected layer.

[0433] This indication enables a correct association of the feature map elements of different resolutions with the input data region.

[0434] In any of the preceding examples, the positions of the selected and non-selected feature map elements are represented by a plurality of binarization flags, the representation being based on the position of the flags in the bitstream.

[0435] The binarization flags provide a particularly efficient way of coding the partition information.

[0436] According to an embodiment, in the above method or apparatus, the processing of the jth layer of the plurality of N cascaded layers comprises determining a first cost resulting from reconstructing a portion of a reconstructed image using feature map elements output by the jth layer, determining a second cost resulting from reconstructing the portion of the reconstructed image using feature map elements output by the (j-1)th layer; if the first cost is higher than the second cost, selecting the (j-1)th layer, and selecting information related to the portion in the (j-1)th layer.

[0437] Optimizations including distortion are provided to achieve the desired quality in a more efficient manner.

[0438] For example, the first cost and the second cost include data amount and / or distortion. By considering rate (amount of data generated by the encoder) and distortion of the reconstructed image for optimization, various application or user requirements can be flexibly met.

[0439] Alternatively, or in addition, the data to be encoded is a motion vector field. The above method is applicable to compressing a motion vector field, such as dense optical flow or sub-sampled optical flow. Application of these methods can efficiently code the motion vector (in terms of rate and distortion or other criteria) and further reduce the size of the code stream of the encoded image or video data.

[0440] In some embodiments, the prediction information includes reference index and / or prediction mode. In addition to or alternatively to the motion vector field, other information related to prediction can be processed. The reference index and the prediction mode can be similar to the motion vector field, related to the content of the image, and thus, encoding feature map elements with different resolutions can improve efficiency.

[0441] For example, the data amount includes the amount of data required to transmit data related to the selected layer. In this way, during optimization, the overhead resulting from providing information related to a layer different from the output layer can be calculated.

[0442] In addition or alternatively, distortion is calculated by comparing the reconstructed image with a target image. This end-to-end quality comparison ensures that distortion in the reconstructed image is properly considered. Thus, optimization can select coding methods in an efficient manner and meet quality requirements proposed by the application or user in a more accurate manner.

[0443] In any of the above examples, for example, the processing includes additional convolution layers between the cascaded layers with different resolutions.

[0444] Providing such additional layers in the network of cascaded layers can introduce additional processing, such as various types of filtering, to improve the quality or efficiency of coding.

[0445] According to an example implementation, the processing circuitry of the method or of the apparatus as described in the above embodiments comprises, when downsampling at a layer, downsampling an input feature map using a first filter to obtain a first feature map, downsampling the input feature map using a second filter to obtain a second feature map, determining a third cost resulting from reconstructing a portion of a reconstructed image using the first feature map, determining a fourth cost resulting from reconstructing the portion of the reconstructed image using the second feature map; when selecting, selecting the first feature map if the third cost is lower than the fourth cost.

[0446] Applying different downsampling filters helps to adapt to different characteristics of the content.

[0447] For example, the shape of the first filter and the second filter can be any one of a square, a rectangle in horizontal direction and a rectangle in vertical direction.

[0448] The shape of these filters is still simple, but can be improved in other aspects, e.g. to adapt to object boundaries.

[0449] The method steps or steps performed by the processing circuitry of the apparatus can further comprise obtaining a mask, wherein the mask consists of flags, the mask representing an arbitrary filter shape, the first filter or the second filter having the arbitrary filter shape.

[0450] A filter of any shape can be designed more flexibly.

[0451] The method steps or steps performed by the processing circuitry of the apparatus can further comprise processing data related to the same image being partitioned into blocks having different block sizes and shapes in the different layers, the selecting comprising selecting a layer according to a calculated cost of a predetermined set of coding modes.

[0452] In some example implementations, the processing comprises determining a cost of different sets of coding modes for at least one layer, and selecting one of the sets of coding modes according to the determined cost.

[0453] Applying optimization to coding modes can enable efficient rate-distortion optimization, thereby improving coding efficiency.

[0454] For example, the indication of data related to the selected layer comprises the selected set of coding modes.

[0455] According to an aspect of the present application, there is provided a computer program stored in a non-transitory medium, the computer program comprising code which when executed on one or more processors performs the steps of any of the methods described above.

[0456] According to an aspect of the application, there is provided a device for encoding an image or video, comprising processing circuitry configured to perform a method according to any of the examples described above.

[0457] Any of the apparatuses described above can be implemented on an integrated chip. The application can be implemented in hardware (HW) and / or software (SW). Furthermore, a hardware-based implementation can be combined with a software-based implementation.

[0458] It is noted that the application is not limited to a specific framework. Furthermore, the application is not limited to image or video compression, but can also be applied to object detection, image generation and recognition systems.

[0459] For the sake of clarity, any of the embodiments described above can be combined with any one or more of the other embodiments described above to create a new embodiment within the scope of the application.

[0460] Regarding embodiments using feature map elements for decoding

[0461] According to an embodiment, there is provided a method for decoding data for image or video processing from a bitstream, wherein the method comprises: obtaining two or more sets of feature map elements from the bitstream, wherein each set of feature map elements is related to a feature map; inputting each of the two or more sets of feature map elements into two or more feature map processing layers of a plurality of cascaded layers, respectively; obtaining the decoded data for image or video processing as a processing result of the plurality of cascaded layers.

[0462] This method can improve efficiency, as the method enables data from different layers to be used for decoding, thereby parsing feature or other types of layer-related information from the bitstream.

[0463] For example, processing the feature map in each of the two or more feature map processing layers, wherein the resolution of the feature map processed in each of the two or more feature map processing layers is different.

[0464] In some embodiments, the processing of the feature map in the two or more feature map processing layers comprises upsampling.

[0465] Upsampling can reduce the complexity of processing on the one hand (as the first layer has a lower resolution) and on the other hand can reduce the data provided in the bitstream and parsed at the decoding end. Furthermore, processing layers of different resolutions can in this way focus on features of different scales. Thus, a network processing an image (still or video) can be run efficiently.

[0466] In an example implementation, the method further comprises the step of obtaining, from the bitstream, partition information related to two or more layers, wherein obtaining the feature map elements from the bitstream is performed according to the partition information, and wherein inputting each of the two or more sets of feature map elements into the two or more feature map processing layers is performed according to the partition information.

[0467] Using the partition information allows for an efficient decoding of the feature maps from different layers, such that each region of the original (to be reconstructed) can only be comprised in the information of one layer. While this does not limit the invention, in some cases the invention can also provide for an overlap between the layers for specific regions in the feature map (data). For example, the plurality of cascaded layers further comprises a plurality of partition information processing layers, and the method further comprises processing the partition information in the plurality of partition information processing layers. This approach provides the possibility to control the parsing of the feature elements from different layers.

[0468] In some embodiments, processing the partition information in at least one of the plurality of partition information processing layers comprises upsampling. The hierarchical structure of the partition information can provide for a small amount of side information to be inserted into the bitstream, thereby improving efficiency and / or reducing processing time.

[0469] For example, the upsampling of the partition information and / or the upsampling of the feature map comprises nearest neighbor upsampling. Nearest neighbor upsampling has a low computational complexity and can be easily implemented. Nevertheless, nearest neighbor upsampling is very efficient, in particular for logical indications such as flags.

[0470] In some embodiments and examples, the upsampling of the partition information and / or the upsampling of the feature map comprises transposed convolution. The use of convolution helps to reduce blocking artifacts and enables a trainable approach, wherein the upsampling filter is selectable.

[0471] In an example implementation, obtaining the feature map elements from the bitstream is performed according to the processed partition information processed by at least one of the plurality of partition information processing layers.

[0472] In an example implementation, inputting each of the two or more sets of feature map elements into the two or more feature map processing layers is performed according to the processed partition information processed by at least one of the plurality of partition information processing layers.

[0473] According to an embodiment, the obtained segmentation information is represented by a set of syntax elements, wherein a position of an element in the set of syntax elements indicates which feature map element position the syntax element is related to. For each syntax element, the processing of the feature map comprises: when the syntax element has a first value, parsing an element of the feature map from the bitstream at a position indicated by the position of the syntax element in the bitstream, otherwise, not parsing an element of the feature map from the bitstream at a position indicated by the position of the syntax element in the bitstream.

[0474] Such a relationship between the segmentation information and the feature map information enables efficient coding of the frequency information and parsing in a hierarchical structure by considering different resolutions.

[0475] For example, the processing of the feature map in each layer 1 < j < N of the plurality of N feature map processing layers further comprises: parsing a segmentation information element of the jth feature map processing layer from the bitstream; obtaining a feature map processed by a previous feature map processing layer; parsing a feature map element from the bitstream and associating the parsed feature map element with the obtained feature map, wherein a position of the feature map element in the processed feature map is indicated by the parsed segmentation information element, said segmentation information being processed by the previous segmentation information processing layer.

[0476] In particular, the method can comprise: when the syntax element has a first value, parsing an element of the feature map from the bitstream; when the syntax element has a second value or the segmentation information processed by a previous segmentation information processing layer has a first value, not parsing an element of the feature map from the bitstream.

[0477] For example, the syntax element parsed from the bitstream representing the segmentation information is a binarization flag. In particular, the processed segmentation information is represented by a set of binarization flags.

[0478] Providing a binarization flag enables efficient coding. At the decoding side, the processing of the logical flag can be performed with low complexity.

[0479] According to an exemplary implementation, the upsampling of the segmentation information in each segmentation information processing layer j further comprises: for each pth position in the obtained feature map represented by the input segmentation information, determining, as the upsampled segmentation information, an indication for feature map positions comprised in a region in the reconstructed image identical to the pth position.

[0480] For example, the data for image or video processing comprises a motion vector field. Since a dense optical flow or motion vector field with a similar resolution as the image resolution is needed to model the motion, the hierarchical structure of the present invention is suitable for efficiently reconstructing such motion information. By the layer processing and the indication, a good trade-off between rate and distortion can be achieved.

[0481] For example, the data used for image or video processing comprises image data and / or prediction residual data and / or prediction information data. The present application can be used for various different parameters. However, the image data and / or prediction residual data and / or prediction information data can still have some redundancy in the spatial domain, and the hierarchical approach described herein can provide efficient decoding from the bitstream using different resolutions.

[0482] In some embodiments and examples, the filter is used for upsampling of the feature map, and the shape of the filter is any one of a square, a horizontal rectangle, and a vertical rectangle.

[0483] Applying different upsampling filters helps to adapt to different features of the content. For example, upsampling of the feature map using a filter, the information input from the bitstream further comprises information indicative of a shape of the filter and / or filter coefficients from the bitstream.

[0484] Accordingly, the decoder can provide better reconstruction quality according to the information from the encoder sent in the bitstream.

[0485] For example, the information indicative of the shape of the filter represents a mask composed of flags, and the mask represents the shape of the filter in that a flag having a third value represents a non-zero filter coefficient, and a flag having a fourth value different from the third value represents a filter coefficient of 0. Any shape of filter can be designed more flexibly.

[0486] For example, the plurality of cascaded layers comprises convolutional layers without upsampling between layers having different resolutions.

[0487] Providing such additional layers in the network of cascaded layers enables to introduce additional processing, such as various types of filtering, to improve the quality or efficiency of the coding.

[0488] According to an embodiment, there is provided a computer program product stored in a non-transitory medium, which when executed on one or more processors, performs the method according to any of the methods described above.

[0489] According to an embodiment, there is provided a device for decoding an image or a video, comprising processing circuitry configured to perform the method according to any of the embodiments and examples described above.

[0490] According to an aspect, there can be provided an apparatus for decoding data for image or video processing from a bitstream, the apparatus comprising: an obtaining unit configured to obtain two or more sets of feature map elements from the bitstream, wherein each set of feature map elements is associated with a feature map; an input unit configured to input each set of feature map elements of the two or more sets of feature map elements to two or more feature map processing layers of a plurality of cascaded layers, respectively; and a decoded data obtaining unit configured to obtain the decoded data for image or video processing as a processing result of the plurality of cascaded layers.

[0491] Any of the apparatuses described above can be implemented on an integrated chip. The present application can be implemented in hardware (HW) and / or software (SW). Furthermore, a hardware-based implementation can be combined with a software-based implementation.

[0492] It is noted that the present application is not limited to a specific framework. Furthermore, the present application is not limited to image or video compression, but can also be applied to object detection, image generation and recognition systems.

[0493] Some example implementations in hardware and software

[0494] Figure 35 A corresponding system in which the above-described encoder-decoder processing chain can be deployed is shown. Figure 35 For an illustrative coding system, a video, image, audio and / or other coding system (or simply coding system) can be utilized, e.g., in which the techniques of the present application can be employed. Video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of video coding system 10 represent examples of devices that can be used to perform various techniques described in this application. For example, video encoding and decoding can use neural networks or generally processing networks as described in the above embodiments and examples.

[0495] As shown in Figure 35 Coding system 10 includes a source device 12, e.g., configured to provide encoded image data 21 to a destination device 14 for decoding of the encoded image data 13.

[0496] Source device 12 includes an encoder 20, and additionally, optionally, an image source 16, a pre-processor (or pre-processing unit) 18, e.g., an image pre-processor 18, a communication interface or unit 22.

[0497] The image source 16 can comprise or be any type of image capturing device, e.g. a camera for capturing real world images, and / or any type of image generating device, e.g. a computer graphics processor for generating computer animated images, or any other device for acquiring and / or providing real world images, computer generated images (e.g. screen content, virtual reality (VR) images) and / or any combination thereof (e.g. augmented reality (AR) images). The image source can be any type of memory (storage) storing any of the above images.

[0498] The image or image data 17 can also be referred to as raw image or raw image data 17, in distinction from the processing performed by the pre-processor 18 and the pre-processing unit 18.

[0499] The pre-processor 18 is configured to receive (raw) image data 17 and to pre-process the image data 17 to obtain pre-processed image data 19 or pre-processed image 19. The pre-processing performed by the pre-processor 18 can include trimming, color format conversion (e.g. from RGB to YCbCr), color correction or de-noising, etc. It is to be understood that the pre-processing unit 18 can be an optional component. It is noted that the pre-processing can also employ a neural network.

[0500] The video encoder 20 is configured to receive the pre-processed image data 19 and to provide encoded image data 21.

[0501] The communication interface 22 in the source device 12 can be configured to receive the encoded image data 21 and to transmit the encoded image data 21 (or any other processed version thereof) to another device or any other device, e.g. the destination device 14, over the communication channel 13 for storage or direct reconstruction.

[0502] The destination device 14 comprises a decoder 30 (e.g. a video decoder 30) and, additionally, optionally, a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32) and a display device 34.

[0503] The communication interface 28 in the destination device 14 is configured to receive the encoded image data 21 (or any other processed version thereof) directly from the source device 12 or from any other source device, e.g. a storage device, e.g. an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0504] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired or wireless networks or any combination thereof, or any type of private and public network), or any combination thereof.

[0505] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing for transmission on a communication link or communication network.

[0506] For example, the communication interface 28 corresponding to the communication interface 22 can be used to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21.

[0507] Both communication interface 22 and communication interface 28 can be configured as Figure 35 The communication channel 13, indicated by the arrow pointing from source device 12 to destination device 14, is a one-way communication interface, or configured as a two-way communication interface, and can be used to send and receive messages, etc., to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (e.g., using the neural network described in the embodiments or examples above).

[0508] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0509] The display device 34 in the destination device 14 is configured to receive the post-processed image data 33 for displaying an image to a user or viewer, or the like. The display device 34 can be or can comprise any type of display for representing a reconstructed image, such as an integrated or external display or display screen. For example, the display can comprise a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any type of other display.

[0510] Although Figure 35 Although the source device 12 and the destination device 14 are depicted as separate devices, embodiments of the devices can also comprise the source device 12 and the destination device 14 or comprise corresponding functionalities of the source device 12 and the destination device 14 at the same time. In such embodiments, the source device 12 or the corresponding functionalities and the destination device 14 or the corresponding functionalities can be implemented by the same hardware and / or software, by separate hardware and / or software, or any combination thereof.

[0511] From the description, it will be obvious to a skilled person that Figure 35 The presence and (precise) division of the different units or functionalities in the source device 12 and / or the destination device 14 as shown can differ depending on the actual device and application.

[0512] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both, can be implemented by processing circuitry, such as one or more microprocessors, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more discrete logic components, one or more hardware processors, one or more video coding- dedicated processors, or any combination thereof. The encoder 20 can be implemented by processing circuitry 46 to encompass various modules including a neural network. The decoder 30 can be implemented by processing circuitry 46 to include various modules discussed in the above embodiments and examples. The processing circuitry can be used to perform various operations as will be discussed later. When the techniques are implemented partially in software, a device can store instructions for the software in a suitable, non- transitory computer-readable storage medium and execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. The video encoder 20 or the video decoder 30 can be integrated in a single device as part of a combined video encoder / decoder (CODEC), like Figure 36 as shown.

[0513] The source device 12 and the destination device 14 can comprise any of a variety of devices including handheld or fixed devices, e.g., notebook or laptop computers, mobile phones, smart phones, tablets, tablet computers, camcorders, desktop computers, set-top boxes, televisions, display devices, digital media players, video gaming consoles, video streaming devices (e.g., content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and can use or not use any type of operating system. In some cases, the source device 12 and the destination device 14 can be equipped for wireless communication. Thus, the source device 12 and the destination device 14 can be wireless communication devices.

[0514] In some cases, Figure 35The illustrated video coding system 10 is merely exemplary and the techniques provided herein can be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve any data communication between an encoding device and a decoding device. In other examples, data is retrieved from local storage, transmitted across a network, and / or the like. A video encoding device can encode data and store the data into a memory, and / or a video decoding device can retrieve data from the memory and decode the data. In some examples, encoding and decoding are performed by devices that do not communicate with each other, but merely encode data to and / or retrieve and decode data from a memory.

[0515] Figure 37 A schematic diagram of a video coding device 3700 is provided for embodiments of the disclosure. The video coding device 3700 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video coding device 3700 can be a decoder (e.g., the video decoder 30 of FIG. 1) or an encoder (e.g., the video encoder 20 of FIG. 1). Figure 35 Figure 35

[0516] The video coding device 3700 includes an ingress port 3710 (or input port 3710) and a receiver unit 3720 for receiving data, a processor, logic unit, or central processing unit 3730 for processing data, a transmitter unit 3740 and an egress port 3750 (or output port 3750) for transmitting data, and a memory 3760 for storing data. The video coding device 3700 can also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the ingress port 3710, the receiver unit 3720, the transmitter unit 3740, and the egress port 3750 for egress or ingress of optical or electrical signals.

[0517] ​​The processor 3730 is implemented by hardware and software. The processor 3730 can be implemented as one or more CPU chips, cores (e.g., a multi-core processor), FPGAs, ASICs, and DSPs. The processor 3730 is in communication with the ingress port 3710, the receiving unit 3720, the sending unit 3740, the egress port 3750, and the memory 3760. The processor 3730 includes a coding module 3770. The coding module 3770 implements the embodiments disclosed above. For example, the coding module 3770 performs, processes, prepares, or provides various coding operations. Therefore, the coding module 3770 substantially improves the functions of the video coding device 3700 and affects the transformation of the video coding device 3700 to different states. Alternatively, the coding module 3770 is implemented by instructions stored in the memory 3760 and executed by the processor 3730.

[0518] The memory 3760 can include one or more disks, tape drives, and solid-state drives, can be used as an overflow data storage device, to store programs when such programs are selected for execution, and to store instructions and data that are read during program execution. The memory 3760 can be volatile and / or non-volatile, and can be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM), for example.

[0519] Figure 38 A simplified block diagram of an apparatus 3800 is provided for an exemplary embodiment, which can be used as either or both of the source device 12 and the destination device 14 in FIG. 1. Figure 35

[0520] The processor 3802 in the apparatus 3800 can be a central processing unit. Alternatively, the processor 3802 can be any other type of device, either existing or developed in the future, capable of manipulating or processing information. While a single processor 3802 as shown in the figure can be used to implement the disclosed implementations, using more than one processor can improve speed and efficiency.

[0521] ​In one implementation, the memory 3804 in the apparatus 1100 can be a read only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of memory can be used as the memory 3804. The memory 3804 can include code and data 3806 that is accessed by the processor 3802 via the bus 3812. The memory 3804 can also include an operating system 3808 and an application program 3810 that includes at least one program that causes the processor 3802 to perform a method described herein. For example, the application program 3810 can include applications 1 through N, including an image coding (encoding or decoding) application that performs a method described herein.

[0522] The apparatus 3800 can also include one or more output devices, such as a display 3818. In one example, the display 3818 can be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 3818 can be coupled to the processor 3802 via the bus 3812.

[0523] Although the bus 3812 in the apparatus 3800 is described herein as a single bus, the bus 3812 can include multiple buses. Also, the secondary storage can be directly coupled to the other components of the apparatus 3800 or can be accessed via a network, and can comprise a single integrated unit (such as one memory card) or multiple units (such as multiple memory cards). Thus, the apparatus 3800 can have a wide variety of configurations.

[0524] Generally, the present invention relates to a method and apparatus for encoding data (for still image or video processing) into a bitstream. Specifically, the data is processed by a network comprising a plurality of cascaded layers. During the processing, feature maps are generated by the layers. At least two different layers processing (outputting) the feature maps have different resolutions. During the processing, a layer is selected from the cascaded layers, which is different from the layer generating the lowest resolution feature map (e.g. latent space). The bitstream comprises information related to the selected layer. By this method, a scalable processing is provided that can be applied on different resolutions, so that the bitstream can signal information related to such different resolutions. Thus, the data can be efficiently coded in the bitstream depending on the resolution, which can vary depending on the content of the encoded image data.

[0525] The present application also relates to a method and apparatus for decoding data (for still image or video processing) from a bitstream. Specifically, two or more sets of feature map elements are obtained from the bitstream. Each set of feature map elements is associated with a feature map. Each of the two or more sets of feature map elements is respectively input into two or more feature map processing layers in a plurality of cascaded layers. Then, decoded data for image or video processing is obtained as a processing result of the plurality of cascaded layers. Thus, data can be decoded from the bitstream in an efficient manner in a hierarchical structure.

[0526] The present application also relates to a method and apparatus for decoding data (for still image or video processing) from a bitstream. Two or more sets of partition information elements are obtained from the bitstream. Then, each of the two or more sets of partition information elements is respectively input into two or more partition information processing layers in a plurality of cascaded layers. In each of the two or more partition information processing layers, the respective set of partition information is processed. Decoded data for image or video processing is obtained according to the partition information processed by the plurality of cascaded layers. Thus, data can be decoded from the bitstream in an efficient manner in a hierarchical structure.

Claims

1. A method for decoding data from a bitstream for image or video processing, characterized in that, The method includes: Obtaining two or more sets of segmentation information elements from the bitstream; Inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers respectively; In each of the two or more segmentation information processing layers, processing the respective segmentation information sets, wherein, Obtaining the decoded data for image or video processing is performed according to the segmentation information processed by the plurality of cascaded layers; For each segmentation information processing layer j in N segmentation information processing layers in the plurality of cascaded layers, N≥2, The input includes: if j = 1, inputting the initial segmentation information from the bitstream, otherwise inputting the segmentation information processed by the (j - 1)th segmentation information processing layer, N≥j≥1; Outputting the processed segmentation information.

2. The method according to claim 1, wherein, Obtaining the set of segmentation information elements is performed according to the segmentation information processed by at least one segmentation information processing layer in the plurality of cascaded layers.

3. The method according to claim 1 or 2, wherein, Inputting the set of segmentation information elements is performed according to the processed segmentation information output by at least one cascaded layer in the plurality of cascaded layers.

4. The method according to claim 1 or 2, wherein, The resolutions of the segmentation information processed respectively in the two or more segmentation information processing layers are different.

5. The method according to claim 1 or 2, wherein, The processing performed on the segmentation information in the two or more segmentation information processing layers includes upsampling.

6. The method according to claim 5, characterized in that, The upsampling of the segmentation information includes nearest neighbor upsampling.

7. The method according to claim 5, characterized in that, The upsampling of the segmentation information includes transposed convolution.

8. The method according to claim 1, characterized in that, For each layer i < N in the N segmentation information processing layers, the processing performed on the input segmentation information further includes: Parsing segmentation information elements from the bitstream, and associating the parsed segmentation information elements with the segmentation information output by the previous layer, wherein the position of the parsed segmentation information elements in the associated segmentation information is determined according to the segmentation information output by the previous layer.

9. The method according to claim 8, wherein, The number of segmentation information elements parsed from the bitstream is determined according to the segmentation information output by the previous layer.

10. The method according to claim 8 or 9, wherein, The parsed segmentation information elements are represented by a set of binarization flags.

11. The method according to any one of claims 1, 2, 6-9, characterized in that, Obtaining the decoded data for image or video processing includes determining at least one of the following according to the segmentation information: Intra-frame or inter-frame prediction mode; Image reference index; Single-reference or multi-reference prediction; Existence or non-existence of prediction residual information; Quantization step size; Motion information prediction type; Motion vector length; Motion vector resolution; Motion vector prediction index; Motion vector difference magnitude; Motion vector difference resolution; Motion interpolation filter; In-loop filter parameters; Post-filter parameters.

12. The method according to any one of claims 1, 2, 6-9, characterized in that, It further includes: The feature map element set is obtained from the bitstream, and according to the segmentation information processed by the segmentation information processing layer, the feature map element set is respectively input to the feature map processing layer in the multiple cascaded layers; Based on the feature maps of the multiple cascaded layers, the decoded data for image or video processing is obtained.

13. The method according to claim 12, characterized in that, At least one of the multiple cascaded layers is a segmentation information processing layer and a feature map processing layer.

14. The method according to claim 12, characterized in that, Each of the multiple cascaded layers is a segmentation information processing layer or a feature map processing layer.

15. A computer program product stored in a non-transient medium, characterized in that, When the computer program product is executed on one or more processors, it performs the method according to any one of claims 1 to 14.

16. An apparatus for decoding images or videos, characterized in that, It includes processing circuitry for performing the method according to any one of claims 1 to 14.

17. An apparatus for decoding data for image or video processing from a bitstream, characterized in that, The device includes: An acquisition unit is used to acquire two or more sets of segmentation information elements from the bitstream; The input unit is used to input each of two or more segmentation information element sets into two or more segmentation information processing layers in multiple cascaded layers; The processing unit is used to process the respective segmentation information set in each of two or more segmentation information processing layers; The decoded data acquisition unit is used to acquire the decoded data for image or video processing based on the segmentation information processed in the plurality of cascaded layers; The input unit is further configured to, for each of the N segmentation information processing layers j in the plurality of cascaded layers, where N≥2, if j=1, input the initial segmentation information from the bitstream; otherwise, input the segmentation information processed by the (j-1)th segmentation information processing layer, where N≥j≥1.

Images