Decoding by indicating feature map data

Abstract

The present invention also relates to a method and apparatus for decoding data (for still image or video processing) from a code stream. Specifically, two or more feature map element sets are obtained from the code stream. Each feature map element set is associated with a feature map. Then, each of the two or more feature map element sets is input into two or more feature map processing layers in a plurality of cascaded layers. Then, decoded data for image or video processing is obtained as a result of the processing of the plurality of cascaded layers. Therefore, the data can be decoded from the code stream in an efficient manner in a hierarchical structure.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of decoding data for image or video processing from a bitstream using multiple processing layers. In particular, some embodiments relate to methods and apparatus for such decoding. Background Art

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In these codecs, the signal is usually encoded block by block by predicting the block and by further decoding only the difference between the original block and its prediction. In particular, this decoding may include transform, quantization and generating a bitstream, usually including some entropy decoding. Usually, the three components of the hybrid decoding method, namely transform, quantization and entropy decoding, are optimized separately. Modern video compression standards (such as high-efficiency video coding (HEVC), versatile video coding (VVC) and essential video coding (EVC)) also use a transformed representation to decode the residual signal after prediction.

[0003] Recently, machine learning has been applied to image and video decoding. In general, machine learning can be applied to image and video decoding in a variety of different ways. For example, some end-to-end optimized image or video decoding schemes have been discussed. In addition, machine learning is used to determine or optimize some parts of end-to-end decoding, such as the selection of prediction parameters or compression. What these applications have in common is that they produce some feature map data that will be transmitted between the encoder and the decoder. The efficient structure of the bitstream can greatly help to reduce the number of bits required to encode the image / video source signal.

[0004] A neural network typically includes two or more layers. A feature map is the output of a layer. In a neural network that is partitioned between devices (e.g., between an encoder and a decoder, between a device and the cloud, or between different devices), the feature map of the output of the partitioning site (e.g., the first device) is compressed and sent to the remaining layers of the neural network (e.g., to the second device).

[0005] It is hoped that the trained network architecture can be used to further improve encoding and decoding. Summary of the invention

[0006] Some embodiments of the present invention provide methods and apparatus for decoding images in an efficient manner and achieving some scalability to accommodate required parameters and content.

[0007] The above and other objects are achieved by the subject matter claimed in the independent claims. Other implementations are obvious from the dependent claims, the description and the drawings.

[0008] According to an embodiment, a method for decoding data for image or video processing from a bitstream is provided, the method comprising: obtaining two or more feature map element sets from the bitstream, wherein each feature map element set is related to a feature map, inputting each of the two or more feature map element sets into two or more feature map processing layers in a plurality of cascade layers, respectively, and obtaining decoded data for image or video processing as a result of processing by the plurality of cascade layers.

[0009] This approach may provide improved efficiency because it enables data from different layers to be used for decoding, thereby enabling features or other types of layer-related information to be parsed from the codestream.

[0010] For example, a feature map is processed in each of the two or more feature map processing layers, wherein resolutions of the feature maps processed in the two or more feature map processing layers are different.

[0011] In some embodiments, processing the feature map in two or more feature map processing layers comprises upsampling.

[0012] The application of upsampling can reduce the complexity of processing on the one hand (because the first layer has a lower resolution), and on the other hand, it can also reduce the data provided in the bitstream and parsed at the decoder. In addition, layers processing different resolutions can focus on features of different scales in this way. As a result, networks processing images (still images or videos) can operate efficiently.

[0013] In an exemplary implementation, the method further includes the following steps: obtaining segmentation information related to the two or more layers from the code stream, wherein the feature map elements are obtained from the code stream based on the segmentation information; and inputting the feature map element sets into two or more feature map processing layers based on the segmentation information respectively.

[0014] Using segmentation information can provide efficient decoding of feature maps from different layers, so that each area of ​​the original resolution (to be reconstructed) can be covered by information from only one layer. Although this is not limited to the present invention, in some cases, the present invention can also provide overlap between layers for specific areas in the feature map (data). For example, the multiple cascaded layers also include multiple segmentation information processing layers, and the method also includes processing the segmentation information in the multiple segmentation information processing layers. This method provides the possibility of controlling the resolution of feature elements from different layers.

[0015] In some embodiments, processing the segmentation information in at least one of the plurality of segmentation information processing layers includes upsampling. The hierarchical structure of the segmentation information can provide a small amount of side information to be inserted into the bitstream, thereby improving efficiency and / or reducing processing time.

[0016] For example, the upsampling of the segmentation information and / or the upsampling of the feature map includes nearest neighbor upsampling. Nearest neighbor upsampling has a low computational complexity and can be easily implemented. Nevertheless, such upsampling is efficient, especially for logical indications such as flags.

[0017] In some embodiments and examples, the upsampling of the segmentation information and / or the upsampling of the feature map includes transposed convolution. Using convolution can help reduce blocking artifacts and can implement a trainable scheme, where the upsampling filter is selectable.

[0018] In an exemplary implementation, obtaining the feature map elements from the bitstream is based on the processed segmentation information processed by at least one of the plurality of segmentation information processing layers.

[0019] In an exemplary implementation, each of the two or more feature map element sets is separately input into two or more feature map processing layers based on the processed segmentation information processed by at least one of the plurality of segmentation information processing layers.

[0020] According to an embodiment, the obtained segmentation information is represented by a set of syntax elements, where the position of the elements in the set of syntax elements indicates which feature map element position the syntax element is related to. Processing the feature map includes, for each of the syntax elements: when the syntax element is a first value, parsing the element of the feature map at the position indicated by the position of the syntax element in the bitstream from the bitstream; otherwise, not parsing the element of the feature map at the position indicated by the position of the syntax element in the bitstream from the bitstream.

[0021] This relationship between the segmentation information and the feature map information enables efficient decoding of frequency information and parsing in a hierarchical structure by considering different resolutions.

[0022] For example, each layer j (1 < j < N) of the plurality of (N) feature map processing layers processing the feature map further includes: parsing the segmentation information element of the j-th feature map processing layer from the bitstream; obtaining the feature map processed by the previous feature map processing layer, parsing the feature map element from the bitstream, and associating the parsed feature map element with the obtained feature map, where the position of the feature map element in the processed feature map is indicated by the parsed segmentation information element, and the segmentation information is processed by the previous segmentation information processing layer.

[0023] Specifically, when the syntax element is a first value, the elements of the feature map are parsed from the code stream; when the syntax element is a second value or the segmentation information processed by the previous segmentation information processing layer is a first value, the elements of the feature map are not parsed from the code stream.

[0024] For example, the syntax element parsed from the code stream representing the segmentation information is a binarization flag. Specifically, the processed segmentation information is represented by a binarization flag set.

[0025] Providing binary flags allows efficient decoding. On the decoder side, the processing of logical flags can be performed with low complexity.

[0026] According to an exemplary implementation, upsampling the segmentation information in each segmentation information processing layer j also includes: for each p-th position in the acquired feature map indicated by the input segmentation information, determining an indication of the feature map position included in the same area as the p-th position in the reconstructed image as the upsampled segmentation information.

[0027] For example, data used for image or video processing includes motion vector fields. Since dense optical flow or motion vector fields with a resolution similar to that of the image are desired to model motion, the hierarchical structure of the present invention is easy to use to efficiently reconstruct such motion information. Through layer processing and indication, a good trade-off between rate and distortion can be achieved.

[0028] For example, the data for image or video processing includes image data and / or prediction residual data and / or prediction information data. The present invention can be used for various parameters. However, the image data and / or prediction residual data and / or prediction information data can still have some redundancy in the spatial domain, and the hierarchical method described herein can be used to efficiently decode from the code stream using different resolutions.

[0029] In some embodiments and examples, the upsampling of the feature map uses a filter, and the shape of the filter is any one of a square, a horizontal rectangle, and a vertical rectangle.

[0030] Applying different upsampling filters can help adapt to different characteristics of the content. For example, upsampling of the feature map uses a filter, and the input information from the bitstream also includes obtaining information indicating the filter shape and / or filter coefficients from the bitstream.

[0031] Correspondingly, the decoder can provide better reconstruction quality based on the information from the encoder transmitted in the bitstream.

[0032] For example, the information indicating the filter shape indicates a mask consisting of flags, the mask representing the filter shape, since a flag having a third value indicates a non-zero filter coefficient, and a flag having a fourth value different from the third value indicates a zero filter coefficient. This provides flexibility for designing filters of any shape.

[0033] For example, multiple cascaded layers include convolutional layers without upsampling between layers with different resolutions.

[0034] Providing such additional layers in a cascaded layer network enables the introduction of additional processing, such as various types of filtering, to improve the quality or efficiency of decoding.

[0035] According to an embodiment, a computer program product stored in a non-transitory medium is provided. When the computer program product is executed on one or more processors, any one of the above methods is performed.

[0036] According to an embodiment, there is provided a device for decoding an image or a video, the device comprising a processing circuit for executing a method according to any one of the above embodiments and examples.

[0037] According to one aspect, a device for decoding data for image or video processing from a bitstream is provided, the device comprising: an acquisition unit for acquiring two or more feature map element sets from the bitstream, wherein each feature map element set is related to a feature map, an input unit for inputting each of the two or more feature map element sets into two or more feature map processing layers in a plurality of cascade layers, respectively, and a decoded data acquisition unit for acquiring decoded data for image or video processing as a result of processing by the plurality of cascade layers.

[0038] Any of the above devices can be implemented on an integrated chip. The present invention can be implemented by hardware (HW) and / or software (SW). In addition, the HW-based implementation can be combined with the SW-based implementation.

[0039] It should be noted that the present invention is not limited to a specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0040] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0042] Figure 1 It is a schematic diagram of the channels processed by the layers of a neural network.

[0043] Figure 2 is a diagram of an autoencoder type of neural network.

[0044] Figure 3A is a schematic diagram of an exemplary network architecture including the encoder and decoder sides of the hyper-prior model.

[0045] Figure 3B is a schematic diagram of the general network architecture on the encoder side including the hyper-prior model.

[0046] Figure 3C is a schematic diagram of the general network architecture on the decoder side including the hyper-prior model.

[0047] Figure 4 is a schematic diagram of an exemplary network architecture including the encoder and decoder sides of the hyper-prior model.

[0048] Figure 5A A block diagram of a neural network-based end-to-end video compression framework.

[0049] Figure 5B is a block diagram of some exemplary details of a neural network application for motion field compression.

[0050] Figure 5C is a block diagram of some exemplary details of a neural network application for motion compensation.

[0051] Figure 6 It is a layer diagram of U-net.

[0052] Fig. 7A is a block diagram of an exemplary hybrid encoder.

[0053] Figure 7B is a block diagram of an exemplary hybrid decoder.

[0054] Figure 8 is a flow chart of an exemplary method for encoding data for image / video processing (eg, encoding).

[0055] Fig. 9 is a block diagram of the structure of a network that transmits information from layers of different resolutions in a bitstream.

[0056] Fig. 10A It is a schematic diagram of maximum pooling.

[0057] Fig. 10B It is a schematic diagram of average pooling.

[0058] Fig.11 is a schematic diagram of an exemplary encoder-side processing of feature maps and segmentation information.

[0059] Fig.12 is a block diagram of the generalized processing of motion information feature maps at the encoder and decoder sides.

[0060] Fig.13 is a block diagram of the structure of a network that transmits information from layers of different resolutions in a bitstream and is used to process motion vector related information.

[0061] Fig.14 is a block diagram of an exemplary cost computation unit with higher cost tensor resolution.

[0062] Fig.15 is a block diagram of an exemplary cost computation unit with lower cost tensor resolution.

[0063] Fig.16 is a block diagram illustrating the functional structure of the signal selection logic.

[0064] Fig.17 is a block diagram illustrating the functional structure of the signal selection logic, wherein the cost calculation unit provides several decoding options.

[0065] Fig.18 is a block diagram of the architecture of a network with convolutional downsampling and upsampling layers transmitting information at different resolutions in the bitstream.

[0066] Fig.19 is a block diagram of a structure for transmitting information from layers of different resolutions with additional layers in a codestream.

[0067] Fig. 20 is a block diagram of an architecture for transmitting information from layers of different resolutions in a bitstream, where the layers support downsampling or upsampling filter selection.

[0068] Fig.21 is a block diagram of the architecture of a network that transfers information from layers of different resolutions in a bitstream, where the layers support convolutional filter selection.

[0069] Fig. 22 is a block diagram illustrating the functional structure of a network-based RDO decision unit for selecting a decoding mode.

[0070] Fig.23 is a block diagram of an exemplary cost calculation unit that may be used in a network-based RDO decision unit for selecting a decoding mode.

[0071] Fig.24 is a block diagram of an exemplary cost calculation unit that may be used in a network-based RDO decision unit for selecting a decoding mode supporting multiple options.

[0072] Fig.25 is a schematic diagram of possible block partitions or filter shapes.

[0073] Fig.26 It is a schematic diagram of the derivation of segmentation information.

[0074] Fig. 27 is a schematic diagram of processing segmentation information at the decoder side.

[0075] Fig.28 is a block diagram of exemplary signal feeding logic for reconstructing dense optical flow.

[0076] Fig.29 is a block diagram of exemplary signal feeding logic for reconstructing dense optical flow.

[0077] Fig.30 is a block diagram of a set of convolutional filters.

[0078] Fig.31 is the block diagram of the upsampling filter bank.

[0079] Fig.32A is a schematic diagram of the upsampling process using nearest neighbor replication at the decoder side.

[0080] Fig.32B is a schematic diagram of the upsampling process using convolution processing at the decoder side.

[0081] Fig.33 is a flowchart of an exemplary method for decoding data, the data being feature map information used in image or video decoding, etc.

[0082] Fig.34 is a flowchart of an exemplary method for encoding data, the data being segmentation information used in image or video encoding, etc.

[0083] Fig.35 is a block diagram of an example of a video decoding system for implementing an embodiment of the present invention.

[0084] Fig.36 is a block diagram of another example of a video decoding system for implementing an embodiment of the present invention.

[0085] Fig.37 is a block diagram of an example of an encoding device or a decoding device.

[0086] Fig.38 is a block diagram of another example of an encoding device or a decoding device. DETAILED DESCRIPTION

[0087] In the following description, reference is made to the accompanying drawings that form a part of the present invention, which illustrate specific aspects of embodiments of the present invention or specific aspects of embodiments of the present invention that can be used by way of illustration. It should be understood that embodiments of the present invention can be used in other aspects and include structural or logical changes not shown in the accompanying drawings. Therefore, the following detailed description is not intended to be limiting, and the scope of the present invention is defined by the appended claims.

[0088] For example, it should be understood that what is disclosed with reference to a described method may also be true for a corresponding device or system for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units for performing the described one or more method steps (e.g., one unit that performs one or more steps; or multiple units, each unit performing one or more of the multiple steps), even if the one or more units are not explicitly described or shown in the drawings. In addition, for example, if a specific device is described in terms of one or more units (e.g., functional units), the corresponding method may include a step for performing the function of the one or more units (e.g., one step that performs the function of the one or more units, or multiple steps, each step performing the function of one or more of the multiple units), even if the one or more steps are not explicitly described or shown in the drawings. In addition, it should be understood that, unless otherwise specified, the features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0089] The purpose of some embodiments is to improve the quality of encoded and decoded image or video data and / or reduce the amount of data required to represent the encoded image or video data. Some embodiments provide efficient selection of information to be indicated from the encoder to the decoder. In the following, an overview of some used technical terms and frameworks of embodiments of the present invention is provided.

[0090] Artificial Neural Networks

[0091] An artificial neural network (ANN) or connectionist system is a computing system vaguely inspired by the biological neural networks that make up animal brains. These systems "learn" to perform tasks by considering examples, and are usually not programmed with task-specific rules. For example, in image recognition, these systems might learn to recognize images that include cats by analyzing example images that have been manually labeled as "cat" or "no cat" and using the results to recognize cats in other images. These systems do not have any prior knowledge about cats, such as whether these cats have fur, tails, whiskers, and cat-like faces. Instead, these systems automatically generate identifying features from the examples they are processed.

[0092] ANNs are based on a collection of connected units or nodes called artificial neurons, which loosely mimic neurons in biological brains. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. An artificial neuron receives a signal, processes it, and can send signals to neurons connected to it.

[0093] In the ANN implementation, the "signals" at the connections are real numbers, and the output of each neuron is calculated as some nonlinear function of the sum of its inputs. These connections are called edges. Neurons and edges usually have weights that are adjusted as learning occurs. Weights can increase or decrease the strength of the signal on a connection. Neurons can have a threshold so that they are only sent if the aggregate signal exceeds that threshold. Typically, neurons are grouped into layers. Different layers can perform different transformations on their inputs. Signals travel from the first layer (input layer) to the last layer (output layer), possibly traversing the layers multiple times during this time.

[0094] The original goal of the ANN approach was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, leading to a deviation from biology. ANNs have been used for a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even activities traditionally considered exclusive to humans, such as painting.

[0095] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its multiple layers.

[0096] Figure 1 Schematic diagram showing the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is where the input (e.g. Figure 1 The hidden layers of a CNN are usually composed of a series of convolutional layers, which perform convolutions using multiplication or other dot products. The result of the layer is one or more feature maps ( Figure 1 Some or all layers may involve subsampling. As a result, feature maps may become smaller, such as Figure 1As shown. The activation function in a CNN is typically a rectified linear unit (ReLU) layer, followed by additional convolutional (e.g. pooling) layers, fully connected layers, and normalization layers. These convolutions are called hidden layers because the inputs and outputs of these convolutions are masked by the activation function and the final convolution. By convention, the layers are colloquially called convolutions. Mathematically speaking, a convolution is technically a sliding dot product or cross correlation. This has important implications for the indexing in the matrix because the convolution affects how the weights are determined at a specific index point.

[0097] When programming a CNN to process images, such as Figure 1 As shown, the input is a tensor with the shape of (number of images) × (image width) × (image height) × (image depth). Then, after passing through the convolution layer, the image is abstracted into a feature map with the shape of (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolution layer in a neural network should have the following properties: The convolution kernel is defined by width and height (hyperparameters). The number of input channels and output channels (hyperparameters). The depth of the convolution filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0098] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, these nodes suffer from high dimensionality and do not scale well to higher resolution images. An image of 1000×1000 pixels with RGB color channels has 3 million weights, which is too large to be efficiently processed at scale with full connectivity. In addition, this network architecture does not take into account the spatial structure of the data, thereby treating input pixels that are far apart in the same way as close pixels. This ignores the reference locality in the image data both computationally and semantically. Therefore, full connectivity of neurons is wasteful for purposes such as image recognition dominated by spatially local input patterns.

[0099] Convolutional Neural Networks are biologically inspired variants of Multilayer Perceptrons specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of a CNN. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above) that have a small receptive field but extend to the full depth of the input volume. During the forward pass, each filter is convolved over the width and height of the input volume, computing the dot product between the filter entries and the input and generating a 2-dimensional activation map for that filter. Thus, the network learns filters that activate when certain specific types of features are detected at a certain spatial location in the input.

[0100] The activation maps of all filters stacked along the depth dimension form the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map or activation map is the output activation of a given filter. Feature map and activation have the same meaning. It is called an activation map in some papers because it is a map of activations corresponding to different parts of the image, and it is also called a feature map because it is also a map of where a certain feature is found in the image. A high activation means that a certain feature is found.

[0101] Another important concept of CNN is pooling, which is a form of nonlinear downsampling. There are several nonlinear functions that can implement pooling, among which maximum pooling is the most common. Maximum pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0102] Intuitively, the exact location of a feature is less important than its rough location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to progressively reduce the spatial size of the representation to reduce the number of parameters, memory footprint, and computation in the network, thus also controlling overfitting. In CNN architectures, it is common to insert pooling layers periodically between consecutive convolutional layers. The pooling operation provides another form of translation invariance.

[0103] Pooling layers operate independently on each depth slice of the input and spatially adjust these depth slices. The most common form is a pooling layer with a filter of size 2×2, which applies a stride of 2 to each depth slice of the input, along both the width and height, and discards 75% of the activations. In this case, each max operation is over 4 numbers. The depth dimension remains unchanged. In addition to max pooling, pooling units can also use other functions such as average pooling or l2-norm pooling. Average pooling was often used historically, but it has recently lost its advantage over max pooling, which generally performs better in practice. Due to the large reduction in the size of the representation, there has been a recent trend to use smaller filters or to discard pooling layers entirely. "Region of interest" pooling (also known as ROI pooling) is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is an important component of convolutional neural networks and is used for object detection based on the Fast R-CNN architecture.

[0104] The above ReLU is the abbreviation of Rectified Linear Unit, and ReLU applies a non-saturated activation function. ReLU sets negative values ​​to zero, effectively removing negative values ​​from the activation map. ReLU increases the nonlinear nature of the decision function and the overall network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, such as saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because ReLU trains neural networks several times faster without significantly affecting generalization accuracy.

[0105] After several convolutional and max pooling layers, high-level reasoning in a neural network is done with fully connected layers. Neurons in a fully connected layer have connections to all activations in the previous layer, as in a regular (non-convolutional) artificial neural network. Therefore, their activations can be calculated as an affine transformation, a matrix multiplication followed by a bias shift (a vector addition of a learned or fixed bias term).

[0106] The "loss layer" (including the calculation of the loss function) specifies how the training penalizes the deviation between the predicted label (output) and the true label, and is usually the last layer of the neural network. A variety of loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class among K mutually exclusive classes. Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0107] In conclusion, Figure 1 The data flow in a typical convolutional neural network is shown. First, the input image passes through a convolutional layer and is abstracted into a feature map including several channels, corresponding to multiple filters in the learnable filter set of the layer. The feature map is then subsampled using a pooling layer, etc., which reduces the dimension of each channel in the feature map. Next, the data reaches another convolutional layer, which can have a different number of output channels. As mentioned above, the number of input channels and output channels are hyperparameters of the layer. In order to establish the connection of the network, these parameters need to be synchronized between the two connected layers so that the number of input channels of the current layer should be equal to the number of output channels of the previous layer. For the first layer that processes the input data (such as an image), the number of input channels is usually equal to the number of channels represented by the data, such as 3 channels for RGB or YUV representation of images or videos, or 1 channel for grayscale image or video representation.

[0108] Autoencoders and Unsupervised Learning

[0109] An autoencoder is an artificial neural network that is used to learn efficient data encoding in an unsupervised manner. Figure 2As shown. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore the signal "noise", usually for dimensionality reduction. In addition to the dimensionality reduction side, a reconstruction side is also learned, where the autoencoder tries to generate a representation as close to its original input as possible from the simplified encoding, hence the name. In the simplest case, given one hidden layer, the encoder stage of the autoencoder takes the input x and maps it to h

[0110] h=σ(Wx+b).

[0111] This image h is often called a code, latent variable, or latent representation. Here, σ is an element-wise activation function, such as a sigmoid function or a rectified linear unit. W is the weight matrix and b is the bias vector. The weights and biases are usually initialized randomly and then iteratively updated via backpropagation during training. Afterwards, the decoder stage of the autoencoder maps h to a reconstruction x′ of the same shape as x:

[0112] x′=σ′(W′h′+b′)

[0113] Here, σ′, W′, and b′ of the decoder may be independent of the corresponding σ, W, and b of the encoder.

[0114] Variational autoencoder models make strong assumptions about the distribution of latent variables. These variational autoencoder models use a variational approach to latent representation learning, which results in an additional loss component and a specific estimator for the training algorithm, called the Stochastic Gradient Variational Bayes (SGVB) estimator. It is assumed that the data is represented by a directed graphical model p θ (x|h) is generated, and the encoder is learning the posterior distribution p θ The approximate value q of (h|x) φ (h|x), where φ and θ represent the parameters of the encoder (recognition model) and decoder (generation model), respectively. The probability distribution of the latent vector of a VAE is usually closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0115]

[0116] Here, D KL represents the KL divergence. The priors on the latent variables are usually set to centered isotropic multivariate Gaussian Typically, the shapes of the variational and likelihood distributions are chosen to be factorized Gaussians:

[0117]

[0118]

[0119] Among them, ρ(x) and ω 2 (x) is the encoder output, and μ(h) and σ 2 (h) is the decoder output.

[0120] Recent advances in the field of artificial neural networks, especially convolutional neural networks, have made researchers interested in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using a variational autoencoder-based network was proposed.

[0121] Data compression is therefore considered a fundamental and intensively studied problem in engineering, with the goal of designing a code for a given set of discrete data with minimum entropy. The scheme relies heavily on knowledge of the probabilistic structure of the data, and the problem is therefore closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0122] In this case, the lossy compression problem, two competing costs must be weighed: the entropy of the discrete representation (rate) and the error (distortion) introduced by quantization. Different compression applications (such as data storage or transmission over limited-capacity channels) require different rate-distortion tradeoffs.

[0123] Joint optimization of rate and distortion is difficult. The general problem of optimal quantization in high-dimensional space is intractable without further constraints. Therefore, most existing image compression methods perform the following operations: linearly transform the data vector into a suitable continuous-valued representation, independently quantize its elements, and then encode the resulting discrete representation using lossless entropy decoding. Due to the central role of the transform, this scheme is called transform coding.

[0124] For example, JPEG uses discrete cosine transform on pixel blocks, and JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of the transform coding method (transform, quantizer, and entropy decoding) are optimized separately (usually through manual parameter tuning). Modern video compression standards (such as HEVC, VVC, and EVC) also use transformed representations to encode the predicted residual signal. Several transforms such as discrete cosine transform (DCT), discrete sine transform (DST), and low frequency non-separable manually optimized transform (LFNST) are used for this purpose.

[0125] Variational Image Compression

[0126] The variational auto-encoder (VAE) framework can be considered as a nonlinear transformation decoding model. The transformation process can be divided into four parts: Figure 3A An example is given in Figure 2 and the VAE framework is shown.

[0127] The transformation process can be divided into four main parts. Figure 3A The VAE framework is illustrated in Figure 3A In the example, encoder 101 maps an input image x to a latent representation (represented by y) by a function y=f(x). Hereinafter, this latent representation may also be referred to as a portion or point of a "latent space". Function f() is a transformation function that transforms the input signal x into a more compressible representation y. Quantizer 102 Transform the latent representation y into a quantized latent representation with (discrete) values Where Q represents the quantizer function. The entropy model or super encoder / decoder (also called super prior) 103 estimates the quantized latent representation to obtain the minimum rate achievable by lossless entropy source coding.

[0128] The latent space can be understood as a compressed representation of the data, where similar data points are closer in the latent space. The latent space is very useful for learning data features and finding simpler representations of the data for analysis. Quantized latent representation T and hyperprior 3 and side information The use of arithmetic coding (AE) is included in the bitstream 2 (binarized). In addition, a decoder 104 is provided which transforms the quantized latent representation into a reconstructed image. Signal is an estimate of the input image x. We hope that x is as close as possible In other words, the reconstruction quality is as high as possible. However, The higher the similarity between x and x, the more side information is needed to send. Side information includes Figure 3A The bitstream 1 and bitstream 2 shown are generated by the encoder and sent to the decoder. Generally speaking, the greater the amount of side information, the higher the reconstruction quality. However, a large amount of side information means a low compression ratio. Therefore, Figure 3A One goal of the system described in is to balance the quality of reconstruction and the amount of side information transmitted in the bitstream.

[0129] exist Figure 3A , component AE 105 is an arithmetic coding module that converts the quantized latent representation Samples and side information The samples are converted into binary representation code stream 1. For example, Samples and The samples may include integers or floating point numbers. One purpose of the arithmetic coding module is to convert the sample values ​​into a binary digital string (through a binarization process) (which is then included in the codestream, which may include other parts corresponding to the encoded image or other side information).

[0130] Arithmetic decoding (AD) 106 is the process of restoring the binarization process, wherein the binarized numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106 .

[0131] It should be noted that the present invention is not limited to this specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0132] exist Figure 3A In this example, there are two subnets cascaded to each other. In this context, a subnet is a logical division between parts of the overall network. For example, in Figure 3A In FIG. 1 , modules 101 , 102 , 104 , 105 and 106 are referred to as the “encoder / decoder” subnet. The “encoder / decoder” subnet is responsible for encoding (generating) and decoding (parsing) the first bitstream “bitstream 1”. Figure 3A The second network in the example includes modules 103, 108, 109, 110 and 107 and is called the "super encoder / decoder" subnet. The second subnet is responsible for generating the second bitstream "Bitstream 2". The purposes of these two subnets are different.

[0133] The first subnet is responsible for:

[0134] Transform (101) the input image x to its latent representation y (which makes it easier to compress x),

[0135] Quantize (102) the latent representation y to a quantized latent representation

[0136] Arithmetic coding module 105 uses AE to compress the quantized latent representation To get the code stream "code stream 1",

[0137] · Parsing the code stream 1 via AD using the arithmetic decoding module 106, and

[0138] Reconstruct the image using the parsed data (104)

[0139] The purpose of the second subnet is to obtain the statistical properties of the samples of "code stream 1" (such as the average value, variance and correlation between the samples of code stream 1) so that the first subnet can more efficiently compress code stream 1. The second subnet generates a second code stream "code stream 2", which includes the information (such as the average value, variance and correlation between the samples of code stream 1).

[0140] The second network includes an encoding part, which includes converting the quantized latent representation Transform (103) into side information z, and quantize the side information z into quantized side information And the quantized side information The encoding (e.g., binarization) (109) is performed into the bitstream 2. In this example, the binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes arithmetic decoding (AD) 110, which transforms the input bitstream 2 into decoded quantized side information. Possibly The same, because arithmetic coding ends the decoding operation is a lossless compression method. Then, the decoded quantized side information is transformed (107) into the decoded side information express The statistical properties of The mean of the samples, or the variance of the sample values, etc.). Then, the decoded latent representation is provided to the arithmetic encoder 105 and the arithmetic decoder 106 to control The probability model of .

[0141] Figure 3AAn example of a VAE (Variational Autoencoder) is described, the details of which may vary in different implementations. For example, in a particular implementation, there may be additional components to more efficiently obtain statistical properties of the samples of bitstream 1. In one such implementation, there may be a context modeler whose goal is to extract cross-correlation information for bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and arithmetic decoder (AD) 106 components.

[0142] Figure 3A The encoder and decoder are shown in a single figure. As will be appreciated by those skilled in the art, the encoder and decoder can be and often are embedded in mutually different devices.

[0143] Figure 3B shows the encoder, Figure 3C The decoder component of the VAE framework is shown. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The output of the encoder (e.g. Figure 3B ) are code stream 1 and code stream 2. Code stream 1 is the output of the first subnet of the encoder, and code stream 2 is the output of the second subnet of the encoder.

[0144] Similarly, in Figure 3C In this example, two bitstreams (bitstream 1 and bitstream 2) are received as input and a reconstructed (decoded) image is generated at the output. As mentioned above, the VAE can be divided into different logical units that perform different operations. Figure 3B and Figure 3C An example is given in . Figure 3B The components involved in encoding a signal such as a video and providing encoding information are shown. Then, for example, the encoding information is composed of Figure 3C It should be noted that the components of the encoder and decoder denoted by the numbers 12x and 14x may correspond in their functions to those in the above Figure 3A Components mentioned in and denoted by the number 10x.

[0145] Specifically, if Figure 3B As shown in FIG. 1 , the encoder includes an encoder 121, which transforms an input x into a signal y and then provides the signal y to a quantizer 322. The quantizer 122 provides information to an arithmetic coding module 125 and a super encoder 123. The super encoder 123 provides the code stream 2 discussed above to a super decoder 147, which in turn provides information (125) to the arithmetic coding module 105.

[0146] The output of the arithmetic coding module is code stream 1. Code stream 1 and code stream 2 are the outputs of signal encoding, and then code stream 1 and code stream 2 are provided (sent) to the decoding process. Although unit 101 (121) is called an "encoder", it can also be called Figure 3B The complete subnetwork described in is called an "encoder". The encoding process usually refers to the unit (module) that converts the input into an encoded (e.g., compressed) output. Figure 3B It can be seen that unit 121 can actually be considered the core of the entire subnetwork, as it performs a transformation of the input x to y, which is a compressed version of x. Compression in encoder 121 can be achieved by applying a neural network or any processing network that generally has one or more layers. In such a network, compression can be performed by a cascade of processes including downsampling, which reduces the size and / or number of channels of the input. Therefore, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0147] The remaining parts in the figure (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all parts that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. The AE 125 in conjunction with the super encoder 123 and super decoder 127 used to configure the AE 125 can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to Figure 3B The entire subnetwork in is called an "encoder".

[0148] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into a binary number (bit). For example, in the VAE framework, the encoder is a nonlinear transformation that maps the input image x to y, where y has a smaller width and height than x. Since y has a smaller width and height, it is smaller in size, the dimensionality (size) of the signal is reduced, and therefore, it is easier to compress the signal y. It should be noted that, in general, the encoder does not necessarily need to reduce the size in two (or usually all) dimensions. On the contrary, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or usually a subset).

[0149] In J. Balle, L. Valero Laparra, and EPSimoncelli (2015), arXiv e-published: "Density Modeling of Images Using a Generalized Normalization Transformation" (published at the 4th International Conference on Learning Representations, 2016) (hereinafter referred to as "Balle"), the authors propose an end-to-end optimization framework for image compression models based on nonlinear transformations. The authors optimize for mean squared error (MSE), but use a more flexible transformation built from a cascade of linear convolutions and nonlinearities. Specifically, the authors use a generalized divisive normalization (GDN) joint nonlinearity that is inspired by models of neurons in biological visual systems and has been shown to be effective in Gaussianizing image density. This cascade of transformations is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric inverse nonlinear transform.

[0150] Such an example of a VAE framework is Figure 4 As shown, it uses 6 downsampling layers, labeled 401 to 406. The network architecture includes a hyper-prior model. a , g s ) shows the image autoencoder architecture, the right side (h a ,h s ) corresponds to the autoencoder that implements the hyper-prior. The decomposed prior model uses the same architecture for both analysis and synthesis transformations g a and g s Q represents quantization, AE and AD represent arithmetic encoder and arithmetic decoder respectively. The encoder puts the input image x into g a , producing a response y (latent representation) with spatially varying standard deviation. Encoding g a It consists of multiple convolutional layers with subsampling and generalized divisive normalization (GDN) as the activation function.

[0151] The response is fed to h a , thereby summarizing the standard deviation distribution in z. Then z is quantized, compressed, and sent as side information. The encoder then uses the quantized vector To estimate That is, the spatial distribution of the standard deviation (used to obtain the probability value (or frequency value) of arithmetic coding (AE)), and use it to compress and send the quantized image representation (or latent representation). The decoder first recovers the Then, it uses h s Get This provides it with the correct probability estimate to successfully recover Then, it will Feed to g s to obtain the reconstructed image.

[0152] In the layer descriptions, layers that include downsampling are indicated by a downward-pointing arrow. The description of a layer "Conv N×5×5 / 2↓" means that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, 2↓ means that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 causes one of the dimensions of the input signal to be reduced by half at the output. Figure 4 In the example, 2↓ indicates that the width and height of the input image are both reduced by half. Since there are 6 downsampling layers, if the width and height of the input image 414 (also denoted by x) are w and h, the width and height of the output signal z 413 are equal to w / 64 and h / 64 respectively. The modules denoted by AE and AD are the arithmetic encoder and arithmetic decoder, refer to FIG. 3A to FIG. 3C Explain. Arithmetic encoder and arithmetic decoder are specific implementations of entropy decoding. AE and AD can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation, which is a recoverable process. In addition, the "Q" in the figure corresponds to the above Figure 4 The quantization operation mentioned is explained in the above section "Quantization". In addition, the quantization operation and the corresponding quantization unit as part of the component 413 or 415 do not necessarily exist and / or can be replaced by another unit.

[0153] exist Figure 4 In the figure, a decoder including upsampling layers 407 to 412 is also shown. Another layer 420 is provided between the upsampling layers 411 and 410 in the processing order of the input, which layer 420 is implemented as a convolutional layer, but does not upsample the received input. A corresponding convolutional layer 430 for the decoder is also shown. Such a layer can be provided in the NN to perform operations on the input without changing the size of the input but changing specific characteristics. However, it is not necessary to provide such a layer.

[0154] When the processing order of the code stream 2 is passed through the decoder, the order in which the upsampling layers are passed is opposite, that is, from the upsampling layer 412 to the upsampling layer 407. Each upsampling layer is shown here to provide an upsampling with an upsampling ratio of 2, and the upsampling is represented by ↑. Of course, not all upsampling layers have the same upsampling ratio, and the upsampling layers can use other upsampling ratios such as 3, 4, 8, etc. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since these convolutional layers perform operations on the input opposite to those performed by the encoder, the upsampling layers can apply a deconvolution operation to the received input so that the size of the input increases by a factor corresponding to the upsampling ratio. However, the present invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample replication, etc.

[0155] In the first subnet, some convolutional layers (401 to 403) are followed by generalized divisive normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second subnet, the activation function applied is ReLu. It should be noted that the present invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLu.

[0156] End-to-end image or video compression

[0157] DNN-based image compression methods can take advantage of large-scale end-to-end training and highly nonlinear transformations, which are not used in traditional methods. However, it is not easy to directly apply these techniques to build an end-to-end learning system for video compression. First, learning how to generate and compress motion information tailored for video compression remains an open problem. Video compression methods heavily rely on motion information to reduce temporal redundancy in video sequences.

[0158] A simple solution is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate flow fields as accurately as possible. Accurate optical flow is often not the best choice for a specific video task. In addition, compared with the motion information in traditional compression systems, the amount of optical flow data increases significantly, and directly applying existing compression methods to compress optical flow values ​​will significantly increase the number of bits required to store motion information. Secondly, it is not yet known how to build a DNN-based video compression system by minimizing the rate-distortion-based residual and motion information objectives. The purpose of rate-distortion optimization (RDO) is to achieve higher quality (i.e., less distortion) of the reconstructed frame when the number of compressed bits (or bitrate) is given. RDO is very important for video compression performance. In order to take advantage of the ability of end-to-end training of learning-based compression systems, RDO strategies are needed to optimize the entire system.

[0159] In "DVC: An End-to-end Deep Video Compression Framework" by Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006–11015), the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual decoding.

[0160] This encoder is Figure 5A shown. Specifically, Figure 5A The overall structure of the end-to-end trainable video compression framework is shown. In order to compress motion information, a CNN is specified to transform the optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder style network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 2. Figure 5B As shown in Figure 2. Figure 4 The ga / gs in are somewhat similar. Specifically, the optical flow is fed into a series of convolution operations and nonlinear transformations including GDN and IGDN. The number of output channels of the convolution (deconvolution) except the last deconvolution layer is 128, and the number of output channels of the last deconvolution layer is equal to 2. Given an optical flow of size M×N×2, the MV encoder will generate a motion representation of size M / 16×N / 16×128. The motion representation is then quantized, entropy decoded and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0161] Figure 5C Here, the previously reconstructed frame x is used. t–1 The warping unit generates the warped frame (usually with the help of an interpolation filter, such as a bilinear interpolation filter) using the reconstructed motion information. Then, a separate CNN with three inputs generates the predicted image. The architecture of the motion compensated CNN is also as Figure 5C shown.

[0162] The residual information between the original frame and the predicted frame is encoded by the residual encoder network. A highly nonlinear neural network is used to transform the residual into the corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better utilize the power of nonlinear transformation and achieve higher compression efficiency.

[0163] From the above overview, it can be seen that CNN-based architectures can be applied to image and video compression, thereby considering different parts of the video framework, including motion estimation, motion compensation, and residual decoding. Entropy decoding is a popular method for data compression, widely adopted by the industry, and is also suitable for feature map compression for human perception or computer vision tasks.

[0164] Machine video decoding

[0165] Video coding for machine (VCM) is another popular direction in computer science today. The main idea behind this approach is to send a decoded representation of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video decoding aimed at human perception, the quality features are the performance of computer vision tasks, such as object detection accuracy, rather than reconstruction quality.

[0166] A recent study proposes a new deployment paradigm, called collaborative intelligence, where deep models can be partitioned between mobile and cloud. Extensive experiments on various hardware configurations and wireless connectivity modes show that the optimal operating point in terms of energy consumption and / or computational latency involves partitioning the model (usually at some point deep in the network). The current common schemes (models are either completely in the cloud or completely in mobile) are rarely, if ever, optimal. The concept of collaborative intelligence also extends to model training. In this case, data flows in both directions: from cloud to mobile during backpropagation in training and inference, and from mobile to cloud during forward pass in training and inference.

[0167] In the context of recent deep models for object detection, lossy compression of deep feature data has been studied based on HEVC intra-coding. It is noted that the detection performance degrades with increasing compression levels, and compression-augmented training has been proposed to minimize this loss by generating models that are more robust to quantization noise in the feature values. However, this is still a suboptimal solution as the codecs used are highly complex and optimized for natural scene compression rather than deep feature compression.

[0168] Through a method using the popular YOLOv2 network for object detection tasks, the trade-off between compression efficiency and recognition accuracy is studied, and the problem of deep feature compression for collaborative intelligence is addressed. Here, the term deep feature has the same meaning as feature map. The word "deep" comes from the collaborative intelligence idea when the output feature map of some hidden (deep) layers is captured and transmitted to the cloud to perform inference. This seems to be more efficient than sending compressed natural image data to the cloud and using the reconstructed image to perform object detection.

[0169] Efficient compression of feature maps is beneficial for image and video compression and reconstruction, both for human perception and machine vision. The shortcomings of state-of-the-art autoencoder-based compression methods also exist in machine vision tasks.

[0170] Artificial neural network with skip connections

[0171] A residual neural network (ResNet) is an artificial neural network (ANN) built on the known structure of pyramidal cells in the cerebral cortex. Residual neural networks achieve this by skipping some layers using skip connections or shortcuts. Typical ResNet models are implemented with two or three layers of skips that include nonlinearities (ReLU) and batch normalization in between. Additional weight matrices can be used to learn skip weights; these models are called highway networks (HighwayNet). Models with several parallel skips are called densely connected networks (DenseNet). In the context of residual neural networks, non-residual networks can be described as ordinary networks.

[0172] One motivation for skipping layers is to avoid the vanishing gradient problem by reusing the activations of the previous layer until the adjacent layers learn their weights. During training, the weights are adjusted to mute the upstream layers and amplify the previously skipped layers. In the simplest case, only the weights of the adjacent layer connections are adjusted, without adjusting the explicit weights of the upstream layers. This works best when a single non-linear layer is skipped, or when the intermediate layers are all linear. If not, an explicit weight matrix should be learned for the skipped connections (high-speed networks should be used).

[0173] Skipping effectively simplifies the network so that fewer layers are used during the initial training phase. This speeds up learning by reducing the effects of vanishing gradients because there are fewer layers to propagate through. The network then gradually recovers the skipped layers as it learns the feature space. Towards the end of training, when all layers have been expanded, it will remain closer to the manifold and thus learn faster. A neural network without the residual part can explore more of the feature space. This makes it more susceptible to perturbations that cause it to leave the manifold and require additional training data to recover.

[0174] U-Net introduces longer skip connections, such as Figure 6 The U-Net architecture is derived from the so-called "fully convolutional networks" first proposed by Long and Shelhamer. The main idea is to supplement the usual contracting networks with successive layers, where the pooling operation is replaced by an upsampling operator. Thus, these layers increase the resolution of the output. Furthermore, successive convolutional layers can learn to assemble a precise output based on this information.

[0175] An important modification in U-Net is that the upsampling part has a large number of feature channels, which supports the network to propagate contextual information to higher resolution layers. Therefore, the expansion path is more or less symmetric to the contraction path and produces a U-shaped architecture. The network only uses the valid part of each convolution without any fully connected layers. To predict pixels in the boundary areas of the image, the missing context is extrapolated by mirroring the input image. This tiling strategy is very important for applying the network to large images, because otherwise the resolution will be limited by the GPU memory.

[0176] Introducing skip connections can better capture features at different spatial resolutions, which has been successfully applied to computer vision tasks such as object detection and segmentation. However, implying such skip connections for image or video compression is not a trivial task, because the information from the encoding side needs to be transmitted in the communication channel, and the direct connection of the layers will require the transmission of a large amount of data.

[0177] Traditional hybrid video encoding and decoding

[0178] The neural network framework can also be used in combination or in conventional hybrid encoding and decoding, which will be illustrated later. In the following, a very brief overview of exemplary hybrid encoding and decoding is given.

[0179] Fig. 7A FIG. 2 is a schematic block diagram of an exemplary video encoder 20 for implementing the techniques of the present application. Fig. 7AIn the example of , the video encoder 20 includes an input terminal 201 (or an input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270 and an output terminal 272 (or an output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254 and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown in the figure). Fig. 7A The illustrated video encoder 20 may also be referred to as a hybrid video encoder or a video encoder according to a hybrid video codec.

[0180] The encoder 20 can be used to receive an image 17 (or image data 17) through an input terminal 201, for example, an image in an image sequence forming a video or a video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, it is referred to as image 17 in the following description. Image 17 can also be called a current image or a to-be-decoded image (specifically, in video decoding, it is to distinguish the current image from other images (e.g., previously encoded images and / or decoded images) in the same video sequence (i.e., a video sequence that also includes the current image)).

[0181] A (digital) image is or can be considered as a two-dimensional array or matrix of samples with intensity values. The samples in the array can also be called pixels (pixel / pel) (short for picture elements). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the size and / or resolution of the image. Three color components are usually used to represent colors, that is, an image can be represented by three sample arrays or an image can include three sample arrays. In the RGB format or color space, the image includes corresponding red, green and blue sample arrays. However, in video decoding, each pixel is usually represented in a brightness and chrominance format or color space, for example, the color space is YCbCr, which includes a brightness component represented by Y (sometimes also represented by L) and two chrominance components represented by Cb and Cr. The brightness component Y represents the brightness or grayscale intensity (for example in a grayscale image), while the two chrominance components Cb and Cr represent the chrominance or color information components. Accordingly, an image in the YCbCr format includes a brightness sample array (Y) of brightness sample values ​​and two chrominance sample arrays (Cb and Cr) of chrominance values. An image in RGB format can be converted or transformed into YCbCr format and vice versa, a process also referred to as color conversion or transformation. If the image is in black and white, the image may include only a luminance sample array. Accordingly, the image may be, for example, a luminance sample array in black and white format or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2 and 4:4:4 color formats.

[0182] An embodiment of the video encoder 20 may include an image segmentation unit ( Fig. 7A The image segmentation unit may be used to segment the image 17 into a plurality of (typically non-overlapping) image blocks 203 (not shown). These blocks may also be referred to as root blocks or macroblocks (H.264 / AVC standard) or as coding tree blocks (CTB) or coding tree units (CTU) (H.265 / HEVC and VVC standards). The image segmentation unit may be used to use the same block size for all images in a video sequence and a corresponding grid defining the block size, or to vary the block size between images or subsets or groups of images, and to segment each image into corresponding blocks. The abbreviation AVC stands for Advanced Video Coding.

[0183] In other embodiments, the video encoder may be configured to directly receive a block 203 of the image 17, for example, one, several or all blocks constituting the image 17. The image block 203 may also be referred to as a current image block or an image block to be encoded.

[0184] Although the size of the image block 203 is smaller than that of the image 17, like the image 17, the image block 203 is also or can also be considered as a two-dimensional array or matrix of samples with intensity values ​​(sample values). In other words, the block 203 may include, for example, one sample array (e.g., a brightness array in the case of the black and white image 17), three sample arrays (e.g., one brightness array and two chrominance arrays in the case of the color image 17), or any other number and / or type of arrays, depending on the color format applied. The number of samples of the block 203 in the horizontal and vertical directions (or axes) defines the size of the block 203. Accordingly, an image block may be an M×N (M columns×N rows) sample array, or an M×N transform coefficient array, etc.

[0185] exist Fig. 7A In the illustrated embodiment of the video encoder 20 , the video encoder 20 may be configured to encode the image 17 block by block, for example, performing encoding and prediction on each block 203 .

[0186] Fig. 7A The illustrated embodiment of the video encoder 20 may also partition and / or encode an image using slices (also referred to as video slices), wherein an image may be partitioned or encoded using one or more slices (typically non-overlapping), and each slice may include one or more blocks (e.g., CTUs).

[0187] Fig. 7A The illustrated embodiment of the video encoder 20 may also segment and / or encode an image using block groups (also referred to as video block groups) and / or blocks (also referred to as video blocks), wherein one or more block groups (usually non-overlapping) may be used to segment or encode an image, each block group may include one or more blocks (e.g., CTU) or one or more blocks, etc., wherein each block may be a rectangular shape, etc., and may include one or more blocks (e.g., CTU), such as complete or partial blocks.

[0188] Figure 7B 3 is an example of a video decoder 30 for implementing the technology of the present application. The video decoder 30 is used to receive the encoded image data 21 (e.g., the encoded code stream 21) encoded by the encoder 20, for example, to obtain a decoded image 331. The encoded image data or code stream includes information for decoding the encoded image data, such as data representing image blocks of encoded video slices (and / or partition groups or partitions) and related syntax elements.

[0189] The entropy decoding unit 304 is used to parse the code stream 21 (or generally referred to as the encoded image data 21), and perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded decoding parameters (not shown in FIG. 3), such as any or all of inter-frame prediction parameters (such as reference image index and motion vector), intra-frame prediction parameters (such as intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters and / or other syntax elements. The entropy decoding unit 304 can be used to perform a decoding algorithm or scheme corresponding to the encoding scheme described by the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters and / or other syntax elements to the mode application unit 360, and provide other parameters to other units of the decoder 30. The video decoder 30 can receive syntax elements at the video slice level and / or the video block level. In addition to or as a substitute for slices and corresponding syntax elements, block groups and / or blocks and corresponding syntax elements can also be received and / or used.

[0190] The reconstruction unit 314 (eg, adder or summer 314 ) may be configured to add the reconstructed residual block 313 to the prediction block 365 to obtain the reconstructed block 315 in the sample domain, eg, by adding sample values ​​of the reconstructed residual block 313 to sample values ​​of the prediction block 365 .

[0191] Figure 7B The embodiment of the video decoder 30 shown in FIG. 1 may be used to segment and / or decode a picture using slices (also referred to as video slices). A picture may be segmented into or decoded using one or more (usually non-overlapping) slices, each of which may include one or more blocks (e.g., CTUs).

[0192] In an embodiment, Figure 7B The video decoder 30 shown may be used to segment and / or decode an image using block groups (also referred to as video block groups) and / or blocks (also referred to as video blocks). An image may be segmented into one or more block groups (usually non-overlapping) or decoded using one or more block groups (usually non-overlapping); each block group may include one or more blocks (e.g., CTU) or one or more blocks, etc.; each block may be rectangular, etc., and may include one or more complete or partial blocks (e.g., CTU), etc.

[0193] Other variations of the video decoder 30 may be used to decode the encoded image data 21. For example, the decoder 30 may generate an output video stream without the loop filter unit 320. For example, for certain blocks or frames, a non-transform based decoder 30 may directly quantize the residual signal without the inverse transform processing unit 312. In another implementation, the video decoder 30 may combine the inverse quantization unit 310 and the inverse transform processing unit 312 into a single unit.

[0194] It should be understood that in the encoder 20 and the decoder 30, the processing result of the current link can be further processed and then output to the next link. For example, after interpolation filtering, motion vector derivation or loop filtering, the processing result of interpolation filtering, motion vector derivation or loop filtering can be further clipped or shifted.

[0195] Improve decoding efficiency

[0196] As mentioned above, image and video compression methods based on variational autoencoder methods lack spatially adaptive processing and object segmentation objectives to capture real object boundaries. Therefore, content adaptability is limited. In addition, for some types of video information, such as motion information or residual information, sparse representation and decoding are required to keep the indication overhead at a reasonable level.

[0197] Therefore, some embodiments of the present invention introduce segmentation information decoding and feature map decoding from different spatial resolution layers of the autoencoder to achieve content adaptability and sparse signal representation and transmission.

[0198] In some exemplary implementations, connections are introduced between encoder and decoder layers except for lower resolution layers (latent space), which are sent in the bitstream. In some exemplary implementations, only part of the feature maps of different resolution layers are provided in the bitstream to save bandwidth. For example, signal selection and signal feeding logic are introduced to select, send and use parts of feature maps from different resolution layers. On the receiver side, tensor combination logic is introduced to combine the output of the previous resolution layer with the information received from the bitstream corresponding to the current resolution layer.

[0199] In the following, some detailed embodiments and examples related to the encoder side and the decoder side are provided.

[0200] Coding method and device

[0201] According to an embodiment, a method for encoding data for image or video processing into a code stream is provided. The method comprises the steps of processing the data, and the processing of the data comprises generating feature maps in a plurality of cascaded layers, each feature map comprising a corresponding resolution, wherein the resolutions of at least two of the generated feature maps are different from each other.

[0202] In other words, the resolutions of two or more of the cascaded layers may be different from each other. Here, when referring to the resolution of a layer, it refers to the resolution of the feature map processed by the layer. In an exemplary implementation, it is the resolution of the feature map output by the layer. A feature map including a resolution means that at least a part of the feature map has the resolution. In a certain implementation, the entire feature map may have the same resolution. For example, the resolution of a feature map may be given by the number of feature map elements in the feature map. However, it may also be more specifically defined by the number of feature map elements in one or more dimensions (e.g., x, y; alternatively or in addition, the number of channels may be considered).

[0203] The term layer here refers to a processing layer. It does not necessarily have to be a layer with trainable or trained parameters (weights), like the layers of some neural networks mentioned above. Instead, a layer can represent a specific processing of the layer input to obtain the layer output. In some embodiments, a layer can be trained or trainable. Training here refers to machine learning or deep learning.

[0204] When referring to cascaded layers, it is meant that these layers have a certain predetermined order (sequence), and the input of the first layer (in a given order) is processed by the first layer and (then) the other layers in sequence according to the given order. In other words, the output of the jth layer is the input of the j+1th layer, j being an integer from 1 to the total number of cascaded layers. In a specific non-limiting example, for all possible values ​​of j, the j+1th layer includes (or has) the same or lower resolution as the jth layer. In other words, the resolution of the layer does not increase (for example, on the encoder side) with the sequence (order) of the cascade (processing order). However, it should be noted that the present invention is not limited to this particular cascaded layer. In some embodiments, the layers processed in cascade may also include layers that increase the resolution. In any case, there may be layers that do not change the resolution.

[0205] For example, a lower resolution of a feature map may mean that each feature map has fewer feature elements. For example, a higher resolution of a feature map may mean that each feature map has more feature elements.

[0206] The method also includes the steps of selecting a layer different from the layer generating the lowest resolution feature map among the multiple layers, and generating a code stream, including inserting information related to the selected layer into the code stream.

[0207] In other words, in addition to (or instead of) outputting the processing results of all layers in the cascade into the bitstream, information is also provided to another (selected) layer. There can be one or more selected layers. The information related to the selected layer can be any type of information, such as the output of the layer or some segmentation information of the layer (discussed later), or other information related to the feature map processed by the layer and / or the processing performed by the layer. In other words, in some examples, the information can be an element of a feature map and / or the position of an element within a feature map (within a layer).

[0208] The input to the cascade process is data for image or video processing. For example, this data may be related to predictive decoding, such as inter-frame or intra-frame prediction. It may be a motion vector or other parameters of the prediction, such as prediction mode or reference image or direction, or other decoding parts besides prediction, such as transform, filter, entropy decoding or quantization. Code stream generation may include any conversion that converts values ​​into bits (binarization), including fixed codewords, variable length codes or arithmetic decoding.

[0209] Here, an image may be a still image or a video image. An image refers to one or more samples, such as samples captured by a camera or generated by computer graphics, etc. An image may include samples representing brightness levels in grayscale, or may have several channels including one or more of a brightness channel, a chroma channel, a depth channel, or other channels. Image or video encoding may be any of hybrid decoding (e.g., similar to HEVC or VVC, etc.) or autoencoders as described above.

[0210] Figure 8 is a flow chart of the above method. Therefore, the method comprises step 810: processing input data. From the processed data, a portion is selected in a selection step 820 and included in the code stream in a generation step 830. Not all data generated in the processing step must be included in the code stream.

[0211] According to an exemplary implementation, processing further includes downsampling by one or more of the cascaded layers. Fig. 9 An exemplary network 900 implementing (performing in operation) such processing is shown in FIG.

[0212] Specifically, Fig. 9 Input data for image or video processing 901 is shown entering the network 900. The input data for image or video processing can be any type of data used for such processing, such as samples of images or videos, prediction modes, motion vectors, etc. as directly described above. Fig. 9The processing applied to input 901 in is performed by multiple processing layers 911 to 913, each of which reduces the resolution of each motion vector array. In other words, the cascaded layers 911 to 913 are downsampling layers. It should be noted that when a layer is referred to as a downsampling layer, it performs downsampling. There are embodiments in which downsampling layers 911 to 913 perform downsampling as the only task; and there may be embodiments in which downsampling layers 911 to 913 do not perform downsampling as the only task. Instead, downsampling layers can generally also perform other types of processing.

[0213] like Fig. 9 As shown in , in addition to the processed data inputs and outputs, the down-sampling layers 911 to 913 also have additional selection outputs leading to signal selection logic 920. It should be noted that the term "logic" here refers to any circuit that implements the function (here, signal selection). Signal selection logic 920 selects information from the selection output of any layer to include in the code stream 930. Fig. 9 In the example of , each of layers 911 to 913 downsamples the layer input. However, layers that do not apply downsampling can also be added between the downsampling layers. For example, these layers can process the input by filtering or other operations.

[0214] exist Fig. 9 In the example shown, the signal selection logic 920 selects information included in the bitstream from the outputs of layers 911 to 913. The goal of the selection may be to select information related to the reconstructed image or video from a plurality of feature maps output by different layers. In other words, the downsampling layer and the signal selection logic may be implemented as part of an encoder (image or video encoder). For example, the encoder may be Figure 3A The encoder 101 shown, Figure 3B Encoder 121, MV encoder network ( Figure 5A part of end-to-end compression in Figure 5B MV encoder, or according to Fig. 7A a portion of the encoder (e.g., the loop filter 220 or the mode selection unit 260 or a portion of the prediction units 244, 254), etc.

[0215] Fig. 9 Also included is a decoder side portion (which may be referred to as an extended path), including signal feed logic 940 and upsampling layers 951 to 953. The input to the encoder side is the bitstream 930. The output 911 is the reconstructed input 901, etc. The decoder side will be described below with greater delay.

[0216] For example, downsampling can be accomplished by max pooling, average pooling, or any other operation that results in downsampling. Another example of such an operation includes a convolution operation. Fig. 10AAn example of maximum pooling is shown. In this example, every four elements (adjacent 2×2 squares) of array 1010 are grouped together and used to determine one element in array 1020. Arrays 1020 and 1010 may correspond to feature maps in some embodiments of the present invention. However, the arrays may also correspond to portions of feature maps of the present embodiment. The fields (elements) in arrays 1020 and 1010 may correspond to elements of feature maps. In this image, feature map 1020 is determined by downsampling feature map 1010. The numbers in the fields of arrays 1010 and 1020 are exemplary only. For example, a field may also include motion vectors instead of numbers. Fig. 10A In the example of maximum pooling shown, the four fields in the upper left corner of array 1010 are grouped together and the maximum value of their values ​​is selected. This group of values ​​determines the upper left corner field of array 1020 by assigning the maximum value to the field. In other words, the largest of the four upper left corner values ​​of array 1010 is inserted into the upper left corner field of array 1020.

[0217] Alternatively, in some implementations, minimum pooling can be used. In minimum pooling, the field with the smallest value is selected instead of the field with the largest value. However, these downsampling techniques are just examples, and various downsampling strategies can be used in different embodiments. Some implementations can use different downsampling techniques in different layers, different regions within a feature map, and / or for different types of input data.

[0218] In some implementations, downsampling is performed using average pooling, in which the average of a set of feature map elements is calculated and associated with the corresponding field in the feature map of the downsampled feature map.

[0219] Fig. 10B An example of average pooling is shown. In this example, the feature map elements in the upper left of feature map 1050 are averaged, and the upper left element of feature map 1060 adopts the average value. Fig. 10B Three groups are also shown in FIG: upper right, lower right, and lower left.

[0220] In another embodiment, the convolution operation is used for downsampling in some or all layers. In convolution, the filter kernel is applied to a group or block of elements in the input feature map. The kernel itself can be an array of elements with the same size as the input element block, wherein each element of the kernel stores the weight of the filter operation. In downsampling, the sum of the elements in the input block is calculated, and each element is weighted with a corresponding value obtained from the kernel. If the weights of all elements in the kernel are fixed, the convolution can correspond to the filter operation described above. For example, a convolution with a kernel of the same, fixed weight and kernel size step corresponds to an average pooling operation. However, the step size of the convolution used in this embodiment may be different from the kernel size, and the weights may be different. In one example, the kernel weights may enable certain features in the input feature map to be enhanced or distinguished from each other. In addition, the weights of the kernel may be learnable or learned in advance.

[0221] According to an embodiment, the information related to the selected layer includes an element 1120 of a feature map of the layer. For example, the information may transmit feature map information. In general, a feature map may include any feature related to a motion picture.

[0222] Fig.11 An exemplary implementation is shown, where the feature map 1110 is a dense optical flow of motion vectors with width W and height H. The motion segmentation network 1140 includes three downsampling layers (e.g., corresponding to Fig. 9 The downsampling layers 911 to 913 in the figure) and the signal selection circuit (logic) 1100 (for example, corresponding to the signal selection logic 920). Fig.11 Examples of the outputs of different layers (L1 to L3) in the contraction path on the right are shown.

[0223] In this example, the output of each layer (L1 to L3) is a feature map of decreasing resolution. The input of L1 is the dense optical flow 1110. In this example, one element of the feature map output from L1 is determined from sixteen (4×4) elements of the dense optical flow 1110. Each square ( Fig.11 The lower right corner) corresponds to a motion vector obtained by downsampling (downspl4) from the 16 motion vectors of the dense optical flow. For example, the downsampling can be average pooling or another operation, as discussed above. In this exemplary implementation, only a portion of the feature map L1 of the layer is included in the information 1120. Layer L1 is selected, and the portion corresponding to the four motion vectors (feature map elements) associated with the selected layer is indicated in the selected information 1120.

[0224] Then, the output L1 of the first layer is input to the second layer (downspl2). The output L2 feature map element of the second layer is determined from the four elements of L1. However, in other examples, each element of the feature map with lower resolution can also be determined by a group consisting of any other number of elements of the feature map with the next higher resolution. For example, the number of elements in the group that determines an element in the next layer can also be any power of 2. In this example, the output L2 feature map corresponds to three motion vectors, which are also included in the selected information 1120, so the second layer is also the selected layer. The third layer (downspl2) downsamples the output L2 of the second layer by a factor of 2 in each of the two dimensions. Accordingly, based on the four elements of L2, a feature map element of the output L3 of the third layer is obtained. In the feature map L3, no element is indicated, that is, the third layer is not the selected layer in this example.

[0225] The signal selection module 1100 of the motion segmentation network 1140 selects the above motion vectors (elements of the feature maps from the outputs of the first and second layers) and provides them to the code stream 1150. This provision can be a simple binarization, which can, but does not necessarily, include entropy decoding.

[0226] Groups of elements can be arranged in a square shape, as in Fig.11 However, the groups may also be arranged in any other shape, such as a rectangular shape, wherein the long sides of the rectangular shape may be arranged in the horizontal or vertical direction. These shapes are just examples. In one implementation, any shape may be used. Such a shape may also be indicated in the code stream 1150. The indication may be implemented by a flag indicating which feature elements belong to the shape and which do not belong to the shape. Alternatively, the indication may be accomplished using a more abstract description of the shape.

[0227] In this exemplary implementation, feature map elements are grouped so that each element belongs to exactly one group of elements that determines an element of a feature map of the next layer. In other words, groups of feature map elements are non-overlapping, with only one group contributing to feature map elements of a higher (later in the cascade processing order) layer. However, it is conceivable that an element of one layer may contribute to more than one element of the next layer. In other words, in process 810, a filter operation may be used when generating a new layer output (e.g., layer output L2) based on layer output L1 having a higher resolution.

[0228] In this embodiment, selection 820 (e.g., by signal selection 1100) selects elements to be included in the codestream from the plurality of output feature maps (L1 to L3). The selection may be implemented so that the amount of data required to indicate the selected data is low while keeping the amount of information relevant to decoding as large as possible. For example, rate-distortion optimization or other optimization may be used.

[0229] The example described above shows three layers of processing. In general, the method is not limited to this. Any number of processing layers (one or more) can be used. In other words, according to a more general example, the method includes obtaining data to be encoded. This can be a dense stream 1110 of motion vectors, as shown above. However, the present invention is not limited to this, and other data such as prediction mode, prediction direction, filtering parameters, and even spatial image information (samples) or depth information can be processed instead of or in addition to motion vectors.

[0230] The processing 810 of the data to be encoded in this example includes processing by each layer j of a plurality of (N) cascaded layers. The processing of the jth layer includes:

[0231] - If j = 1, then get the data to be encoded as layer input, otherwise get the feature map processed by the (j-1)th layer as layer input (i.e., if the jth layer is the currently processed layer, then j-1 is the previous layer);

[0232] - Processing the obtained layer input, which includes downsampling;

[0233] -Output downsampled feature map.

[0234] In this example, j=1 is the layer with the highest resolution among the N processing layers. It should be noted that the input of this layer can be a dense optical flow (which can also be regarded as a feature map in a general way). Therefore, in some specific embodiments, the layer with j=1 can be the input layer. However, this is not necessarily the case, because it is conceivable that there are some pre-processing layers before the N processing layers. A typical feature of an encoder is that earlier processing layers have a higher resolution than later processing layers (contraction path). Correspondingly, this situation can be reversed on the decoder side. It is conceivable that some processing layers do not change the resolution, or even do not enhance the resolution, and the present invention can still be applied.

[0235] In the example described above, the bitstream 1150 carries the selected information 1120. For example, it can be a motion vector or any other feature. In other words, the bitstream 1150 carries feature map elements from at least one layer that is not an output layer of the processing network (the encoder side processing network). Fig.11In the example of , only a portion of the selected feature map is transmitted in the code stream. The portion has one or more characteristic elements. In order to enable the decoder to determine which portion of the feature map is sent, rules for determination can be defined. In some embodiments, segmentation information can be transmitted in the code stream 1150 to configure which portions of the feature map are transmitted. This exemplary embodiment will be described below. However, it should be noted that the embodiments described above are merely exemplary and that such additional signaling is generally not required because there may be rules for deriving information that depend on other known or indicated parameters.

[0236] In an exemplary embodiment related to segmentation information, the information related to the selected layer includes (in addition to or alternatively to the selected information 1120) information 1130 indicating from which layer of the feature map of the layer and / or from which portion of the feature map of the layer the elements of the feature map of the layer are selected.

[0237] exist Fig.11 In the example shown, the segmentation information is shown by a binarization flag. For example, on the right, each lower resolution feature map or feature map portion is assigned 0 or 1. For example, L3 is assigned zero (0) because it is not selected and no motion vector (feature element) is indicated for L3. Feature map L2 has four parts. Layer processing L2 is the selected layer. The feature map elements (motion vectors) of three of the four parts are indicated, and correspondingly, the flag is set to 1. The remaining part of the feature map L2 does not include a motion vector, so the flag is set to 0 because the motion vector corresponding to the part is indicated by the L1 feature map. Since L1 is the first layer, it is implied that the remaining motion vectors are provided in this layer. It should be noted that when the corresponding feature map portion is partially selected information, the binarization flag here adopts a first value (e.g., 1), and when the corresponding feature map portion is not partially selected information, a second value (e.g., 0) is adopted. Since it is a binarization flag, it can only take one of these two values.

[0238] This segmentation information can be provided in the codestream. Fig.11The processing of the segmentation information 1130 is shown on the left side. It should be noted that the segmentation information 1130 can also be processed by the layers of the motion segmentation network 1140. It can be processed in the same layer as the feature map or in a separate layer. The segmentation information 1130 can also be explained as follows. One superpixel of the layer with the lowest resolution covers 16×16 cells of the feature map obtained by downsampling the dense optical flow 1110 by downspl4. Since the flag assigned to the superpixel covering the 16×16 cells is set to 0, this means that the feature map elements (here, motion vectors) are not indicated for this layer (this layer is not selected). Therefore, the feature map elements can be indicated in the area corresponding to 16×16 cells of the next layer, which is represented by four superpixels of equal size, each superpixel covering 8×8 feature elements cells. Each of these four superpixels is associated with a flag. For those superpixels associated with a flag having a value of 1, the feature map elements (motion vectors) are indicated. For superpixels with the flag set to 0, the motion vectors are not indicated. The unindicated motion vectors are indicated for the layer of superpixels with cells covering 4×4 elements.

[0239] More generally, the method for encoding data for image / video decoding may also include selecting (segmenting) information to insert into the bitstream. This information relates to the first region (superpixel) in the feature map processed by layer j (j > 1). The first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j that includes multiple elements. The method also includes the step: in the selection made from the feature map processed by layer k, the region corresponding to the first region is not selected, where k is an integer equal to or greater than 1 and k < j. The correspondence of regions between different layers means here that the corresponding regions (superpixels) spatially cover the same feature elements (initial data elements) in the feature map (initial data) to be encoded. In Fig.11 the example of, the initial data to be segmented is L1 data. However, the correspondence can also be obtained by referring to the dense optical flow 1110.

[0240] By Fig.11The specific arrangement in can ensure that each feature element of the initial feature map (e.g., L1) is covered by super pixels of only one layer in the N layers. This arrangement provides the advantage of efficiently decoding feature maps and segmentation information. The cascade layer processing framework corresponds to a neural network processing framework, in which data can be segmented in this way and data of each segmented part of different resolutions can be provided. In particular, the advantages of downsampling in some layers may include reducing the amount of data required to indicate the representation of the initial feature map. In particular, in the example of indicating motion vectors, due to downsampling, groups of similar motion vectors can be indicated by a common motion vector. However, in order to achieve good inter-frame prediction, the prediction error caused by the grouped motion vectors should be small. This may mean that for different areas of the image, different grouping levels of motion vectors may be the best level to achieve the desired prediction quality, while requiring a small amount of data to indicate the motion vector. This can be achieved using multiple layers with different resolutions.

[0241] In an embodiment where the feature map element is a motion vector, for the purpose of downsampling, the length and direction of the motion vector can be averaged, and the average motion vector is associated with the corresponding feature map element of the downsampled feature map. In the usual average, all elements of the element group corresponding to an element in the downsampled feature map have the same weight. This corresponds to applying a filter with equal weights to an element group or element block to calculate the downsampled feature map element. However, in other implementations, the filter can have different weights for different elements in the layer input. In other implementations, the median of the corresponding element group can be calculated instead of calculating the average value of the element group or element block in the downsampling.

[0242] exist Fig.11 In the example, the downsampling filter operation uses a square filter of size 2×2 for the input element and computes the filter output that maps to one element in the downsampled feature map according to the selected filter operation. The filter operation uses a stride of 2 equal to the edge length of the square filter. This means that between two filter operations, the filter is moved by a stride equal to the filter size. Therefore, in downsampling, the downsampled elements are computed based on non-overlapping blocks in the layer where the downsampling filter is applied.

[0243] However, in some further conceivable embodiments, the step size may be different from the length of the edge of the filter. For example, the step size may be smaller than the length of the edge of the filter. Thus, the filter blocks used to determine the elements in the downsampled layer may overlap, which means that one element in the feature map to be downsampled contributes to the calculation of more than one element in the downsampled feature map.

[0244] Typically, the data associated with the selected layer includes an indication of the location of the feature map element in the feature map of the selected layer. Fig.11 The concept of (feature maps L1 to L3), the feature map of the selected layer refers to the output from the selected layer, that is, the feature map processed by the selected layer.

[0245] For example, the positions of selected and unselected feature map elements are indicated by a plurality of binary flags, which are based on the positions of the flags in the bitstream. Fig.11 In the description of , the binarization flag is included in the code stream 1150 as segmentation information 1130. In order for the decoder to parse and correctly interpret the segmentation information, the flags within the feature maps processed by the layers and the allocation between the layers and / or regions should be defined. This can be achieved by defining a flag binarization order that is known to both the encoder and the decoder.

[0246] The above examples have been provided for the case where the data used in encoding the image / video is a motion vector. However, the present invention is not limited to such embodiments. In one embodiment, the data to be encoded includes image information and / or prediction residual information and / or prediction information. The image information here refers to the sample values ​​of the original image (or the image to be decoded). The sample values ​​may be samples of one or more color channels or other channels.

[0247] The information related to the selected layer is not necessarily a motion vector or a motion vector of a super pixel. In addition or alternatively, in some embodiments, the information includes prediction information. The prediction information may include a reference index and / or a prediction mode. For example, a reference index may indicate which specific image from a reference image set should be used for inter-frame prediction. The index may be relative to the current image where the current block to be predicted is located. For example, the prediction mode may indicate whether to use a single reference frame or multiple reference frames and / or a combination of different predictions, such as combined intra-frame-inter-frame prediction, etc.

[0248] However, when the data to be encoded is a motion vector field, efficient motion vector field decoding and reconstruction can be achieved. Fig.12 The corresponding general block scheme of a device that can perform such encoding and decoding of motion fields is shown. On the encoding side, the motion information is obtained using some motion estimation or optical flow estimation module (unit) 1210. The input to the motion vector (optical flow) estimation is the current image and one or more reference images (stored in a reference image buffer). Fig.12In the optical flow estimation unit 1210, images are called "frames", a term sometimes used for images of video. The optical flow estimation unit 1210 outputs an optical flow 1215. In different implementations, the motion estimation unit may output motion information already with different spatial resolutions for some N×N blocks or each pixel of the original resolution, etc., which may be referred to as dense optical flow. The motion vector information is to be sent (embedded in the bitstream 1250) to the decoding side and used for motion compensation. In order to obtain a motion compensated area, each pixel of the area should have a defined motion vector. Sending motion vector information for each pixel of the original resolution may be too costly. In order to reduce the indication overhead, a motion specification (or segmentation) module 1220 is used. The corresponding module 1270 on the decoding side performs the motion generation (dense) task to reconstruct the motion vector field 1275. The motion specification (or segmentation) module 1220 outputs motion information (e.g., motion vectors, and / or possible reference images) and segmentation information. This information is added (encoded) to the bitstream.

[0249] In this embodiment, the motion segmentation unit 1220 and the motion generation unit 1270 only include a downsampling layer dwnspl and a corresponding upsampling layer upspl, such as Fig.13 As shown. The nearest neighbor method can be used for downsampling and upsampling; average pooling can be used for downsampling. Feature map data from layers of different spatial resolutions are selected by the encoder and sent in the bitstream as selected information 1120 together with segmentation information 1130, which instructs the decoder how to interpret and utilize the selected information 1120. Motion segmentation (sparseness) network 1220 is used in Fig.13 1310. Thus, the dense optical flow 1215 is inserted into the motion segmentation (thinning) network 1310. The network 1310 includes three downsampling layers and signal selection logic 1320, which selects the information to be included in the code stream 1350. This functionality is similar to that already mentioned with reference to the more general Fig. 9 The functions discussed are similar.

[0250] In the above embodiments, the indication of information related to a layer different from the output layer improves the scalability of the system. The information may be information related to a hidden layer. In the following, embodiments and examples are given that involve exploiting the scalability and flexibility provided. In other words, some methods are provided on how to select layers and information appearance.

[0251] Some embodiments herein illustrate image or video compression systems using an autoencoder architecture that includes one or several dimensionality (or spatial resolution) reduction steps in the encoding part (implemented by layers combined with downsampling operations). Together with the reduction (encoding) side, a reconstruction (decoding) side is learned, wherein the autoencoder tries to generate a representation as close as possible to its original input from the reduced encoding, which usually means one or several resolution increase steps on the decoding side (implemented by layers combined with upsampling operations).

[0252] Here and below, under encoder, it refers to the encoding part of the autoencoder, which generates the potential signal representation included in the bitstream. For example, the encoder is 101 or 121 mentioned above. Under decoder, it refers to the generating part of the autoencoder, which perceives the potential signal representation obtained from the bitstream. For example, the decoder is the decoder 104 or 144 mentioned above.

[0253] As reference Fig.11 As described above, the encoder selects a portion (or multiple portions) of feature map information (selected information 1120) from layers of different spatial resolutions according to the signal selection logic 1100, and sends the selected information 1120 in the bitstream 1150. The segmentation information 1130 indicates which layer and which portion of the feature map of the corresponding layer to obtain the selected information.

[0254] According to an embodiment, the processing of the jth layer in the plurality of (N) cascaded layers includes:

[0255] - determining a first cost incurred by reconstructing a portion of the reconstructed image using the feature map elements output by the j-th layer;

[0256] - determining a second cost incurred by reconstructing a portion of the image using elements of the feature map output by the (j–1)th layer;

[0257] - If the first cost is higher than the second cost, the (j-1)th layer is selected, and information related to the portion in the (j-1)th layer is selected.

[0258] The decision which layer to select may be performed according to the distortion or based on a distortion function.For example, for motion vector field decoding, the reconstructed image (or image portion) may be a motion compensated image (or image portion).

[0259] In order to select the selected information in this exemplary implementation, the encoder includes a cost calculation unit (module) that estimates the cost of sending motion information from a specific resolution layer at a specific location. The cost is calculated by combining the distortion caused by motion compensation using the selected motion vector with an estimate of the amount of bits required to send the motion information multiplied by a Lagrangian multiplier. In other words, according to an embodiment, rate-distortion optimization (RDO) is performed.

[0260] In other words, in some embodiments, the first cost and the second cost include the amount of data and / or the distortion. For example, the amount of data includes the amount of data required to send the data associated with the selected layer. This can be motion information or other information. It can also be or include overhead caused by residual decoding. The distortion is calculated by comparing the reconstructed image with the target image (the original image to be encoded or a part of such an image to be encoded). It should be noted that RDO is only one possibility. The present invention is not limited to this method. In addition, complexity or other factors may be included in the cost function.

[0261] Fig.14 The first part of the cost calculation is shown. Specifically, the cost calculation (or estimation) unit 1400 obtains the optical flow L1 downsampled by the downsampling layer (downspl4) of the motion segmentation unit 1140. Then, the cost calculation unit 1400 upsamples the optical flow 1415 to its original resolution, for example, in this case, it performs an upsampling of 4 in each of the two directions (x and y). Then, motion compensation 1420 is performed using the upsampled motion vector output from 1410 and the reference image 1405 to obtain a compensated frame (image) or a portion of the motion compensated frame (image) 1420. Then, the distortion 1430 is calculated by comparing the motion compensated image (portion) 1420 with the target image 1408. For example, the target image 1408 can be an image to be decoded (original image). In some exemplary implementations, the comparison can be performed by calculating the mean squared error (MSE) or the sum of absolute difference (SAD) between the target image 1408 and the motion compensated image 1420. However, other types of metrics may be used alternatively or in addition, such as more advanced metrics targeting subjective perception, such as MS-SSIM or VMAF. The calculated distortion 1430 is then provided to a cost calculation module 1460 .

[0262] In addition, the rate estimation module 1440 calculates an estimate of the amount of bits for each motion vector. The rate estimate may include not only bits for indicating the motion vector, but also bits for indicating the segmentation information (in some embodiments). The number of bits obtained may be normalized (1450), for example per pixel (feature map element). The resulting rate (amount of bits) is provided to the cost calculation module 1460. In order to obtain the rate (amount of bits) estimate, an estimate of the amount of bits transmitted for each motion vector is performed, for example, using a motion information decoding module (e.g., by performing decoding and recording the amount of bits generated), or in some simplified implementations, using the length of the motion vector of its x or y component as a rough estimate. Another estimation technique may be applied. In order to take into account the segmentation information, it may be estimated by a segmentation information decoding module (e.g., by generating and decoding the segmentation information and counting the number of bits obtained), or in a simpler implementation, by adding the bits to the total amount of bits.

[0263] In this example, the next step of the cost calculation is cost calculation 1460, followed by downsampling 1470 by a factor of 4 (downspl 4) to reach the resolution of the corresponding downsampling layer of the motion segmentation unit 1100. Only one motion vector can be sent per point (image sample value). Therefore, the resulting cost tensor can have a corresponding size (dimension). Therefore, the bit evaluation value can be normalized by the square of the downsampling filter shape (e.g., 4×4).

[0264] Then, the cost estimation unit 1460 uses the Lagrange multiplier to calculate the cost using the following formula:

[0265] Cost = D + λ*R, or

[0266] Cost=R+β*D

[0267] Where D represents the distortion (calculated by 1430), R is a bit estimate (a rate estimate output by 1440 or 1450), and λ and β are Lagrangian multipliers. Downsampling 1470 outputs a cost tensor 1480. As is known in the art, the Lagrangian multipliers and λ and β can be obtained empirically.

[0268] Thus, a tensor 1480 with cost estimates for each position in the feature map (in this case, W×H positions of dense optical flow) is obtained. It should be noted that the result of using sequential average pooling and upsampling using the nearest neighbor method is to average the motion vectors in an N×N (e.g., 4×4) area, where N×N is the average pooling filter shape and scaling factor of the upsampling operation. During upsampling using the nearest neighbor method, the values ​​of the lower resolution layer are replicated (repeated) in all points of the higher resolution layer corresponding to the filter shape. This corresponds to a translational motion model.

[0269] Various implementations of the cost selection unit are possible. For example, Fig.15 Another exemplary implementation is shown. In this example, Fig.14 Differently, the motion vector field obtained after downsampling the dense optical flow by a factor of 4 in each of the x and y dimensions (1501) is not upsampled (1415). Instead, it is directly provided to the motion compensation 1510 and the estimated rate 1540. Instead, the reference image 1505 and the target image 1505 may be downsampled (1515, 1518) to corresponding resolutions before the motion compensation 1510 and the distortion estimation 1530. This results in the exclusion of Fig.14 The initial motion field upsampling (1415) to the original resolution in , and excludes Fig.14 The final cost downsampling step 1470 in . This also makes the bit normalization 1450 unnecessary. This implementation may require less memory to store tensors during processing, but may provide less accurate results. It should be noted that in order to speed up or reduce the complexity of RDO, it is conceivable to downsample the dense optical flow and the reference and target images even more than what is done in L1. However, the accuracy of such RDO may be further reduced.

[0270] The result of applying the cost estimation unit (1400, 1500) to each down-sampling layer of the motion segmentation unit (1220, 1310) is to obtain costs with different motion vector average levels (different spatial resolutions). In the next step, the signal selection logic 1100 uses the cost information from each down-sampling layer to select motion information of different spatial resolutions. In order to enable the signal selection logic 1100 to perform a pairwise comparison of costs from sequential (cascaded) down-sampling layers, the signal selection logic 1100 selects the minimum cost at each spatial location and propagates it to the next down-sampling layer (in the processing sequence). Fig.16 An exemplary architecture of the signal selection unit 1600 is shown.

[0271] The dense optical flow 610 enters three downsampling layers: downspl4, downspl2 and downspl2, similar to Fig.11 The layers shown in . Fig.16 The signal selection logic 1600 is Fig.11100. Specifically, the LayerMv tensor 611 is a subsampled motion vector field (feature map) that enters the cost calculation unit 613. The LayerMv tensor 611 also enters the layer information selection unit 614 of the first layer. The layer information selection unit 614 provides the selected motion vector to the code stream if the selected motion vector exists on the (first) layer. Its function will be further described below.

[0272] The cost calculation unit 613 calculates the cost, for example, as shown in FIG. Fig.14 1400. It outputs a cost tensor which is then downsampled by a factor of 2 to match the resolution of the second layer operation. After the second downsampling layer downspl 2, the LayerMV tensor 621 is provided to the next (third) layer and the cost calculation unit 623 of the second layer. The cost calculation unit 623 operates in a similar manner to the cost calculation unit 1400. Those skilled in the art will appreciate that the same Fig.14 Unlike the upsampling / downsampling by a factor of 4 in the described example, a downsampling by a factor of 2 is applied in each direction.

[0273] In order to perform a pairwise comparison of the cost tensors from the cost calculation units 613 and 623, the cost tensor from the previous (first) downsampling layer has been downsampled (downsampled by a factor of 2) to the current resolution layer (the second layer). Then, a pooling operation 625 is performed between the two cost tensors. In other words, the pooling operation 625 maintains the lower cost in the cost tensor for each element. Based on the element index of the result of the pooling operation, the layer with the lower cost is selected. For example, if in a particular tensor element, the cost of the first tensor is lower than the cost of the corresponding element of the second tensor, the index is equal to zero, otherwise the index is equal to 1.

[0274] To ensure gradient propagation for training purposes, a soft arg max can be used to obtain the pooling index with the gradient. If gradient propagation is not needed, regular pooling with indices can be used. As a result 622 of the pooling operation 625, the index of the motion vector indicating whether the motion vector from the current resolution layer or the previous resolution layer is selected (LayerFlag tensor) and the index of the motion vector of the corresponding downsampling layer from the motion segmentation unit (LayerMv tensor) are transmitted to the layer information selection unit 624 of the current (here second) layer. The best pooling cost tensor is propagated to the next downsampling level (downspl2), and the operation is then repeated for the third layer.

[0275] Specifically, the output LayerMv 621 of the second layer is further downsampled (downspl 2) by the third layer, and the resulting motion vector field LayerMv 631 is provided to the cost calculation unit 633 of the third layer. The calculated cost tensor is compared element-by-element 635 with the downsampled cost tensor propagated from the second layer provided by the MinCost pooling unit 625. After the processing of the MinCost pooling 635, the index of the motion vector of the corresponding downsampled layer from the motion segmentation unit (LayerMv tensor) indicating whether the motion vector from the current (third) resolution layer or the previous (second) resolution layer is selected is transmitted to the layer information selection unit 634 of the current (here third) layer. In this example, only three layers are used for exemplary purposes. However, in general, there may be more than three layers, where the other layers and the signal selection logic of these layers have functions similar to those shown for the second and third layers.

[0276] In order to collect pooling information from each spatial resolution layer, the next process is performed in the reverse order, i.e., from lower resolution layers to higher resolution layers using layer information selection units 634, 624, and 614. First, the TakeFromPrev tensor of the same size as the lowest resolution layer (here the third layer) is initialized by 0 (601). Then, the same operation is repeated for layers of different resolutions, as shown below. At the position where the value of the tensor (NOT TakeFromPrev) is equal to 1, the value of the LayerFlag tensor (in the current layer) is selected as the segmentation information to be sent in the bitstream. The (NOT TakeFromPrev) tensor is the element-by-element inversion of the TakeFromPrev tensor. Therefore, in the third layer (here the last layer), all values ​​of the (NOTTakeFromPrev) tensor are set to 1 (set to the inverted 0 by step 601). Therefore, the segmentation information 1130 (LayerFlag) of the last (here the third) layer is always sent.

[0277] The TakeFromCurrent tensor is obtained using the logical operation TakeFromCurrent = (NOT TakeFromPrev) AND LayerFlag. The flag of this tensor TakeFromCurrent indicates whether the motion vector information is selected to be sent in the code stream from the current resolution layer. The layer information selection unit (634, 624, 614) selects the motion vector information from the corresponding down-sampling layer of the motion segmentation unit by taking the value of the LayerMv tensor, where the value of the TakeFromCurrent tensor is equal to 1. This information is sent in the code stream as the selected information 1120.

[0278] For the third (first in reverse order) processing layer corresponding to the lowest resolution, all flags are sent, because TakeFromPrevv is initialized to zero and then all values ​​of (NOT TakeFromPrev) will be equal to 1. For the last processing layer corresponding to the highest resolution layer, the LayerFlag flag does not need to be sent. It is assumed that for all positions where motion information is not selected from the previous layer, these positions should be selected from the current or next (highest resolution) layer.

[0279] It should be noted that Fig.16 The cost computation shown is a parallelizable scheme that can be run on GPU / NPU. The scheme is also trainable since it is transferring gradients, thus enabling its use in an end-to-end trainable video decoding scheme.

[0280] It should be noted that the reverse order processing is similar to the processing performed by the decoder when parsing the segmentation information and motion vector information, as shown below when discussing the decoder function.

[0281] Fig.17 Another exemplary implementation of the signal selection logic 1700 is shown in FIG. Fig.16 compared to, Fig.17 The block diagram introduces multiple decoding options at the same resolution layer. This is illustrated by options 1 to N in the layer 1 cost calculation unit 710. It should be noted that, in general, one or more layers or all layers may include more options. In other words, any of the cost calculation units 613, 623, 633 can provide more options. For example, these options can be one or more or all of the following: different reference images for motion estimation / compensation, single, dual or multiple hypothesis prediction, different prediction methods (such as inter-frame or intra-frame prediction, direct decoding without prediction, multiple hypothesis prediction), presence or absence of residual information, quantization level of residual, etc. In the cost calculation unit 710, the cost is calculated for each decoding option. Then, the best option is selected using minimum cost pooling 720. The indicator (e.g., index) 705 of the selected best option is sent to the layer information selection module 730, and then the indicator BestOpt is transmitted in the bitstream if the corresponding point of the current layer is selected to send information. In the given example, options are shown only for the first layer, but it should be understood that similar option selection logic can also be applied to other layers of different resolutions, or to all layers.

[0282] The above method also applies to the segmentation and transmission of logical information, for example, a flag or switch that controls the image reconstruction process, and information that remains unchanged after decoding and the same as the encoding side. Fig.16Instead of encoding the motion vector field (dense optical flow) processed in the exemplary implementation of , any one or more other parameters may be encoded in a similar manner (including segmentation). It may be one or more or all of the following: indicators indicating different reference images used for motion estimation / compensation, etc., single, dual or multi-hypothesis prediction indicators, different prediction methods (e.g. inter-frame or intra-frame prediction), indicators for direct decoding without prediction, multi-hypothesis prediction, presence or absence of residual information, quantization level of the residual, parameters of the in-loop filter, etc.

[0283] Further modifications to the above embodiments and examples

[0284] According to the first modification, the down-sampling layer of the motion segmentation unit 1310 and / or the up-sampling layer of the motion generation unit 1360 includes a convolution operation. Fig.18 This is shown in Fig.18 As shown, Fig.13 In comparison, the downsampling layer "dwnspl" and the upsampling layer "upspl" are replaced by the downsampling convolution layer "conv↓" in the motion segmentation unit 1810 and the upsampling convolution layer "conv↑" in the motion generation unit 1860, respectively. Some advantages of the convolutional rescaling (downsampling, upsampling) layer are that a learnable downsampling and upsampling process is implemented. For example, in the case of dense motion information, this supports finding the optimal upsampling transform, and thus can reduce the blocking effects caused by motion compensation using block-by-block average motion vector information, as described in the above embodiments and examples. The same situation also applies to the texture restoration process of the original image intensity value, etc., or the generation of prediction residuals for cascade layer processing.

[0285] exist Fig.18 In the above example, all downsampling layers and upsampling layers are convolutional layers. Generally, the present invention is not limited to these implementations. Generally, within the segmentation unit (1310, 1810) and / or within the generation unit (1360, 1860), a subset (one or more) of the downsampling and corresponding upsampling operations can be implemented as convolutions.

[0286] Although the examples described herein are provided for dense optical flow / motion vector field processing, and therefore reference is made to the motion segmentation unit (1310, 1810) and / or the motion generation unit (1360, 1860), the invention is not limited to such data / feature maps. Instead, in any embodiment and example herein, any decoding parameter or even texture, such as samples of an image or prediction residuals (prediction errors), etc., may be processed instead of or in addition to the motion vector field.

[0287] It should be noted that, for example, an encoder with motion information averaging as downsampling can be used in combination with a decoder including a convolutional upsampling layer. In addition, an encoder with a convolutional layer that wants to find a better potential representation can be combined with a motion generation network (decoder) that implements a nearest neighbor based upsampling layer. Further combinations are possible. In other words, the upsampling layer and the downsampling layer are not necessarily of similar type.

[0288] According to a second modification that can be combined with any one of the above embodiments and examples (and with the first modification), the processing of the network includes one or more additional convolutional layers between the cascaded layers with the above different resolutions. For example, the motion segmentation unit 1310 and / or the motion generation unit 1360 also includes one or more intermediate convolutional layers between some or all of the downsampling and upsampling layers. This is Fig.19 It is shown in Fig.19 An exemplary implementation of such a motion segmentation network (module) 1910 and a motion generation network (module) 1860 is shown. It should be noted that the terms "module" and "unit" are used interchangeably herein to represent functional units. In this specific embodiment, units 1910 and 1960 are more specifically network structures having multiple cascaded layers.

[0289] For example, compared to the motion segmentation unit 1310, the motion segmentation unit 1910 has an additional convolutional layer “conv” before each downsampling layer “conv↓” (which may also be other types of downsampling). In addition, compared to the motion generation unit 1360, the motion generation unit 1960 has an additional convolutional layer “conv” before each upsampling layer “conv↑” (which may also be other types of upsampling).

[0290] This can further reduce the blocking effects caused by the sparseness of motion information and increase the generalization effect of finding a better potential representation. Similar to what was mentioned above with respect to the first modification, encoders and decoders from the different embodiments / modifications described above can be combined in one compression system. For example, there may be only an encoder with additional layers between the downsampling layers, and a decoder without such additional layers, or vice versa. Alternatively or in addition, there may be different numbers and positions of such additional layers at the encoder and decoder.

[0291] According to the third amendment, Fig.19 As shown, direct connection is provided for input and output signals. It should be noted that, although shown in the same figure here, the second modification and the third modification are independent. They can be applied together or separately to the previously described embodiments and examples and other modifications. Direct connection is shown with dotted lines.

[0292] In addition to the bottleneck information of the latent representation of the autoencoder (the output of the lowest resolution layer), in some embodiments, information from higher resolution layers is added to the bitstream. In order to optimize the indication overhead, only part of the information from different resolution layers is inserted into the bitstream, controlled by the signal selection logic. On the receiving (decoder) side, the corresponding signal feeding logic feeds information from the bitstream to layers of different spatial resolutions, as described in more detail below. Additionally, information from the input signal before the downsampling layer can be added to the bitstream, through which variability and flexibility can be further increased. For example, decoding can be aligned with real object boundaries and segmented parts with higher spatial resolution and adjusted according to the characteristics of a specific sequence.

[0293] According to a fourth modification, the shapes of the downsampling and upsampling filters can have shapes other than square shapes, such as rectangles with horizontal or vertical orientations, asymmetric shapes, or other arbitrary shapes, by using masking operations. This supports further increasing the variability of the segmentation process to better capture real object boundaries. Fig. 20 This modification is shown in . In the motion segmentation unit 2010, after the first downsampling layer (which can be the same as in any of the above embodiments), there are two further downsampling layers, which use selected filters from the set of filter shapes. This modification is not limited to the processing of motion vector information.

[0294] Typically, in layer downsampling, what is applied is: downsampling the input feature map using a first filter to obtain a first feature map, and downsampling the input feature map using a second filter to obtain a second feature map. The cost calculation includes determining a third cost generated by reconstructing a portion of the reconstructed image using the first feature map, and determining a fourth cost generated by reconstructing a portion of the reconstructed image using the second feature map. Then, in the selection, if the third cost is less than the fourth cost, the first feature map is selected, and if the third cost is greater than the fourth cost, the second feature map is selected. In this example, the selection is made in two filters. However, the present invention is not limited to two filters, but can be selected from a predefined number of filters in a similar manner, such as by estimating the costs of all selectable filters and minimizing the costs by selecting a filter.

[0295] The shape of the first filter and the second filter can be any one of a square, horizontal and vertically oriented rectangle. However, the present invention is not limited to these shapes. Usually, any arbitrary filter shape can be designed. The filter can also include a filter that can be defined as any desired shape. This shape can be indicated by obtaining a mask, wherein the mask is composed of flags, wherein the mask represents an arbitrary filter shape, and wherein one of the first filter and the second filter (usually from any selectable filter of the filter bank) has an arbitrary filter shape.

[0296] In an exemplary implementation, to provide variability, the encoder further comprises pooling between cost tensors obtained with filters having mutually different shapes. The index of the selected filter shape is indicated in the bitstream as (a part of) the segmentation information, similar to the motion vectors described above. For example, for selecting between rectangular shapes in horizontal and vertical directions, a corresponding flag may be indicated in the bitstream. For example, reference Fig.17 The described method of selecting multiple coding options can be used to select different filter shapes at the same resolution layer.

[0297] According to the fifth modification, a motion model from a set of predefined different motion models may be selected in the same resolution layer. In the previous embodiments, specific cases of downsampling and / or upsampling filters have been described. In these cases, the motion information may be averaged over square blocks representing the translational motion model. In this fifth modification, in addition to the translational motion mode, other motion models may also be used. These other motion models may include one or more of the following:

[0298] - Affine motion model,

[0299] -High-order motion models, or

[0300] -CNN layers specially trained to represent specific motion models, such as scale, rotation, affine, perspective, etc.

[0301] In an exemplary implementation of the fifth modification, the autoencoder further comprises a set of CNN layers and / or “handcrafted” layers representing models other than the translational motion model. Fig.21 Such an autoencoder (and decoder) is shown in Fig.21 In , layers including filter banks denoted as “convolutional fit sets” are provided on the encoder side and the decoder side.

[0302] For example, at each spatial layer, the encoder selects the appropriate filter from a filter set corresponding to a particular motion model and inserts the indication into the bitstream. On the receiving side, the signal feed logic interprets the indicator and performs convolution at the particular layer using the corresponding filter from the set.

[0303] The examples of the methods described above use motion information, specifically motion vectors, as exemplary input for encoding. It should be noted again that these methods are also applicable to compression of different types of image or video information, such as direct image sample values, prediction residual information, intra-frame and inter-frame prediction parameters, etc.

[0304] According to the sixth amendment, the above reference Fig.16 or Fig.17 The illustrated RDO can be applied to conventional block-based codecs.

[0305] Conventional video coding approaches, such as state-of-the-art video coding standards, such as AVC, HEVC, VVC or EVC, use a block-based coding concept according to which the image is recursively divided into blocks of square or rectangular shape. For these blocks, signal reconstruction parameters are estimated or evaluated at the encoder side and sent to the decoder in the bitstream. Typically, the encoder aims to find the best reconstruction parameters for the set of blocks representing the image in terms of rate-distortion cost, trying to maximize the reconstruction quality (i.e. minimize the distortion of the original image) and minimize the number of bits required to send the reconstruction process parameters. The task of parameter selection (or decoding mode decision) is a complex and resource-intensive task, which is the main reason for the complexity of the encoder. Due to processing time constraints, for example, in real-time applications, the encoder may sacrifice the quality of the mode decision, which in turn affects the quality of the reconstructed signal. Optimizing the mode decision process has always been a desirable technical improvement.

[0306] One of the coding mode decisions is whether to divide a current block (or coding unit (CU)) into multiple blocks according to a partitioning method.

[0307] According to the sixth modification, the motion segmentation unit 1310 (or 1810) is adapted to make a partitioning mode decision based on minimizing cost (eg, rate-distortion optimization criterion). Fig. 22 An example of such an optimization is shown.

[0308] like Fig. 22 As shown, a block segmentation structure is used to represent information of different spatial resolutions instead of using downsampling layers. For each block of a given size N×N (considering square blocks) of an image or a part of an image, the cost calculation unit calculates the distortion tensor and further downsamples the resolution by a factor of 16 (to match the original resolution). Fig. 22In the given example of , the first block size is 16×16; for example, downsampling is performed by an average pooling operation to obtain a tensor whose each element represents the average distortion in each 16×16 block. In the first layer, the image is segmented into 16×16 blocks 2201 at the initial highest resolution. In the second layer, the resolution is reduced so that the block size 2202 in the image is 32×32 (corresponding to the connection of four blocks in the previous layer). In the third layer, the resolution is reduced again so that the block size 2203 is 64×64 (corresponding to the connection of four blocks in the previous layer). It should be noted that in this case, the connection of four blocks from the previous layer can be regarded as a subsampling of block-related information. This is because, in the first layer, block-related information is provided for each 16×16 block, while in the second layer, block-related information is only provided for 32×32 blocks, that is, one-fourth fewer parameters are provided. Similarly, in the third layer, block-related information is only provided for 64×64 blocks, that is, one-fourth fewer parameters are provided than in the second layer and one-sixteenth fewer than in the first layer.

[0309] In this context, block-related information is any information for each block decoding, such as prediction mode; prediction mode specific information, such as motion vectors, prediction direction, reference images, etc.; filtering parameters; quantization parameters; transform parameters or other settings that may change on block (coding unit) level.

[0310] Then, the cost calculation units 2211, 2212 and 2213 of the corresponding first layer, second layer and third layer calculate the cost according to the block reconstruction parameters for the corresponding block sizes 2201, 2202 and 2203 and according to the input image of size W×H.

[0311] The output cost tensor is obtained as the average distortion in each block, combined with an estimate of the bits required to send the decoding parameters of the N×N (e.g. 16×16 in the first layer) block using Lagrangian multipliers. Fig.23 An exemplary structure of a block N×N cost calculation unit 2300 (which may correspond to each or any one of the cost calculation units 2211 , 2212 , and 2213 ) is shown.

[0312] Fig.23An exemplary block diagram of a cost calculation unit 2300 for a general block size 230x of N×N is shown. The cost calculation unit 2300 obtains (2310) block reconstruction parameters (block-dependent parameters) associated with a block of size N×N. The obtaining may correspond to obtaining parameters (parameter values) from a memory or the like. For example, the block-dependent parameters may be a specific prediction mode, such as an inter-frame prediction mode. In unit 2310, the block reconstruction parameters are obtained and in a reconstruction unit 2320, these parameters are used to reconstruct a portion of an image (in this example, all blocks are reconstructed using the inter-frame prediction mode). A distortion calculation unit 2330 calculates the distortion of the reconstructed portion of the image by comparing it with a corresponding portion of a target image, which may be the original image to be encoded. Since the distortion may be calculated per sample, in order to obtain it on a block basis, downsampling 2340 of the distortion may be performed (one value per N×N block). In the lower branch, the rate or number of bits required to decode the image is estimated (2360). Specifically, the bit estimation unit 2360 can estimate the number of bits to be indicated for each N×N size block. For example, the number of bits required for each block for the inter-frame prediction mode can be calculated. With the estimated distortion and bit amount (or rate), the cost can be calculated (2350) according to the Lagrangian optimization mentioned above. The output is a cost tensor.

[0313] It should be noted that throughout the description, if only 2D images of samples such as grayscale images are observed, the term "tensor" here can be a matrix. However, images can have multiple channels, such as color or depth channels, so that the output can also have more dimensions. In general, feature maps can also exceed 2 or 3 dimensions.

[0314] The same cost evaluation process is performed for the first level (with 16×16 block granularity) and the next level quadtree partitioned into blocks of size 32×32 samples. In order to decide whether it is better to use one 32×32 block as reconstruction parameters (block-dependent parameters) or four 16×16 blocks, the cost tensor evaluated for the 16×16 block is downsampled by a factor of 2 (ref. Fig. 22). The minimum cost pooling operation 2222 then provides the best decision for each 32×32 block. The index of the pooling cost is passed to the layer information selection unit 2232 to be sent in the bitstream as split_flag. The reconstruction parameters blk_rec_params of the best selected block according to the pooling index are also passed to the layer information selection unit 2231. The pooling cost tensor is further passed (downsampled by a factor of 2) to the next quadtree aggregation level of blocks of size 64×64, that is, to MinCost pooling 2223. MinCost pooling 2223 also receives the cost calculated for the 64×64 block resolution 2203 in the cost calculation unit 2213. It passes the index of the pooling cost as split_flags to the layer information selection unit 2233 to indicate in the bitstream. It also passes the reconstruction parameters blk_rec_params of the best selected block to the layer information selection unit 2233 according to the pooling index.

[0315] In order to collect pooling information from each block aggregation level, the layer information selection units 2233, 2232 and 2231 are used as mentioned above. Fig.16 In the manner described, processing is performed in reverse order from higher (in this example highest) aggregation levels (64x64 samples) to lower (in this example lowest) aggregation levels (16x16 samples).

[0316] The result is a code stream that encodes the quadtree partitioning obtained by optimization as well as the coded values ​​and possible further decoding parameters of the resulting partitioned parts (blocks). The method described above supports making decisions on the partitioning flags of the block partitioning. In order to obtain the reconstruction parameters for each block, a traditional method based on evaluating each or part of the possible decoding modes can be used.

[0317] Fig.24 An example of the seventh modification is shown. The seventh modification is the Fig. 22 and Fig.23 An evolution of the sixth modification described. The seventh modification represents a scheme in which the evaluation of decoding modes is incorporated into the design. Specifically, as shown in the figure, the cost calculation unit 710 can evaluate N options. It should be noted that the term "N" here is a placeholder for a certain integer. The "N" representing the number of options is not necessarily the same as the "N" in "N×N" representing a general block size. In the cost calculation unit 710, for the same level of block partitioning, for example, for blocks of size 16×16 samples (such as in the first layer), the encoder iterates all possible (or a finite set thereof) decoding modes for each block.

[0318] Consider an encoder that has N options to decode each 16×16 block (denoted as blk_rec_params 0, blk_rec_params 1, ... blk_rec_params N). For example, the parameter combination blk_rec_params k (k is an integer from 0 to N) can be a combination of certain prediction modes (such as in inter-frame prediction and intra-frame prediction), certain transforms (such as in DCT and KLT), certain filtering orders or filter coefficient sets (in predefined filters), etc. In some implementations, if only one parameter is optimized, blk_rec_params k can be the value k of a single parameter. Those skilled in the art will appreciate that any one or more parameters can be optimized by examining their cost of use.

[0319] For each given set of block reconstruction parameters (blk_rec_params k), the cost calculation unit 2410 calculates a tensor representing the cost of each block. Then, using minimum cost pooling 2420, the best decoding mode for each block is selected and transmitted to the layer information selection unit 2430. The best pooled cost tensor is further downsampled by a factor of 2 and transmitted to the next quadtree aggregation level (in this case, the second level corresponds to an aggregation with a block size of 32×32). Then, the partitioning (splitting) decision is made in the same way as the sixth modification above. Fig.24 In , options 0..N are evaluated only in the first layer (aggregation level 16×16). However, the present invention is not limited to this approach. Instead, the evaluation of options 0..N may be performed at each aggregation level.

[0320] For example, at the next level of quadtree aggregation (32×32, 64×64), the encoder evaluates (by evaluating the cost in the corresponding cost unit) and pools (by the corresponding MinCost pooling unit) the best decoding mode for each block (not shown in the figure for clarity), and compares the best decoding mode with the best decoding mode of the previous aggregation level. The decision on the best mode and the corresponding reconstruction parameters set accordingly are provided to the layer information selection unit (e.g., the layer information selection unit 2430 shown for the first layer). In order to collect pooling information from each block aggregation level, the layer information selection unit is used to perform processing in reverse order from a higher aggregation level (64×64) to a lower aggregation level (16×16) in the manner described in the sixth modification.

[0321] Different block shapes can be used to represent more advanced partitioning methods, such as binary trees, ternary trees, asymmetric, and geometric partitioning. Fig.25 This segmentation of blocks is illustrated by way of example. In other words, the optimization does not have to be performed only on different block sizes, it can also be performed for different segmentation types (e.g. via corresponding options). Fig.25 The following example is shown:

[0322] - Quadtree partitioning 2510. In quadtree partitioning, a block is divided (partitioned) into four blocks of equal size.

[0323] - (Symmetric) Binary Tree Partitioning 2520. In symmetric binary partitioning, a block is divided into two blocks of equal size. The partitioning can be vertical or horizontal. Vertical or horizontal is an additional parameter of the partitioning.

[0324] - Asymmetric binary tree partitioning 2530. In asymmetric binary partitioning, a block is divided into two blocks of different sizes. The size ratio can be fixed (to save the overhead caused by the indication) or variable (in this case, some ratio options can also be optimized, i.e. configurable).

[0325] - Ternary tree partitioning 2540. In ternary partitioning, a block is partitioned into three partitions by two vertical lines or two horizontal lines. Vertical or horizontal is an additional parameter of the partitioning.

[0326] The invention is not limited to these exemplary segmentation patterns. Triangular segmentation or any other type or segmentation may be used.

[0327] In a seventh modification, a hybrid architecture adapted to popular video coding standards is supported and enabled by a method based on powerful (neural) networks. The technical advantages of the described method can provide a highly parallelizable GPU / NPU-friendly solution that can support the calculations required to accelerate the mode decision process. It can enable global image optimization because multiple blocks are considered at the same decision level, merging learnable parts to speed up decision making, for example, for evaluating the amount of bits required for decoding reconstruction parameters.

[0328] In summary, the processing of the cascade layer structure according to the sixth modification or the seventh modification includes processing data related to the same image that is segmented (i.e., divided) into blocks having different block sizes and / or shapes in different layers. The selection of the layer includes: selecting the layer according to the cost calculated for the predetermined decoding mode set.

[0329] In other words, different layers may process image data at different block sizes. Therefore, the cascaded layers include at least two layers that process mutually different block sizes. When a "block" is mentioned herein, it means a unit, i.e., a portion of an image for which decoding is performed. The block may also be referred to as a decoding unit or a processing unit, etc.

[0330] The predetermined set of coding modes corresponds to a combination of decoding parameter values. Different block sizes can be evaluated under a single set of decoding modes (combinations of values ​​of one or more decoding parameters). Alternatively, the evaluation may include various combinations of block sizes and partition shapes (e.g. Fig.25 ). However, the present invention is not limited thereto, and, as specifically mentioned in the seventh modification, there may be several predetermined decoding mode sets (combinations of decoding parameter values), which may further include encoding modes for each block, such as intra / inter prediction type, intra prediction mode, residual skip, residual data, etc.

[0331] For example, the processing includes determining, for at least one layer, costs of different sets of coding modes (combinations of values ​​of coding parameters), and selecting one of the sets of coding modes based on the determined costs. Fig.24 It is shown that only the first layer performs this selection. However, this is not a limitation. Quickly, each cost calculation unit can have the same structure as the first cost calculation unit 2410, including 0..N options. In order to make the figure simpler, this is not shown in the figure.

[0332] As mentioned above, this is a GPU-friendly RDO that can be performed by the codec and choose the best decoding mode for each block. Fig.24 In , the input image is the same in each layer. However, decoding of the image (for computing the cost) is performed in each layer with a different block size. In addition to the block size, further decoding parameters can be tested and selected according to RDO for one or more block sizes.

[0333] Specifically, in these modifications, the indication of data associated with the selected layer includes the selected set of coding modes (eg, blk_rec_params).

[0334] In summary, in some embodiments, an encoder may be provided, which corresponds in structure to a neural network autoencoder for decoding video or image information. The encoder may be used to analyze input image or video information through a neural network including layers of different spatial resolutions; transmit a potential representation corresponding to the output of the lowest resolution layer in a bitstream; and transmit outputs other than the lowest resolution layer in a bitstream.

[0335] decoding

[0336] The encoder described above provides a code stream, which includes the feature data and / or segmentation information of the selected layer. Correspondingly, the decoder processes the data received from the code stream in multiple layers. In addition, the selected layer receives additional (direct) input from the code stream. The input can be some feature data information and / or segmentation information.

[0337] Correspondingly, in the following, the description focuses on embodiments of information related to the selected layer as feature data. Other embodiments described focus on information related to the selected layer as segmentation information. There are also hybrid embodiments in which the codestream carries feature data and segmentation information and the layers handle both feature data and segmentation information.

[0338] In a simple example, a decoder of a neural network autoencoder can be provided for decoding video or image information. The decoder can be used to read a potential representation corresponding to a lower resolution layer input from a bitstream; obtain layer input information based on corresponding information other than the lower resolution layer read from the bitstream; obtain a combined input of the layer based on the layer information obtained from the bitstream and the output of the previous layer; feed the combined input into the layer; and synthesize an image based on the output of the layer.

[0339] Here, the term "lower resolution" refers to layers that process feature maps with lower resolution (e.g., feature maps of the latent space provided from the bitstream). The lower resolution may actually be the lowest resolution of the network.

[0340] The decoder may also be configured to obtain segmentation information based on corresponding information read from the bitstream; and to obtain a combined input of the layer based on the segmentation information. The segmentation information may be a quadtree, a dual (binary) tree, or a ternary tree data structure or a combination thereof. For example, the layer input information may correspond to motion information, image information, and / or prediction residual information, etc.

[0341] In some examples, information obtained from a bitstream corresponding to layer input information is decoded using a super prior neural network. Information obtained from a bitstream corresponding to segmentation information can be decoded using a super prior neural network.

[0342] The decoder can be easily applied to the decoding of motion vectors (such as motion vector fields or optical flow). Some of these motion vectors may be similar or related. For example, in a video showing an object moving on a constant background, there may be two groups of similar motion vectors. The first group of motion vectors may be vectors for predicting the pixels showing the object, and the second group may be vectors for predicting the pixels of the background. Therefore, instead of indicating all motion vectors in the encoded data, it is better to indicate groups of motion vectors to reduce the amount of data representing the encoded video. This can support the indication of the representation of motion vector fields that require a smaller amount of data.

[0343] Fig. 9A codestream 930 generated by the encoder as described above and received at the decoder is shown. On the decoding side, the decoder portion of the system 900 includes signal feed logic 940, which in some embodiments interprets the segmentation information obtained from the codestream 930. Based on the segmentation information, the signal feed logic 940 identifies the specific (selected) layer, spatial size (resolution), and location of the feature map portion where the corresponding selected information (also obtained from the codestream) should be placed.

[0344] It should be noted that in some embodiments, the segmentation information is not necessarily processed by the cascade network. It can be provided independently or derived from other parameters in the code stream. In other embodiments, the feature data is not necessarily processed in the cascade network, but the segmentation information is. Therefore, the two sections "Decoding using feature information" and "Decoding using segmentation information" describe examples of these embodiments and combinations of these embodiments.

[0345] It should be noted that, for the two-part embodiment, the encoder-side modifications (first to seventh) described above are correspondingly applied to the decoder side. For better understandability, the modified additional functions are not replicated in the two parts. However, those skilled in the art will appreciate that they can be applied to the decoding methods of the two parts alternatively or in combination.

[0346] Decoding using feature information

[0347] In this embodiment, a method for decoding data for image or video processing from a bit stream is provided. Fig.33 Correspondingly, a device for decoding data for image or video processing from a bitstream is provided. The device may include a processing circuit for executing the steps of the method.

[0348] The method includes acquiring (3310) two or more feature map element sets from a code stream, wherein each feature map element set is associated with a (corresponding) feature map. The acquisition may be performed by parsing the code stream. In some exemplary implementations, the code stream parsing may also include entropy decoding. The present invention is not limited to any particular way of acquiring data from a code stream.

[0349] The method also includes the step of inputting (3320) each of the two or more feature map element sets into two or more feature map processing layers in a plurality of cascade layers.

[0350] Cascaded layers may form part of a processing network. In the present invention, the term "cascaded" means that the output of one layer is subsequently processed by another layer. Cascaded layers do not have to be closely adjacent (the output of one cascaded layer goes directly into the input of the second layer in the cascade). Fig. 9, data from bitstream 930 is input to signal feed logic 940, which feeds feature map element sets to the appropriate layers 953, 952, and / or 951 (indicated by arrows). For example, a first feature element set is inserted into the first layer 953 (the first in the processing sequence), and a second feature element set is inserted into the third layer 951. It is not necessary to insert the set into the second layer. The number and position of the layers (within the processing sequence) can vary, and the present invention is not limited to any specific number and position.

[0351] The method also includes: obtaining (3330) decoded data for image or video processing as a result of processing by the plurality of cascaded layers. For example, the first set is a set of potential feature map elements processed by all layers of the network. The second set is an additional set provided to another layer. When reference is made to Fig. 9 , the decoded data 911 is obtained after processing the first set of three layers 953, 952 and 951 (in this order).

[0352] In an exemplary implementation, a feature map is processed in each of the two or more feature map processing layers, wherein the resolutions of the feature maps processed in the two or more feature map processing layers are different. For example, a first feature map processed by a first layer has a resolution different from a resolution of a second feature map processed by a second layer.

[0353] Specifically, processing the feature map in two or more feature map processing layers includes upsampling. Fig. 9 A network is shown where the decoding part comprises three (directly) cascaded upsampling layers 953, 952 and 951.

[0354] In an exemplary implementation, the decoder includes only upsampling layers of different spatial resolutions, and the nearest neighbor method is used for upsampling. The nearest neighbor method repeats the value of the lower resolution in the higher resolution area corresponding to a given shape. For example, if one lower resolution element corresponds to four higher resolution elements, the value of the one element is repeated four times in the higher resolution area. In this case, the term "corresponding" means describing the same area in the highest resolution data (initial feature map, initial data). This upsampling method supports sending information from a lower resolution layer to a higher resolution layer without modification (this can be suitable for some data, such as a logical flag or indicator information), or it is desired to keep the same information as that obtained on the encoder side without the need for some convolutional layer modifications. An example of such data is prediction information, such as motion information of motion vectors estimated on the encoder side, a reference index indicating which specific image from a reference image set should be used, a prediction mode indicating whether a single or multiple reference frames are used, a combination of different predictions (such as combined intra-inter prediction), presence or absence, or residual information, etc.

[0355] However, the present invention is not limited to upsampling performed by the nearest neighbor method. Alternatively, upsampling can be performed by applying some interpolation or extrapolation, or by applying convolution, etc. These methods may be particularly suitable for upsampling data that is expected to have smooth characteristics, such as motion vectors or residuals or other sample-related data.

[0356] exist Fig. 9 In the embodiment, the encoder (e.g., symbols 911 to 920) and the decoder (e.g., symbols 940 to 951) have the same number of downsampling and upsampling layers, the nearest neighbor method can be used for upsampling, and the average pooling can be used for downsampling. The shape and size of the pooling layer are aligned with the scale factor of the upsampling layer. In some other possible implementations, another pooling method, such as maximum pooling, can be used.

[0357] As already illustrated in several encoder embodiments, the data for image or video processing may include a motion vector field. For example, Fig.12 The encoder and decoder sides are shown. On the decoder side, the bitstream 1250 is parsed and motion information 1260 is obtained therefrom (possibly with segmentation information discussed below). The obtained motion information is then provided to a motion generation network 1270. The motion generation network can increase the resolution of the motion information, i.e., dense motion information. The reconstructed motion vector field (e.g., dense optical flow) 1275 is then provided to a motion compensation unit 1280. The motion compensation unit 1280 uses the reconstructed motion vector field to obtain predicted image / video data based on a reference frame and reconstruct a motion compensated frame based on the predicted image / video data (e.g., by adding a decoded residual - e.g., Figure 5A The decoder portion of the encoder illustrated in , or Figure 7B The reconstruction unit 314 illustrated in FIG.

[0358] Fig.13 Also shown is a decoder-side motion generation (densification) network 1360. Network 1360 includes signal feed logic 1370, similar in function to Fig. 9 The signal feed logic 940, and three upsampling (processing) layers. Fig. 9 The main difference of the described embodiments is that Fig.13 In the embodiment, network 1360 is dedicated to motion vector information processing and outputs a motion vector field.

[0359] As described above, according to an embodiment, the method further includes obtaining segmentation information related to the two or more layers from the bitstream. Then, obtaining feature map elements from the bitstream based on the segmentation information. Inputting the feature map element sets into two or more feature map processing layers based on the segmentation information respectively. Some detailed examples related to using segmentation information in parsing and processing are provided below in the section related to decoding using segmentation information. For example, Fig.28 and Fig.29 Very specific (and merely exemplary) layer processing options are provided.

[0360] In some embodiments, the plurality of cascaded layers further comprise a plurality of segmentation information processing layers. The method further comprises processing the segmentation information in the plurality of segmentation information processing layers. For example, processing the segmentation information in at least one of the plurality of segmentation information processing layers comprises upsampling. In some embodiments, such upsampling of the segmentation information and / or the upsampling of the feature map comprises nearest neighbor upsampling. Typically, the upsampling applied to the feature map information and the upsampling applied to the segmentation information may be different. Furthermore, the upsampling within the same network may be different, so one network (segmentation information processing or feature map processing) may comprise different types of upsampling layers. For example, in Fig. 20 This example is shown in FIG. 21. It should be noted that the upsampling types other than the nearest neighbor may include some interpolation methods, such as polynomial methods, such as bilinear, cubic, etc.

[0361] According to an exemplary implementation, the upsampling of the segmentation information and / or the upsampling of the feature map comprises a (transposed) convolution. This corresponds to the first modification of the encoder discussed above. Fig.18 On the decoder side, a motion generation unit 1869 is shown, which includes a convolution operation "conv↑" instead of nearest neighbor upsampling. This allows for a learnable upsampling process, for example, supporting finding the best upsampling transform when the upsampling process is used for motion information densification, and can reduce blocking artifacts caused by motion compensation by using block average motion vector information as described above. The same applies to texture recovery processes, such as original image intensity value or prediction residual generation. The motion generation unit 1869 also includes signal feed logic, the functionality of which corresponds to Fig.13 The signal feeds logic 940 or Fig. 9 Or the function of 1370.

[0362] Fig.30A block diagram of an exemplary decoder side layer processing according to the first modification is shown. Specifically, the code stream 3030 is parsed, and the signal feed logic 3040 (functionally corresponding to the signal feed logic 940 or 1370) provides a selection instruction to the convolution upsampling filter 300. In some embodiments, the convolution filter can be selected from a set of N filters (indicated as filters 1 to N). The filter selection can be based on information indicating the selected filter and parsed from the code stream. The indication of the selected filter can be provided (generated and inserted into the code stream) by an optimization method such as encoder perception RDO. Specifically, it can be applied Fig.17 or Fig.24 The RDO illustrated in FIG. 1 (treating filter size / shape / order as one of the options (i.e., decoding parameters to be optimized)). However, the present invention is not limited thereto, and in general, the filter can be derived based on other signal parameters (e.g., decoding mode, interpolation direction, etc.).

[0363] In summary, the signal feeding logic unit controls the input of different layers with different filter shapes and selectively bypasses the layer output to the next layer based on the segmentation and motion information obtained from the bitstream. The convolution filter unit 3000 corresponds to the convolution performed by one layer. Several such convolutional flooding filters can be cascaded, such as Fig.18 As shown. It should be noted that the present invention is not limited to variable or trainable filter settings. Generally, convolution upsampling can also be performed using fixed convolution operations.

[0364] Aspects of this embodiment may be combined with aspects of other embodiments. For example, an encoder with motion information averaging in the downsampling layer may be used in combination with a decoder including a convolutional upsampling layer. An encoder with a convolutional layer that wants to find a better potential representation may be combined with a motion generation network including a nearest neighbor based upsampling layer. Other combinations may also be envisioned. That is, the implementation of the encoder and decoder need not be symmetrical.

[0365] Fig.32A Two examples of applying the nearest neighbor method for reconstruction are illustrated. Specifically, Example 1 shows a case where the segmentation information of the lowest resolution layer has a value of the flag set (1). Correspondingly, the motion information indicates a motion vector. Since the motion vector is already indicated in the lowest resolution layer, there are no other motion vectors and other segmentation information in the code stream. The network generates corresponding motion vector fields with higher resolution (2×2) and highest resolution (4×4) from a signal motion vector by copying an indicated motion vector during nearest neighbor upsampling. The result is a 4×4 area with all 16 motion vectors that are the same and equal to the indicated motion vector.

[0366] Fig.32B Two examples of reconstruction applying the convolutional layer based method are illustrated. Example 1 has Fig.32A The same input as in Example 1. Specifically, the segmentation information of the lowest resolution layer has the value of the flag set (to 1). Correspondingly, the motion information indicates a motion vector. However, after applying the convolutional layers (possibly trained), the motion vectors in the higher layers and the highest layer are not exactly the same, rather than just copying a motion vector.

[0367] Similarly, Fig.32A Example 2 in shows the segmentation information of 0 in the lowest resolution layer, and the segmentation information of 0101 in the following (higher resolution) layers. Correspondingly, the two motion vectors at the positions indicated by the segmentation information are indicated in the code stream as motion information. This information is shown in the middle layers. From the bottom layer, it can be seen that the signal motion vectors are replicated, and each vector is replicated four times to cover the highest resolution area. The remaining 8 motion vectors of the highest resolution (bottom) layer are indicated in the code stream. Fig.32B Example 2 applies convolution instead of nearest neighbor copying. Motion vectors are no longer copied. Fig.32A The transitions between motion vectors replicated in the image are now smoother, reducing blocking artifacts.

[0368] Similar to the second modification of the encoder discussed above, on the decoder side, multiple cascaded layers include convolutional layers without upsampling between layers with different resolutions. It should be noted that the encoder and decoder are not necessarily symmetrical in this regard: the encoder can have these additional layers while the decoder does not or vice versa. Of course, the encoder and decoder can also be designed symmetrically and have additional layers between the corresponding downsampling and upsampling layers of the encoder and decoder.

[0369] Regarding the combination of segmentation information processing and feature map processing, obtaining feature map elements from the code stream is based on the processed segmentation information processed by at least one of the multiple segmentation information processing layers. The segmentation layer can parse and interpret the segmentation information, as described in more detail below in the "Decoding Using Segmentation Information" section. It should be noted that the described embodiments and examples can be used in conjunction with the embodiments in this section. Specifically, the following reference Figures 26 to 32B The described layer processing of segmentation information may also be performed in conjunction with the feature map processing described herein.

[0370] For example, inputting each of the two or more feature map element sets into two or more feature map processing layers respectively is based on the processed segmentation information processed by at least one segmentation information processing layer among the multiple segmentation information processing layers. The obtained segmentation information is represented by a set of syntax elements, where the position of the elements in the set of syntax elements indicates which feature map element position the syntax element is related to. For example, the set of syntax elements is a bitstream section, which can be binarized using entropy coding such as fixed code, variable length code, or arithmetic code, and any of these codes can be context adaptive. If the bitstream has a predefined structure known to both the encoder side and the decoder side, the present invention is not limited to any specific coding or form of the bitstream. In this way, the parsing and processing of the segmentation information and the feature map information can be completed associatively. For example, the processing of the feature map includes, for each syntax element: (i) when the syntax element is the first value, parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream, and (ii) otherwise (or more generally, when the syntax element is the second value), not parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream. The syntax element can be a binarization flag, which is sorted into the bitstream at the encoder side and parsed from the decoder in the correct order by a specific layer structure of the processing network.

[0371] It should be noted that options (i) and (ii) can also be provided for non-binarized syntax elements. In this case, the first value means parsing, and the second value means not parsing. The syntax element can take some other values in addition to the first value and the second value. These values can also result in parsing or not parsing, or can indicate a specific type of parsing, etc. The number of parsed feature map elements can correspond to the number of syntax elements equal to the first value.

[0372] According to an exemplary implementation, each layer j (1 < j < N) of the multiple (N) feature map processing layers processing the feature map further includes: parsing the segmentation information element of the j-th feature map processing layer from the bitstream; obtaining the feature map processed by the previous feature map processing layer, and parsing the feature map element from the bitstream, and associating the parsed feature map element with the obtained feature map, where the position of the feature map element in the processed feature map is indicated by the parsed segmentation information element, and the segmentation information is processed by the previous segmentation information processing layer. For example, the association can be a replacement of the previously processed feature map element, or a combination, such as addition, subtraction, or multiplication. Some exemplary implementations are provided below. The parsing can depend on the previously processed segmentation information, which provides the possibility of a very compact and efficient syntax

[0373] For example, the method may include parsing elements of a feature map from a code stream when a syntax element is a first value; and not parsing elements of a feature map from a code stream when a syntax element is a second value or the segmentation information processed by a previous segmentation information processing layer is a first value. This means that if the relevant part is parsed at a previous layer, parsing is not performed. For example, the syntax element parsed from the code stream representing the segmentation information is a binarization flag. As described above, it may be beneficial for the processed segmentation information to be represented by a set of binarization flags. The set of binarization flags is a sequence of binarization flags having each value 1 or 0 (corresponding to the above-mentioned first value and second value).

[0374] In some embodiments, the upsampling of the segmentation information in each segmentation information processing layer j further includes: for each p-th position in the acquired feature map indicated by the input segmentation information, determining an indication of a feature map position included in the same region as the p-th position in the reconstructed image as the upsampled segmentation information. This provides a spatial relationship between the reconstructed image (or reconstructed feature map or general data), the position in the subsampled feature map, and the corresponding segmentation mark.

[0375] As described above, and in the encoder embodiment, the data used for image or video processing may include image data (e.g., image samples) and / or prediction residual data and / or prediction information data. When "residuals" are mentioned in the present invention, it should be noted that these may be pixel domain residuals or transform (spectral) coefficients (i.e., transform residuals, residuals represented in a domain other than the sample / pixel domain).

[0376] Similar to the fourth modification described above for the encoder side, according to an exemplary implementation, the filter is used for upsampling the feature map, and the shape of the filter is any one of a square, a horizontal rectangle, and a vertical rectangle. It should be noted that the filter shape can be similar to Fig.25 The segmentation shape shown in .

[0377] Fig. 20 An exemplary decoder side layer processing is shown in . The motion generation network (unit) 2060 includes signal feeding logic and one or more (here two) upsampling layers using filters (upsampling filters) that can be selected from a predetermined or predefined filter set. The selection can be performed on the encoder side, such as through RDO or other settings, and indicated in the bitstream. On the decoder side, the indication of filter selection is parsed from the bitstream and applied. Alternatively, the filter can be selected at the decoder without being explicitly indicated based on other decoding parameters derived from the bitstream. These parameters can be any parameters related to the content, such as prediction type, direction, motion information, residual, loop filter characteristics, etc.

[0378] Fig.31A block diagram of an upsampling filter unit 3100 supporting the selection of one filter among N filters 1 to N is shown. The indication of filter selection may directly include the index of one of the N filters. It may include filter direction, filter level, filter shape and / or coefficients. On the decoding side, the signal feed logic interprets the filter selection flag (e.g., a direction flag for distinguishing between vertical and horizontal filters or other directions) and feeds the feature map value to the layer with the corresponding filter shape set. Fig.31 In the example, a direct connection from the signal feed logic to the selective bypass logic supports not selecting any filter. The corresponding value of the filter selection indicator can also be indicated or derived in the bitstream.

[0379] Typically, upsampling of the feature map uses a filter, and inputting information from the bitstream also includes: obtaining information indicating the filter shape and / or the filter direction and / or the filter coefficient from the bitstream. There may be an implementation in which each layer has a set of filters to be selected, or each layer is a filter and the signal feeding logic determines which layers are selected and which layers are not selected based on a filter selection flag (indicator).

[0380] In some embodiments, a flexible filter shape can be provided, wherein the information indicating the filter shape indicates a mask consisting of flags, and the mask represents the filter shape, because a flag with a third value indicates a non-zero filter coefficient, and a flag with a fourth value different from the third value indicates a zero filter coefficient. In other words, as already described for the encoder side, the filter shape can be defined by indicating the position of the non-zero coefficients. The non-zero coefficients can be derived according to predefined rules or can be indicated.

[0381] The above decoder embodiments may be implemented as a computer program product stored on a non-transitory medium, and when the computer program product is executed on one or more processors, the steps of any of the above methods are performed. Similarly, the above decoder embodiments may be implemented as a device for decoding an image or video, comprising a processing circuit, the processing circuit being used to perform the steps of any of the above methods. Specifically, a device for decoding data for image or video processing from a bitstream may be provided, the device comprising: an acquisition unit for acquiring two or more groups of feature map elements from the bitstream, wherein each group of feature map elements is associated with a feature map; an input unit for inputting each of the two or more feature map element sets into two or more feature map processing layers in a plurality of cascaded layers, respectively; a decoded data acquisition unit for acquiring the decoded data for image or video processing as a processing result of the plurality of cascaded layers. These units may be implemented in software or hardware or a combination of both, as discussed in more detail below.

[0382] Decoding using segmentation information

[0383] On the receiving side, the decoder of the present embodiment performs parsing and interpretation of the segmentation information. Therefore, a method is provided, such as Fig.34 As shown, it is used to decode data for image or video processing from a bit stream. Correspondingly, a device for decoding data for image or video processing from a bit stream is provided. The device may include a processing circuit for executing the steps of the method.

[0384] The method includes obtaining (3410) two or more sets of segmentation information elements from a bitstream. The obtaining may be performed by parsing the bitstream. In some exemplary implementations, the bitstream parsing may also include entropy decoding. The present invention is not limited to any particular manner of obtaining data from a bitstream. The method also includes inputting (3420) each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers. It should be noted that the segmentation information processing layer may be the same layer as the feature map processing layer or a different layer. In other words, a layer may have one or more functions.

[0385] Furthermore, in each of the two or more segmentation information processing layers, the method includes processing a corresponding segmentation information set.According to the segmentation information processed by the plurality of cascaded layers, decoded data for image or video processing is obtained (3430).

[0386] Fig.26 Exemplary segmentation information for three-layer decoding is shown. The segmentation information can be regarded as selecting (see encoder side description) the layer from which the feature map element will be parsed or otherwise obtained. Feature map element 2610 is not selected. Therefore, the encoder sets flag 2611 to 0. This means that the feature map element 2610 with the lowest resolution is not included in the codestream. However, a flag 2611 indicating that the feature map element is not selected is included in the codestream. For example, if the feature map element is a motion vector, this can mean that the motion vector 2610 of the largest block is not selected and is not included in the codestream.

[0387] exist Fig.26 In the example shown, in the feature map 2620, three feature map elements are selected for indication (indicated by flags 2621, 2622, and 2624) out of the four feature map elements used to determine the feature map 2610, while one feature map element 2623 is not selected. In the example using motion vectors, this may mean selecting three motion vectors from the feature map 2620 and setting their respective flags to 1, while not selecting one feature map element and setting its respective flag 2623 to 0.

[0388] Then, the code stream may include all four flags 2621 to 2624 and the three selected motion vectors. In general, the code stream may include four flags 2621 to 2624 and three selected feature map elements. In the feature map 2630, one or more elements that determine the unselected feature map elements of the feature map 2620 may be selected.

[0389] In this example, when selecting feature map elements, no elements of the higher resolution feature map are selected. In this example, feature map elements of feature map 2630 that are used to determine feature map elements indicated by flags 2621, 2622, and 2624 are not selected. In one embodiment, none of the flags for these feature map elements are included in the codestream. Instead, only flags for feature map elements of feature map 2630 that determine feature map elements with flag 2623 are included in the codestream.

[0390] In the example where the feature map elements are motion vectors, each of the feature map elements 2621, 2622, and 2624 may be determined by a group of four motion vectors in the feature map 2630. In each group of motion vectors determined with flags 2621, 2622, and 2624, these motion vectors may have more similarity to each other than the four motion vectors in the feature map 2630 that determined the motion vectors (feature map elements) in the feature map 2620 that were not selected (signaled by flag 2623).

[0391] Fig.26 It is described by the characteristics of the code stream. It should be noted that the decoder decodes (parses) this code stream accordingly: determines which information is included (indicated) according to the value of the flag as described above, and parses / interprets the parsed information accordingly.

[0392] In an exemplary implementation, the segmentation information is as follows: Fig. 27 For 2D information such as images or videos viewed as image sequences, the feature maps of some layers can be represented in two-dimensional space. The segmentation information includes an indicator (binarization flag) for a 2D spatial position, indicating whether the feature map value corresponding to the position is presented in the code stream.

[0393] exist Fig. 27In , there is a starting layer (layer 0) for decoding segmentation information, for example, the layer with the lowest resolution, i.e., the potential representation layer. For this starting layer, each 2D position includes a binarization flag. If the flag is equal to 1, the selected information includes the feature map value of this position on this particular layer. On the other hand, if the flag is equal to 0, there is no information for this position on this particular layer. The set of flags (or generally the flag tensor, here the flag matrix) will be referred to as TakeFromCurrent. The TakeFromCurrent tensor is upsampled to the next layer resolution, for example using the nearest neighbor method. This tensor is denoted as TakeFromPrev. The flag in this tensor indicates whether the corresponding sample position is filled in the previous layer (here layer 0).

[0394] As a next step, the signal feed logic reads the flag of the current resolution layer (LayerFlag) position. In this exemplary implementation, only the positions that were not filled in the previous layer (not set to 1, not filled with feature map element values) are indicated. Using logical operations, it can be expressed as TakeFromPrev==0 or !TakeFromPrev==1, where "!" represents a logical NOT operation (negation).

[0395] The amount of flags required for this layer can be computed as the number of zero (logically false) elements in the TakeFromPrev tensor, or the number of values ​​with 1 (logically true) in the inverted (!TakeFromPrev) tensor. For non-zero elements in the TakeFromPrev tensor, no flags are required in the codestream. This is represented in the diagram by showing a "-" at positions that do not need to be read. From an implementation perspective, it may be easier to compute the sum of the elements on the inverted tensor as the sum (!TakeFromPrev). The signal feed logic can use this algorithm to determine how many flags need to be parsed from the codestream. Read the flag placed at the position where the value of !TakeFromPrev is equal to 1, and obtain the LayerFlag tensor. Then, obtain the TakeFromCurrent tensor for the current resolution layer (here layer 1) as the combination of the TakeFromPrev and LayerFlag tensors by keeping the flag at the position read from the codestream of the current resolution layer and zeroing the value at the position read from the previous resolution layer (the position marked with "-" in LayerFlag). The AND can be represented and implemented using the logical AND operator as follows: TakeFromCurrent = !TakeFromPrev. TakeFromPrev ANDLayerFlag. Then, to take into account the position read at the previous resolution layer, the TakeFromCurrent tensor is obtained using a logical OR operation, such as TakeFromCurrent = TakeFromCurrent OR TakeFromPrev. It should be understood that Boolean operations can be implemented using regular mathematical operations, such as multiplication for AND and summation for OR. This will provide the benefits of preserving and transferring gradients, supporting the use of the above method in end-to-end training.

[0396] Then, the obtained TakeFromCurrent tensor is upsampled to the next resolution layer (here is layer 2) and the above operation is repeated.

[0397] For generality and simplicity of implementation, it is beneficial to unify the processing of all resolution layers, without special consideration of the first resolution layer, where all flags are parsed from the codestream. This can be achieved by initializing TakeFromPrev to zero before processing in the first (lower resolution) layer (layer 0), and repeating the above steps for each resolution layer.

[0398] In order to further reduce the indication overhead, in some other implementations, the LayerFlag of the last resolution layer (here the third layer, i.e., layer 2) does not need to be transmitted (including at the encoder side and parsed at the decoder side) to the bitstream. This means that for the last resolution layer, the feature map value is used as the selected information of all positions of the last resolution layer (see Fig.111120 in 1120) is sent in the codestream, which is not obtained at (all) previous resolution layers (at any previous resolution layer). In other words, for the last resolution layer, TakeFromCurrent = !TakeFromPrev, that is, TakeFromCurrent corresponds to the inverted TakeFromPrev. Similarly, in order to maintain the commonality of the last resolution layer processing, LayerFlag can be initialized by 1, and the same formula can be used: TakeFromCurrent = !TakeFromPrev AND LayerFlag.

[0399] In some other possible implementations, the last resolution layer has the same resolution as the original image. If the last resolution layer has no additional processing steps, this means that some values ​​of the original tensor are sent without compression in the autoencoder.

[0400] Below, reference Fig.28 An example of signal feed logic 2800 is described. Fig.28 , the decoder's signal feed logic 2800 uses the segmentation information (LayerFlag) to obtain and utilize the selected information (LayerMv) sent in the codestream. Specifically, at each layer, the codestream is parsed to obtain the segmentation information (LayerFlag) in the corresponding syntax interpretation units 2823, 2822, and 2821, and possible selection information (LayerMv) (in this order). As described above, in order to enable the same operations in the first layer (syntax interpretation 2823) as in other layers, the TakeFromPrev tensor is initialized to all zeros in 2820. The TakeFromPrev tensor is propagated in the processing order from the syntax interpretation of the early layer (such as 2823) to the syntax interpretation of the later layer (such as 2822). The propagation here includes upsampling by 2, as above Fig. 27 Explained.

[0401] When interpreting the segmentation information (LayerFlag) of each resolution layer, the tensor TakeFromCurrent is obtained (generated). This tensor TakeFromCurrent includes a flag indicating whether there is feature map information (LayerMv) in the codestream for each specific position of the current resolution layer. The decoder reads the values ​​of the feature map LayerMv from the codestream and places them at the position where the flag of the TakeFromCurrent tensor is equal to 1. The total amount of feature map values ​​included in the codestream of the current resolution layer can be calculated based on the number of non-zero elements in TakeFromCurrent or as the sum (TakeFromCurrent) - the sum of all elements of the TakeFromCurrent tensor. As a next step, the tensor combination logic 2813, 2812, and 2811 in each layer (e.g., at 2812) combines the previous resolution layer (e.g., generated by 2813 and upsampled 2801 to match the resolution of the next layer processing 2812) by replacing the feature map values ​​at the positions where the value of the TakeFromCurrent tensor is equal to 1 with the feature map values ​​(LayerMv) sent as the selected information in the codestream. As described above, in order to enable the same operation in the first layer (tensor combination 2813) as in other layers, the combined tensor is initialized to all zeros in 2810. After processing the LayerFlags of all layers and generating the output tensor of the last layer (in 2811), in 2801, the combined tensor is upsampled by 4 to obtain the original size of the dense optical flow, i.e., W×H.

[0402] Fig.28 The exemplary implementation of provides a fully parallelizable scheme that can run on GPU / NPU and supports exploiting parallelism. The fully trainable scheme of transferring gradients supports the use of the scheme in an end-to-end trainable video coding scheme.

[0403] Fig.29 Another possible exemplary implementation of the signal feed logic 2900 is shown. This implementation generates a LayerIdx tensor (in Fig.29The LayerIdx tensor is called LayerIdxUp in 2920, including indices of layers of different resolutions, indicating which layer should be used to receive motion information transmitted in the codestream (including in the encoder and parsed in the decoder). At each syntax interpretation block (2923, 2922, 2923), the LayerIdx tensor is updated by adding the TakeFromCurrent tensor multiplied by the upsampled layer index numbered from the highest resolution to the lowest resolution. The LayerIdx tensor is then upsampled and transmitted (passed) to the next layer in the order of processing, for example, from 2923 to 2922, and from 2922 to 2921. In order to make the processing in all layers similar, the tensor LayerIdx is initialized to zero in 2920 and passed to the syntax interpretation 2923 of the first layer.

[0404] After the last layer (here the third layer), the LayerIdx tensor is upsampled to the original resolution (upsampled by 4 to 2995). Therefore, each position of LayerIdx includes the index of the layer from which the motion information is to be obtained. The positions of LayerIdx correspond to the original resolution of the feature map data (here the dense optical flow) at the same resolution, which is 2D (matrix) in this example. Therefore, for each position in the reconstructed optical flow, LayerIdx specifies where (from which layer's MayerMV) the motion information is to be obtained.

[0405] Motion information (LayerMv, in Fig.29The tensor combination block (2913, 2912, 2911) combines the LayerMv obtained from the bitstream (via the corresponding syntax interpretation unit 2923, 2922, 2921) with the intermediate tensor based on the segmentation information (LayerFlag) obtained from the bitstream and the intermediate TakeFromCurrent Boolean tensor according to the above method. The intermediate tensor can be initialized by zero (see initialization unit 2910, 2919, 2918) or any other value. The initialization value is not important because ultimately after all steps are completed, according to the method, these values ​​will not be selected for dense optical flow reconstruction 2990. The combined tensor (output from each of 2913, 2912, 2911) including motion information is upsampled and concatenated with the combined tensor of the previous spatial resolution layer (2902, 2901). The cascading is performed along the additional dimensions corresponding to the motion information obtained from the layers of different resolutions (i.e., the 2D tensor 2902 before cascading becomes the 3D tensor after cascading; the 3D tensor 2901 before cascading remains a 43D tensor after cascading, but the size of the tensor increases). Finally, after all upsampling steps of LayerIdxUp and LayerMvUp are completed, the reconstructed dense optical flow is obtained by selecting the motion information from LayerMvUp and using the value of LayerIdxUp as the index on the axis in LayerMvUp, where the axis is the dimension added during the cascading step of LayerMvUp. In other words, the dimension added in LayerMvUp is the dimension on the number of layers, and LayerIdxUp selects the appropriate layer for each position.

[0406] The above specific exemplary implementation does not limit the present invention. Generally, segmentation can be performed and indicated in the code stream in various conceivable ways. Generally, the acquisition of the segmentation information element set is based on the segmentation information processed by at least one segmentation information processing layer in the plurality of cascaded layers. Fig.28 As shown, such a layer may include a syntax interpretation unit (2823, 2822, 2821), which parses / interprets the meaning (semantics) of the parsed segmentation information LayerFlag.

[0407] More specifically, the input of the segmentation information element set is based on the processed segmentation information output by at least one cascade layer of the plurality of cascade layers. Fig.28 In the example, the TakeFromPrev tensor is passed between the syntax interpretation units (2823, 2822, 2821). As already explained in the description of the encoder side, in some exemplary implementations, the resolutions of the segmentation information processed in two or more segmentation information processing layers are different.

[0408] In addition, the processing of the segmentation information in two or more segmentation information processing layers includes upsampling, as already referenced Fig. 9 , 13 and as shown in other figures. For example, the upsampling of the segmentation information includes nearest neighbor upsampling. It should be noted that in this embodiment and in the above embodiments, the present invention is not limited by applying nearest neighbor upsampling. The upsampling may include interpolation instead of simply copying the adjacent sample (element) values. The interpolation can be any known interpolation, such as linear or polynomial, such as cubic upsampling, etc. Regarding copying, it should be noted that the copying performed by the nearest neighbor is to copy the element value from a predefined (available) nearest neighbor (such as the top or the left). If there are neighbors at the same distance from the position to be filled, a predefined one from which to copy the neighbor may be required.

[0409] As described above for the first modification, in some exemplary implementations, the upsampling includes transposed convolution. In addition to applying convolutional upsampling to the feature map information, or alternatively, convolutional upsampling can also be applied to the segmentation information. It should be noted that the type of upsampling performed for the segmentation information is not necessarily the same type of upsampling applied to the feature map elements.

[0410] Generally, for each segmentation information processing layer j in multiple (N) cascaded segmentation information processing layers, the input includes:

[0411] - If j = 1, the initial segmentation information is input from the bitstream (and / or based on initialization, such as initialized to zero in step 2820), otherwise, the segmentation information processed by the (j - 1)th segmentation information processing layer is input;

[0412] - The processed segmentation information is output.

[0413] This is the segmentation information related to the input layer and is not necessarily (still may be) the entire segmentation information from the bitstream. The upsampled segmentation information in the jth layer is the upsampled segmentation information in the jth layer, that is, output by the jth layer. Generally, the processing of the segmentation layer includes upsampling (TakeFromPrev) and includes new elements from the bitstream (LayerFlag).

[0414] For example, the processing of the input segmentation information by each layer j < N in multiple (N) segmentation information processing layers further includes parsing the segmentation information elements (LayerFlag) from the bitstream and associating the parsed segmentation information elements (e.g., in the syntax decoding unit 282x in Fig.28 ) with the segmentation information (TakeFromPrev) output by the previous layer. The position of the parsed segmentation information elements (LayerFlag) in the associated segmentation information is determined according to the segmentation information output by the previous layer. As in Fig.28 and29 As can be seen in , there can be various different ways to associate and propagate location information. The present invention is not limited to any specific implementation.

[0415] For example, the number of segmentation information elements parsed from the code stream is determined based on the segmentation information output by the previous layer. Specifically, if some areas have been covered by the segmentation information of the previous layer, they do not need to be covered again on the following layer. It should be noted that this design provides an efficient parsing method. Each position of the resulting reconstructed feature map data corresponding to the position of the resulting reconstructed segmentation information is associated only with the segmentation information about a single layer (in N processing layers). There is no overlap. However, the present invention is not limited to this method. It is conceivable that the segmentation information is overlapping, but it may result in maintaining some redundancy.

[0416] like Fig. 27 As already shown in , in some embodiments, the parsed segmentation information element is represented by a set of binary flags. The ordering of the flags in the code stream (syntax) can convey the association between the flags and the layer to which the flags belong. The ordering (sequence) can be given by a predefined processing order on the encoder side, and therefore also by a predefined processing order on the decoder side. For example, Fig.16 and Fig.28 These examples are illustrated in .

[0417] In some exemplary embodiments, for example, in the embodiments mentioned above with reference to the seventh modification, obtaining decoded data for image or video processing includes determining at least one of the following parameters based on the segmentation information. The segmentation information may determine the parsing of additional information in a manner similar to the motion information, such as decoding parameters, which may include intra-image or inter-image prediction mode; image reference index; single reference or multiple reference prediction (including dual prediction); presence or absence of prediction residual information; quantization step size; motion information prediction type; length of motion vector; motion vector resolution; motion vector prediction index; motion vector difference size; motion vector difference resolution; motion interpolation filter; in-loop filter parameters; and / or post-filter parameters, etc. In other words, when processed by a segmentation information processing layer, the segmentation information may specify which processing layer of the decoding parameters may obtain the decoding parameters. For example, in Fig. 22 In the above encoder method of 23, instead of motion information (LayerMv) (or in addition to motion information (LayerMv)), reconstruction (decoding) parameters can be received from the bitstream. Such reconstruction (decoding) parameters blk_rec_params can be received at the decoder side in the same Fig.28 and Fig.29 The motion information is parsed in the same way as shown in .

[0418] Typically, segmentation information is used for parsing and inputting feature map elements (motion information or any reconstruction parameters or sample-related data described above). The method may also include obtaining a set of feature map elements from the bitstream, and inputting the set of feature map elements into feature map processing layers in multiple layers respectively according to the segmentation information processed by the segmentation information processing layer. In addition, the method also includes obtaining decoded data for image or video processing based on feature maps processed by multiple cascade layers. Specifically, in some embodiments, at least one cascade layer of the multiple cascade layers is a segmentation information processing layer and a feature map processing layer. As described above, the network can be designed with separate segmentation information processing layers and feature map processing layers, or with a combination of these two functions. In some implementations, each of the multiple layers is either a segmentation information processing layer or a feature map processing layer.

[0419] The above method can be embodied as a computer program product stored on a non-transitory medium, and when the computer program product is executed on one or more processors, the processor performs the steps of any of the methods. Similarly, a device for decoding an image or video is provided, comprising a processing circuit, which is used to perform the method steps of any of the above methods. The functional structure of the device also provided by the present invention can correspond to the functions provided by the above embodiments and steps. For example, a device for decoding data for image or video processing from a bitstream is provided, the device comprising: an acquisition unit for acquiring two or more segmentation information element sets from the bitstream; an input unit for inputting each of the two or more segmentation information element sets into two or more segmentation information processing layers in a plurality of cascade layers; a processing unit for processing the corresponding segmentation information set in each of the two or more segmentation information processing layers; a decoded data acquisition unit for acquiring decoded data for image or video processing according to the segmentation information processed in the plurality of cascade layers. These units and other units can perform all the functions of the above method.

[0420] Brief Overview of Some Embodiments

[0421] Embodiments for encoding using feature information or segmentation information

[0422] According to one aspect of the present invention, a method for encoding data for image or video processing into a bitstream is provided, the method comprising: processing data, the processing comprising generating feature maps in multiple cascaded layers, each feature map comprising a corresponding resolution, wherein the resolutions of at least two of the generated feature maps are different from each other; selecting a layer different from a layer generating a feature map of the lowest resolution from multiple layers; generating a bitstream, comprising inserting information related to the selected layer into the bitstream.

[0423] This approach can improve the efficiency of such coding because it supports encoding data from different layers, thereby including features of different resolutions or other types of layer-related information in the codestream.

[0424] According to one aspect of the present invention, there is provided a device for encoding data for image or video processing into a bitstream, the device comprising: a processing unit for processing the data, the processing comprising: generating feature maps with different resolutions in a plurality of cascaded layers, each feature map comprising a corresponding resolution; a selection unit for selecting a layer different from a layer generating a feature map of the lowest resolution from a plurality of layers; a generation unit for generating a bitstream, comprising inserting an indication of data associated with the selected layer into the bitstream. The processing unit, the selection unit and the generation unit may be implemented by a processing circuit, such as one or more processors or any combination of software and hardware.

[0425] Such a device can improve the efficiency of such decoding because it can decode data from different layers and use them for reconstruction, thereby making it possible to use features of different resolutions or other types of layer-related information.

[0426] In an exemplary implementation, the processing also includes downsampling by one or more cascaded layers. The application of downsampling can reduce the complexity of the processing on the one hand, and on the other hand, it can also reduce the data provided in the bitstream. In addition, layers processing different resolutions can focus on features of different scales in this way. Therefore, the network processing images (still images or videos) can run efficiently.

[0427] For example, one or more downsampling layers include average pooling or max pooling for downsampling. Average pooling and max pooling operations are part of several frameworks that provide efficient methods for downsampling with low complexity.

[0428] In another example, convolution is used for downsampling. Convolution can provide some more sophisticated downsampling methods, using kernels that can be appropriately selected for specific applications and even trained. This enables a learnable downsampling process to find a more appropriate latent representation of motion information and maintain the advantages of representing and transmitting information of different spatial resolutions, thereby improving adaptability.

[0429] In an exemplary implementation, the information related to the selected layer includes elements of a feature map for the layer.

[0430] By providing features with different resolutions, the scalability of encoding / decoding is improved, and the resulting bitstream can provide more flexibility to meet optimization criteria such as rate, distortion, complexity, etc., which may ultimately improve decoding efficiency.

[0431] For example, in any of the above examples, the information related to the selected layer includes information indicating from which layer and / or from which portion of the feature map of the layer the elements of the feature map of the layer are selected.

[0432] Indicating segmentation information can provide efficient decoding of feature maps from different layers, so that each area of ​​the original (to be decoded) feature map (data) can be covered by information from only one layer. Although this is not a limitation of the present invention, in some cases, the present invention can also provide overlap between layers for specific areas in the feature map (data) to be encoded.

[0433] The above method includes a step of obtaining data to be encoded in an exemplary implementation; wherein, processing the data to be encoded includes processing by each layer j of multiple (N) cascaded layers, including: if j=1, obtaining the data to be encoded as the layer input, otherwise obtaining the feature map processed by the (j-1)th layer as the layer input; processing the obtained layer input, the processing including downsampling; outputting the downsampled feature map.

[0434] Correspondingly, in an exemplary implementation, the above-mentioned device has a processing unit, which is used to obtain the data to be encoded and perform processing of the data to be encoded, including processing by each layer j of multiple (N) cascaded layers, including: if j=1, obtain the data to be encoded as the layer input, otherwise, obtain the feature map processed by the (j-1)th layer as the layer input; process the obtained layer input, and the processing includes downsampling; output the downsampled feature map.

[0435] In some embodiments, a method according to any of the above examples includes: selecting information for insertion into a code stream, the information being related to a first region in a feature map processed by layer j (j>1), wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j including multiple elements; in a selection made from the feature map processed by layer k, an area corresponding to the first region is not selected, wherein k is an integer equal to or greater than 1, and k<j.

[0436] In some embodiments, an apparatus according to any of the above examples includes a processing circuit, which is also used to: select information for insertion into a code stream, the information being related to a first region in a feature map processed by layer j (j>1), wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j including multiple elements; in a selection made from a feature map processed by layer k, an area corresponding to the first region is not selected, wherein k is an integer equal to or greater than 1, and k<j.

[0437] This choice in some layers not to cover areas of the original feature map covered by other layers may be particularly efficient in terms of decoding overhead.

[0438] For example, in any of the above examples, the data to be encoded includes image information and / or prediction residual information and / or prediction information.

[0439] Alternatively, the information associated with the selected layer includes prediction information.

[0440] For example, in any of the examples above, the data associated with the selected layer includes an indication of a location of a feature map element in a feature map of the selected layer.

[0441] This indication enables to correctly associate feature map elements of different resolutions with input data regions.

[0442] For example, in any of the above examples, the positions of selected and unselected feature map elements are indicated by a plurality of binary flags, the indication being based on the positions of the flags in the codestream.

[0443] The binary flag provides a particularly efficient way of decoding the segmentation information.

[0444] According to one embodiment, in the above method or device, the processing of layer j among multiple (N) cascaded layers includes: determining a first cost generated by reconstructing a part of a reconstructed image using feature map elements output by the jth layer, and determining a second cost generated by reconstructing a part of an image using feature map elements output by the (j–1)th layer; if the first cost is higher than the second cost, selecting the (j–1)th layer and selecting information related to the part in the (j–1)th layer.

[0445] Providing optimization including distortion provides an efficient means of achieving the required quality.

[0446] For example, the first cost and the second cost include data volume and / or distortion. By optimizing by considering the rate (the amount of data generated by the encoder) and the distortion of the reconstructed image, the requirements of various applications or users can be flexibly met.

[0447] Alternatively or in addition, the data to be encoded is a motion vector field. The above methods are easily applicable to compressed motion vector fields, such as dense optical flow or sub-sampled optical flow. Applying these methods can provide efficient (in terms of rate and distortion or other criteria) decoding of motion vectors and can further reduce the code stream size of the encoded image or video data.

[0448] In some embodiments, the prediction information includes a reference index and / or a prediction mode. In addition to or instead of the motion vector field, other information related to the prediction may be processed. The reference index and the prediction mode may be similar to the motion vector field and may be related to the content of the image, so encoding feature map elements with different resolutions may improve efficiency.

[0449] For example, the amount of data includes the amount of data required to send data associated with the selected layer. In this way, during the optimization process, the overhead caused by providing information associated with a layer different from the output layer can be considered.

[0450] Additionally or alternatively, the distortion is calculated by comparing the reconstructed image with the target image. This end-to-end quality comparison ensures that the distortion in the reconstructed image is properly accounted for. As a result, the optimization may be able to select the decoding method in an efficient manner and meet the quality requirements imposed by the application or user in a more accurate manner.

[0451] For example, in any of the examples above, the processing includes additional convolutional layers between cascaded layers having different resolutions.

[0452] Providing such additional layers in a cascaded layer network enables the introduction of additional processing, such as various types of filtering, to improve the quality or efficiency of decoding.

[0453] According to an exemplary implementation, the method or processing circuit of the device according to the above embodiment includes: in layer downsampling, downsampling the input feature map using a first filter to obtain a first feature map; downsampling the input feature map using a second filter to obtain a second feature map; determining a third cost generated by reconstructing a portion of the reconstructed image using the first feature map; determining a fourth cost generated by reconstructing a portion of the reconstructed image using the second feature map; in selection, if the third cost is less than the fourth cost, selecting the first feature map.

[0454] Applying different downsampling filters may help to adapt to different characteristics of the content.

[0455] For example, the shapes of the first filter and the second filter may be any non-square, horizontally oriented rectangular, and vertically oriented rectangular.

[0456] The shapes of these filters are still simple, but may provide additional improvements in adapting to object boundaries.

[0457] The method steps or steps performed by the processing circuit of the device may further include obtaining a mask, wherein the mask consists of flags, the mask represents an arbitrary filter shape, and the first filter or the second filter has an arbitrary filter shape.

[0458] This provides flexibility in designing filters of any shape.

[0459] The method steps or steps performed by the processing circuit of the device may also include processing data related to the same image partitioned into blocks with different block sizes and shapes in different layers, wherein the selection includes: selecting the layer according to the cost calculated for a predetermined set of decoding modes.

[0460] In some example implementations, the processing includes at least one layer determining costs of different sets of coding modes and selecting one of the sets of coding modes based on the determined costs.

[0461] Applying the optimization to the decoding mode can achieve efficient rate-distortion optimization, thereby improving decoding efficiency.

[0462] For example, the indication of data associated with the selected layer includes the selected set of coding modes.

[0463] According to one aspect of the present invention, there is provided a computer program stored in a non-transitory medium, the computer program comprising codes, which when executed by one or more processors, perform the steps of any of the above methods.

[0464] According to an aspect of the present invention, there is provided a device for encoding an image or a video, comprising a processing circuit, wherein the processing circuit is configured to execute a method according to any of the above examples.

[0465] Any of the above devices can be implemented on an integrated chip. The present invention can be implemented by hardware (HW) and / or software (SW). In addition, the HW-based implementation can be combined with the SW-based implementation.

[0466] It should be noted that the present invention is not limited to a specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0467] For the sake of clarity, any of the embodiments described above may be combined with any one or more of the other embodiments described above to create new embodiments within the scope of the present invention.

[0468] Embodiments of decoding using segmentation information

[0469] According to one aspect, a method for decoding data for image or video processing from a bitstream is provided, the method comprising: obtaining two or more segmentation information element sets from the bitstream; inputting each of the two or more segmentation information element sets into two or more segmentation information processing layers in a plurality of cascade layers, respectively; in each of the two or more segmentation information processing layers, processing the respective segmentation information set, wherein obtaining decoded data for image or video processing is performed based on the segmentation information processed by the plurality of cascade layers.

[0470] This method can improve efficiency because it can decode data in various segments that can be configured on a layer basis in a hierarchical structure. The provision of the segmented parts can take into account characteristics of the decoded data.

[0471] For example, obtaining the set of segmentation information elements is based on the segmentation information processed by at least one segmentation information processing layer among a plurality of cascaded layers.

[0472] In some exemplary embodiments, the input set of segmentation information elements is based on the processed segmentation information output by at least one of the plurality of cascaded layers.

[0473] Cascaded segmentation information processing can efficiently parse the segmentation information.

[0474] For example, the resolutions of the segmentation information processed separately in two or more segmentation information processing layers are different.

[0475] In some embodiments and examples, the processing of the segmentation information in two or more segmentation information processing layers includes upsampling.

[0476] The hierarchical structure of the segmentation information can provide a small amount of side information to be inserted into the bitstream, thereby improving efficiency and / or reducing processing time.

[0477] Specifically, the upsampling of the segmentation information includes nearest neighbor upsampling. Nearest neighbor upsampling has a low computational complexity and can be easily implemented. Nevertheless, this upsampling is efficient, especially for logical indicators such as flags. For example, the upsampling of the segmentation information includes transposed convolution. Performing upsampling can improve the upsampling quality. In addition, such a convolutional upsampling layer can be provided as trainable or configurable on the decoder side, such that the convolutional kernel can be controlled by an indication parsed from the bitstream or otherwise derived.

[0478] In an exemplary implementation, for each segmentation information processing layer j among the multiple (N) segmentation information processing layers of the multiple cascaded layers: the input includes, if j = 1, the initial segmentation information input from the bitstream, otherwise the segmentation information processed by the (j - 1)th segmentation information processing layer; and the output is the processed segmentation information.

[0479] For example, the processing of the input segmentation information by each layer j (j < N) among the multiple (N) segmentation information processing layers further includes: parsing the segmentation information elements from the bitstream and associating the parsed segmentation information elements with the segmentation information output by the previous layer, wherein the position of the parsed segmentation information elements in the associated segmentation information is determined according to the segmentation information output by the previous layer. Specifically, the amount of the segmentation information elements parsed from the bitstream is determined according to the segmentation information output by the previous layer. For example, the parsed segmentation information elements are represented by a set of binarized flags.

[0480] This layered structure provides parallelizable processing and can be easily run on GPU / NPU and can exploit parallelism. The fully trainable scheme of transferring gradients supports the use of this scheme in an end-to-end trainable video coding scheme.

[0481] In some exemplary embodiments and examples, obtaining decoded data for image or video processing includes determining at least one of the following based on the segmentation information: intra-frame or inter-frame prediction mode; image reference index; single reference or multiple reference prediction (including bidirectional prediction); presence or absence of prediction residual information; quantization step size; motion information prediction type; length of motion vector; motion vector resolution; motion vector prediction index; motion vector difference size; motion vector difference resolution; motion interpolation filter; in-loop filter parameters; post-filter parameters. The decoding of the present invention is generally applicable to any type of data related to image or video decoding.

[0482] The method of the above-mentioned embodiment or example may also include: obtaining a feature map element set from the code stream according to the segmentation information processed by the segmentation information processing layer, and inputting the feature map element set into the feature map processing layers in multiple layers respectively; obtaining decoded data for image or video processing according to the feature maps processed by multiple cascade layers.

[0483] Specifically, at least one of the plurality of cascade layers is a segmentation information processing layer and a feature map processing layer. In other embodiments, each of the plurality of layers is a segmentation information processing layer, or a feature map processing layer.

[0484] Separating the layer functions provides a clean design and functional separation. However, the present invention can also work if the layer implements both functions.

[0485] According to one aspect, a computer program product stored on a non-transitory medium is provided, which, when executed on one or more processors, performs the method according to any of the above examples and embodiments.

[0486] According to one aspect, a device for decoding an image or a video is provided, comprising a processing circuit, wherein the processing circuit is configured to execute a method according to any of the above examples and embodiments.

[0487] According to one aspect, a device for decoding data for image or video processing from a bitstream is provided, the device comprising: an acquisition unit for acquiring two or more segmentation information element sets from the bitstream; an input unit for inputting each of the two or more segmentation information element sets into two or more segmentation information processing layers in a plurality of cascaded layers; a processing unit for processing the respective segmentation information set in each of the two or more segmentation information processing layers; and a decoded data acquisition unit for acquiring decoded data for image or video processing based on the segmentation information processed in the plurality of cascaded layers.

[0488] Any of the above devices can be implemented on an integrated chip. The present invention can be implemented by hardware (HW) and / or software (SW). In addition, the HW-based implementation can be combined with the SW-based implementation.

[0489] It should be noted that the present invention is not limited to a specific framework. In addition, the present invention is not limited to image or video compression, and can also be applied to object detection, image generation and recognition systems.

[0490] Some exemplary implementations in hardware and software

[0491] A corresponding system of the above encoder-decoder processing chain can be deployed as Fig.35 shown. Fig.35 1 is a schematic block diagram of an exemplary decoding system (e.g., a video, image, audio, and / or other decoding system (or simply a decoding system)) that can use the techniques in the present application. The video encoder 20 (or simply encoder 20) and the video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform techniques according to various examples described in the present application. For example, video encoding and decoding can use neural networks or generally use those processing networks described in the above embodiments and examples.

[0492] like Fig.35 As shown, the decoding system 10 includes a source device 12 for providing the encoded image data 21 to a destination device 14 or the like for decoding the encoded image data 13 .

[0493] The source device 12 includes an encoder 20 and may additionally (ie, optionally) include an image source 16 , a pre-processor (pre-processing unit) 18 (eg, an image pre-processor 18 ), and a communication interface or communication unit 22 .

[0494] The image source 16 may include or may be: any type of image capture device, such as a camera for capturing real-world images; and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images; or any type of device for acquiring and / or providing real-world images, computer-animated images (e.g., screen content, virtual reality (VR) images), and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device that stores any of the above images.

[0495] In order to distinguish them from the pre-processing unit 18 and the processing performed by the pre-processing unit 18 , the image or the image data 17 may also be referred to as a raw image or raw image data 17 .

[0496] The preprocessor 18 is used to receive (raw) image data 17 and preprocess the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction or denoising, etc. It is understood that the preprocessing unit 18 may be an optional component. It should be noted that the preprocessing may also use a neural network.

[0497] The video encoder 20 is configured to receive the pre-processed image data 19 and provide encoded image data 21 .

[0498] The communication interface 22 in the source device 12 may be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0499] The destination device 14 includes a decoder 30 (eg, a video decoder 30 ), and may additionally (ie, optionally) include a communication interface or communication unit 28 , a post-processing unit 32 (or post-processing units 32 ), and a display device 34 .

[0500] The communication interface 28 of the destination device 14 is used to receive the encoded image data 21 or (or any other processed version), for example directly from the source device 12, or from any other source (such as a storage device, such as an encoded image data storage device), and to provide the encoded image data 21 to the decoder 30.

[0501] The communication interface 22 and the communication interface 28 can be used to send or receive the encoded image data 21 or the encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between the source device 12 and the destination device 14, or via any type of network (e.g., a wired network, a wireless network, any combination of wired and wireless networks, any type of private and public networks, or any type of combination of private and public networks).

[0502] For example, the communication interface 22 may be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or use any type of transmission coding or processing to process the encoded image data for transmission over a communication link or a communication network.

[0503] For example, the communication interface 28 (corresponding to the communication interface 22 ) may be used to receive the transmitted data and process the transmission data using any type of corresponding transmission decoding or processing and / or decapsulation process to obtain the encoded image data 21 .

[0504] Both the communication interface 22 and the communication interface 28 can be used as Fig.35 The decoder 30 is used to receive the encoded image data 21 and provide decoded image data 31 or a decoded image 31 (for example, using the neural network described in the embodiments and examples described above).

[0505] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data), such as the decoded image 31, to obtain post-processed image data 33, such as the post-processed image 33. The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, cropping or resampling, or any other processing, for example, to prepare the decoded image data 31 for display through a display device 34, etc.

[0506] The display device 34 of the destination device 14 is used to receive the post-processed image data 33 to display the image, for example, to a user or viewer. The display device 34 can be or include any type of display for presenting the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0507] although Fig.35 The source device 12 and the destination device 14 are shown as separate devices, but embodiments of the device may also include both devices or the functions of both devices, that is, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions. In such embodiments, the source device 12 or the corresponding functions and the destination device 14 or the corresponding functions may be implemented by the same hardware and / or software, by separate hardware and / or software, or any combination thereof.

[0508] From the above description, it is obvious to those skilled in the art that Fig.35 The presence and the (precise) functional division of the different units or functions of the source device 12 and / or the destination device 14 shown in FIG. 1 may vary depending on the actual device and application.

[0509] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30 may be implemented by a processing circuit, which is one or more microprocessors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), discrete logic, hardware, video decoding dedicated processors, or any combination thereof. The encoder 20 may be implemented by a processing circuit 46 to embody various modules including a neural network. The decoder 30 may be implemented by a processing circuit 46 to implement various modules as discussed in the above embodiments and examples. The processing circuit may be used to perform the various operations described herein. If these techniques are partially implemented in software, the device may store the software instructions in a suitable non-transitory computer-readable storage medium, and these instructions may be executed in hardware by one or more processors to implement the techniques of the present invention. Either of the video encoder 20 and the video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC), such as Fig.36 shown.

[0510] Source device 12 and destination device 14 may include any of a variety of devices, including any type of handheld or fixed device, such as a notebook or laptop computer, a mobile phone, a smart phone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, a video streaming device (such as a content service server or a content distribution server), a broadcast receiver device, a broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be used for wireless communication. Therefore, source device 12 and destination device 14 may be wireless communication devices.

[0511] In some cases, Fig.35 The video decoding system 10 shown is exemplary only, and the techniques of the present application may be applied to video decoding settings (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding device and the decoding device. In other examples, the data can be retrieved from a local memory, streamed over a network, etc. The video encoding device can encode the data and store the data to a memory, and / or the video decoding device can retrieve and decode the data from the memory. In some examples, the encoding and decoding are performed by devices that do not communicate with each other but simply encode the data to the memory and / or retrieve and decode the data from the memory.

[0512] Fig.37 3700 is a schematic diagram of a video decoding device 3700 according to an embodiment of the present invention. The video decoding device 3700 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video decoding device 3700 may be a decoder (e.g. Fig.35 A video decoder 30) or an encoder (such as Fig.35 video encoder 20).

[0513] The video decoding device 3700 includes: an input port 3710 (or input port 3710) and a receiving unit (Rx) 3720 for receiving data; a processor, a logic unit or a central processing unit (CPU) 3730 for processing the data; a transmitting unit (Tx) 3740 and an output port 3750 (or output port 3750) for transmitting the data; and a memory 3760 for storing the data. The video decoding device 3700 may also include an optical-to-electrical (OE) component and an electrical-to-optical (EO) component coupled to the input port 3710, the receiving unit 3720, the transmitting unit 3740 and the output port 3750, which are used as an outlet or an inlet of an optical signal or an electrical signal.

[0514] The processor 3730 is implemented by hardware and software. The processor 3730 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 3730 communicates with the input port 3710, the receiving unit 3720, the sending unit 3740, the output port 3750, and the memory 3760. The processor 3730 includes a decoding module 3770. The decoding module 3770 implements the disclosed embodiments described above. For example, the decoding module 3770 performs, processes, prepares, or provides various decoding operations. Therefore, the decoding module 3770 is included, which provides substantial improvements to the functions of the video decoding device 3700 and affects the transformation of the video decoding device 3700 to different states. Alternatively, the decoding module 3770 is implemented with instructions stored in the memory 3760 and executed by the processor 3730.

[0515] The memory 3760 may include one or more disks, tape drives, and solid-state hard disks, and may be used as an overflow data storage device to store programs when such programs are selected for execution, as well as to store instructions and data read during program execution. For example, the memory 3760 may be volatile and / or non-volatile, and may be a read-only memory (ROM), a random access memory (RAM), a ternary content-addressable memory (TCAM), and / or a static random-access memory (SRAM).

[0516] Fig.38 is a simplified block diagram of an apparatus 3800 according to an exemplary embodiment, which can be used as Fig.35 Either or both of the source device 12 and the destination device 14 in .

[0517] The processor 3802 in the device 3800 may be a central processing unit. Alternatively, the processor 3802 may be any other type of device or devices capable of manipulating or processing information now existing or later developed. Although the disclosed implementations may be implemented by a single processor (e.g., the processor 3802), speed and efficiency may be increased by more than one processor.

[0518] The memory 3804 in the apparatus 1100 may be a read only memory (ROM) device or a random access memory (RAM) device in one implementation. Any other suitable type of storage device may be used as the memory 3804. The memory 3804 may include code and data 3806 accessed by the processor 3802 using a bus 3812. The memory 3804 may also include an operating system 3808 and an application 3810, which includes at least one program that causes the processor 3802 to perform the methods described herein. For example, the application 3810 may include applications 1 to N, which also include an image decoding (encoding or decoding) application that performs the methods described herein.

[0519] The apparatus 3800 may also include one or more output devices, such as a display 3818. In one example, the display 3818 may be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. The display 3818 may be coupled to the processor 3802 via the bus 3812.

[0520] Although described here as a single bus, the bus 3812 of the device 3800 may be composed of multiple buses. In addition, the auxiliary storage device may be directly coupled to other components of the device 3800, or may be accessed through a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, the device 3800 can be implemented in a variety of configurations.

[0521] In summary, the present invention relates to a method and apparatus for encoding data (for still image or video processing) into a bitstream. Specifically, the data is processed by a network comprising a plurality of cascaded layers. In the process, feature maps are generated by layers. The feature maps processed (output) by at least two different layers have different resolutions. In the process, a layer is selected from the cascaded layers, which is different from the layer that generates the lowest resolution (such as latent space) feature map. The bitstream includes information related to the selected layer. By this method, a scalable process that can operate at different resolutions is provided, so that the bitstream can transmit information related to such different resolutions. Therefore, the data can be efficiently decoded in the bitstream according to the resolution, and the resolution can vary according to the content of the decoded image data.

[0522] The present invention also relates to a method and apparatus for decoding data (for still image or video processing) from a code stream. Specifically, two or more feature map element sets are obtained from the code stream. Each feature map element set is associated with a feature map. Then, each of the two or more feature map element sets is input into two or more feature map processing layers in a plurality of cascaded layers. Then, decoded data for image or video processing is obtained as a result of the processing of the plurality of cascaded layers. Therefore, the data can be decoded from the code stream in an efficient manner in a hierarchical structure.

[0523] The present invention also relates to a method and apparatus for decoding data (for still image or video processing) from a code stream. Two or more sets of segmentation information elements are obtained from the code stream. Then, each of the two or more sets of segmentation information elements is input into two or more segmentation information processing layers in a plurality of cascaded layers. In each of the two or more segmentation information processing layers, the respective segmentation information set is processed. The decoded data for image or video processing is obtained based on the segmentation information processed by the plurality of cascaded layers. Therefore, the data can be decoded from the code stream in an efficient manner in a hierarchical structure.

Claims

1. A method for decoding data for image or video processing from a bitstream, characterized in that: The method includes: Obtaining two or more sets of feature map elements from the bitstream, where each set of feature map elements is related to a feature map, Separately inputting each of the two or more sets of feature map elements into two or more feature map processing layers in a plurality of cascaded layers, and processing the feature map in each of the two or more feature map processing layers, where the resolutions of the feature maps processed in the two or more feature map processing layers are different; Obtaining the decoded data for image or video processing as the result of the processing by the plurality of cascaded layers.

2. The method according to claim 1, wherein Processing the feature map in two or more feature map processing layers includes upsampling.

3. The method according to claim 1 or 2, characterized in that: It further includes: Obtaining segmentation information related to the two or more layers from the bitstream, where Obtaining the feature map elements from the bitstream based on the segmentation information; Inputting the sets of feature map elements into two or more feature map processing layers respectively based on the segmentation information.

4. The method according to claim 3, wherein The plurality of cascaded layers further includes a plurality of segmentation information processing layers, The method further includes: Processing the segmentation information in the plurality of segmentation information processing layers.

5. The method according to claim 4, wherein Processing the segmentation information in at least one of the plurality of segmentation information processing layers includes upsampling.

6. The method according to claim 5, characterized in that The upsampling of the segmentation information and / or the upsampling of the feature map includes nearest neighbor upsampling.

7. The method according to claim 5 or 6, characterized in that: The upsampling of the segmentation information and / or the upsampling of the feature map includes transposed convolution.

8. The method according to any one of claims 4 to 6, wherein Obtaining the feature map elements from the bitstream based on the processed segmentation information processed by at least one of the plurality of segmentation information processing layers.

9. The method according to any one of claims 4 to 6, wherein Separately inputting each of the two or more sets of feature map elements into two or more feature map processing layers based on the processed segmentation information processed by at least one of the plurality of segmentation information processing layers.

10. The method according to any one of claims 4 to 6, wherein The obtained segmentation information is represented by a set of syntax elements, where the position of the elements in the set of syntax elements indicates which feature map element position the syntax element is related to, where processing the feature map includes, for each of the syntax elements: - When the syntax element is a first value, parsing the element of the feature map at the position indicated by the position of the syntax element in the bitstream from the bitstream, - Otherwise, not parsing the element of the feature map at the position indicated by the position of the syntax element in the bitstream from the bitstream.

11. The method according to any one of claims 4 to 6, characterized in that Processing the feature map by each layer j (1 < j < N) of the plurality of (N) feature map processing layers further includes: Parsing the segmentation information element of the j-th feature map processing layer from the bitstream; Obtaining the feature map processed by the previous feature map processing layer, A feature map element is parsed from the code stream and the parsed feature map element is associated with the acquired feature map, wherein the position of the feature map element in the processed feature map is indicated by the parsed segmentation information element and the segmentation information is processed by a previous segmentation information processing layer.

12. The method according to claim 10, characterized in that - when the syntax element is a first value, parsing elements of the feature map from the code stream, -When the syntax element is the second value or the partition information processed by the previous partition information processing layer is the first value, the element of the feature map is not parsed from the code stream.

13. The method according to claim 10, characterized in that The syntax element parsed from the code stream indicating the partition information is a binarization flag.

14. The method according to claim 13, characterized in that The processed segmentation information is represented by a binary flag set.

15. The method according to any one of claims 12 to 14, characterized in that Upsampling the segmentation information in each segmentation information processing layer j further includes: -For each p-th position in the acquired feature map indicated by the input segmentation information, - determining as up-sampling segmentation information an indication of a feature map position included in the same region as said p-th position in the reconstructed image.

16. The method according to claim 1 or 2, characterized in that: The data used for image or video processing includes image data and / or prediction residual data and / or prediction information data.

17. The method according to claim 1 or 2, characterized in that: The data for image or video processing includes a motion vector field.

18. The method according to claim 3, characterized in that The upsampling of the feature map uses a filter, and the shape of the filter is any one of a square, a horizontal rectangle, and a vertical rectangle.

19. The method according to claim 3, characterized in that The upsampling of the feature map uses a filter, and the input of information from the bitstream also includes: Information indicating the filter shape and / or filter coefficients is obtained from the code stream.

20. The method according to claim 19, characterized in that The information indicating the filter shape indicates a mask consisting of flags, the mask representing the filter shape, in that a flag having a third value indicates a non-zero filter coefficient and a flag having a fourth value different from the third value indicates a zero filter coefficient.

21. The method according to claim 3, characterized in that The multiple cascade layers include convolutional layers, and no upsampling is performed between layers of different resolutions.

22. A computer program product stored in a non-transitory medium, characterized in that When the computer program product is executed on one or more processors, the method according to any one of claims 1 to 21 is performed.

23. A device for decoding an image or a video, characterized in that Comprising a processing circuit for performing a method according to any one of claims 1 to 21.

24. A device for decoding data for image or video processing from a bit stream, characterized in that The device comprises: an acquisition unit, configured to acquire two or more feature map element sets from the bitstream, wherein each feature map element set is associated with a feature map, An input unit, configured to input each of the two or more feature map element sets into two or more feature map processing layers in the plurality of cascade layers, respectively, and process the feature map in each of the two or more feature map processing layers, wherein the resolutions of the feature maps processed in the two or more feature map processing layers are different; A decoded data acquisition unit is used to acquire decoded data for image or video processing as a result of the processing of the multiple cascade layers.

Images