Signaling decoding using segmentation information

Abstract

The invention also relates to a method and an apparatus for decoding data (for still image or video processing) from a code stream. And obtaining two or more than two segmentation information element sets from the code stream. Then, each split information element set of the two or more split information element sets is respectively input into two or more split information processing layers of a plurality of cascade layers. In each of the two or more divided information processing layers, the respective divided information set is processed. And acquiring decoded data for image or video processing according to the segmentation information processed by the plurality of cascade layers. Thus, the data may be decoded from the code stream in a hierarchical structure in an efficient manner.

Description

[0001] This application is a divisional application. The original application has the application number 202080108181.3 and the original application date is December 24, 2020. The entire contents of the original application are incorporated herein by reference. Technical Field

[0002] The embodiments of the present invention generally relate to the field of decoding data for image or video processing from a bitstream using multiple processing layers. Specifically, some embodiments relate to methods and apparatus for such decoding. Background Technology

[0003] For decades, hybrid image and video codecs have been used to compress image and video data. In such codecs, the signal is typically encoded block-by-block by predicting blocks and decoding the difference between the original block and its predicted block. Specifically, this decoding can include transform, quantization, and bitstream generation, often involving some entropy decoding. Typically, the three components of a hybrid decoding approach—transform, quantization, and entropy decoding—are optimized separately. Modern video compression standards, such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC), also use transform representations to decode the predicted residual signal.

[0004] Recently, machine learning has been applied to image and video decoding. Generally, machine learning can be applied to image and video decoding in a variety of different ways. For example, end-to-end optimized image or video coding schemes have been discussed. Furthermore, machine learning has been used to determine or optimize certain parts of end-to-end decoding, such as the selection of prediction parameters or compression. What these applications have in common is the generation of feature map data, which is transmitted between the encoder and decoder. An efficient structure for the bitstream can significantly help reduce the number of bits required to encode the image / video source signal.

[0005] Neural networks typically consist of two or more layers. Feature maps are the outputs of layers. In a neural network partitioned between devices (e.g., between encoders and decoders, between a device and the cloud, or between different devices), the feature map at the output of the partition location (e.g., the first device) is compressed and transmitted to the remaining layers of the neural network (e.g., to the second device).

[0006] Further improvements to encoding and decoding may be needed using a trained network architecture. Summary of the Invention

[0007] Some embodiments of the present invention provide methods and apparatus for efficiently decoding images and for adapting some scalability to desired parameters and content.

[0008] The above and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent from the dependent claims, the specification, and the drawings.

[0009] According to one aspect, a method is provided for decoding data for image or video processing from a bitstream, wherein the method includes: acquiring two or more sets of segmentation information elements from the bitstream; inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers; and processing a respective set of segmentation information in each of the two or more segmentation information processing layers, wherein the decoded data for image or video processing is acquired based on the segmentation information processed in the plurality of cascaded layers.

[0010] This approach can improve efficiency because it can decode data in various segments based on layer configuration within a hierarchical structure. The segmentation can take into account the characteristics of the decoded data.

[0011] For example, the acquisition of the segmentation information element set is performed based on segmentation information processed by at least one segmentation information processing layer among the plurality of cascaded layers.

[0012] In some exemplary embodiments, the input of the segmentation information element set is based on processed segmentation information output by at least one of the plurality of cascaded layers.

[0013] Cascaded segmentation information processing can efficiently parse the segmentation information.

[0014] For example, the resolution of the segmentation information processed in the two or more segmentation information processing layers is different.

[0015] In some embodiments and examples, the processing of the segmented information in the two or more segmented information processing layers includes upsampling.

[0016] The hierarchical structure of segmented information can provide a small amount of side information to be inserted into the bitstream, thereby improving efficiency and / or shortening processing time.

[0017] Specifically, the upsampling of the segmentation information includes nearest-neighbor upsampling. Nearest-neighbor upsampling has low computational complexity and can be easily implemented. Nevertheless, nearest-neighbor upsampling is still efficient, especially for logical indicators such as flags. For example, the upsampling of the segmentation information includes transposed convolutions. Performing upsampling can improve the upsampling quality. Furthermore, such convolutional upsampling layers can be provided as trainable or configurable at the decoding end, allowing the convolutional kernels to be controlled by indicators parsed from the bitstream or otherwise derived.

[0018] In an exemplary implementation, for each segmentation information processing layer j among the multiple (N) segmentation information processing layers in the plurality of cascaded layers, the input includes: if j=1, then inputting initial segmentation information from the bitstream; otherwise, inputting segmentation information processed by the (j-1)th segmentation information processing layer; and outputting processed segmentation information.

[0019] For example, the processing of the input segmentation information by each of the plurality (N) segmentation information processing layers, where j < N, further includes: parsing segmentation information elements from the bitstream, and associating the parsed segmentation information elements with the segmentation information output by the previous layer, wherein the position of the parsed segmentation information element in the associated segmentation information is determined based on the segmentation information output by the previous layer. Specifically, the number of segmentation information elements parsed from the bitstream is determined based on the segmentation information output by the previous layer. For example, the parsed segmentation information elements are represented by a binary flag set.

[0020] This hierarchical structure provides parallel processing capabilities and can be easily run on GPUs / NPUs, leveraging parallelism. The fully trainable scheme for transporting gradients can be used in end-to-end trainable video decoding solutions.

[0021] In some exemplary embodiments and examples, obtaining decoded data for image or video processing includes determining at least one of the following based on segmentation information: intra-frame or inter-frame prediction mode; image reference index; single-reference or multi-reference prediction (including bidirectional prediction); presence or absence of prediction residual information; quantization step size; motion information prediction type; motion vector length; motion vector resolution; motion vector prediction index; motion vector difference magnitude; motion vector difference resolution; motion interpolation filter; in-loop filter parameters; and post-filter parameters. The decoding of this invention is generally applicable to any type of data related to image or video decoding.

[0022] The method provided in the above embodiments or examples may further include: obtaining a feature map element set from the bitstream, and inputting the feature map element set into the feature map processing layer of the plurality of layers respectively according to the segmentation information processed by the segmentation information processing layer; and obtaining decoded data for image or video processing according to the feature maps processed by the plurality of cascaded layers.

[0023] Specifically, at least one of the plurality of cascaded layers is a segmentation information processing layer and a feature map processing layer. In other embodiments, each of the plurality of layers is either a segmentation information processing layer or a feature map processing layer.

[0024] Separating layer functions provides a concise design and separation of functions. However, the invention can also work if the layers implement both functions.

[0025] According to one aspect, a computer program product stored in a non-transitory medium is provided, which, when executed on one or more processors, performs the method described according to the above examples and embodiments.

[0026] According to one aspect, an apparatus for decoding images or videos is provided, including processing circuitry for performing the method according to any of the examples and embodiments described above.

[0027] According to one aspect, an apparatus is provided for decoding data for image or video processing from a bitstream, wherein the apparatus comprises: an acquisition unit for acquiring two or more sets of segmentation information elements from the bitstream; an input unit for inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers; a processing unit for processing a respective set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data acquisition unit for acquiring the decoded data for image or video processing based on the segmentation information processed in the plurality of cascaded layers.

[0028] Any of the above-described devices can be implemented on an integrated chip. This invention can be implemented in hardware (HW) and / or software (SW). Furthermore, hardware-based implementations can be combined with software-based implementations.

[0029] It should be noted that this invention is not limited to a specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0030] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create a new embodiment within the scope of the present invention. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of a channel processed by layers of a neural network.

[0032] Figure 2 This is a schematic diagram of an autoencoder type for neural networks.

[0033] Figure 3A This is a schematic diagram of an exemplary network architecture that includes an encoder and a decoder with a priori model.

[0034] Figure 3B This is a schematic diagram of a general network architecture that includes a super-prior model at the encoding end.

[0035] Figure 3C This is a schematic diagram of a general network architecture for the decoding end, including a super-prior model.

[0036] Figure 4 This is a schematic diagram of an exemplary network architecture including a hyperprior model for both the encoder and decoder ends.

[0037] Figure 5A This is a block diagram of an end-to-end video compression framework based on neural networks.

[0038] Figure 5B A block diagram illustrating some exemplary details of neural network applications for sports field compression.

[0039] Figure 5C A block diagram showing some exemplary details of neural network applications for motion compensation.

[0040] Figure 6 This is a schematic diagram of the layers of U-net.

[0041] Figure 7A This is a block diagram of an exemplary hybrid encoder.

[0042] Figure 7B This is a block diagram of an exemplary hybrid decoder.

[0043] Figure 8 A flowchart of an exemplary method for encoding data for image / video processing (e.g., encoding).

[0044] Figure 9 A block diagram of a network structure that transmits information from layers of different resolutions in a bitstream.

[0045] Figure 10A This is a schematic diagram of max pooling.

[0046] Figure 10B This is a schematic diagram of average pooling.

[0047] Figure 11 This is a schematic diagram illustrating the processing of feature maps and segmentation information by an exemplary encoding end.

[0048] Figure 12 This is a block diagram illustrating the generalized processing of motion information feature maps at the encoding and decoding ends.

[0049] Figure 13 A block diagram of a network structure for processing motion vector-related information, used to transmit information from layers of different resolutions in a bitstream.

[0050] Figure 14 A block diagram of an exemplary cost calculation unit with high cost tensor resolution.

[0051] Figure 15 A block diagram of an exemplary cost calculation unit with low-cost tensor resolution.

[0052] Figure 16 The block diagram illustrates the functional structure of the signal selection logic.

[0053] Figure 17 The block diagram illustrates a functional structure of signal selection logic with one or more cost calculation units that provides several decoding options.

[0054] Figure 18 A block diagram of a network structure with convolutional downsampling and upsampling layers for transmitting information from layers of different resolutions in a bitstream.

[0055] Figure 19 A block diagram of a structure with additional layers for transmitting information from layers of different resolutions in a bitstream.

[0056] Figure 20 A block diagram of a layer structure capable of downsampling or upsampling filter selection for transmitting information from layers of different resolutions in a bitstream.

[0057] Figure 21 A block diagram of a network structure capable of selecting convolutional filters to transmit information from layers of different resolutions in a bitstream.

[0058] Figure 22 This is a block diagram illustrating the functional structure of a network-based RDO decision unit used for selecting decoding modes.

[0059] Figure 23 A block diagram illustrating an exemplary cost calculation unit that can be used in a network-based RDO decision unit for selecting a decoding mode.

[0060] Figure 24 A block diagram of an exemplary cost calculation unit that can be used in a network-based RDO decision unit for selecting a decoding mode that supports multiple options.

[0061] Figure 25 A schematic diagram of possible block partitioning or filter shapes.

[0062] Figure 26 This is a schematic diagram illustrating the derivation of segmented information.

[0063] Figure 27 This is a schematic diagram of the processing of segmented information by the decoding end.

[0064] Figure 28 This is a block diagram of an exemplary signal feed logic for reconstructing dense optical flow.

[0065] Figure 29 This is a block diagram of an exemplary signal feed logic for reconstructing dense optical flow.

[0066] Figure 30 This is a block diagram of a convolutional filter bank.

[0067] Figure 31 This is a block diagram of the upsampling filter bank.

[0068] Figure 32A This is a schematic diagram of upsampling processing using nearest-neighbor copying at the decoding end.

[0069] Figure 32B This is a schematic diagram of upsampling processing using convolution at the decoding end.

[0070] Figure 33 This is a flowchart of an exemplary method for decoding data such as feature map information used when decoding images or videos.

[0071] Figure 34 A flowchart of an exemplary method for encoding data such as segmentation information used when encoding images or videos.

[0072] Figure 35 This is a block diagram illustrating an example of a video decoding system for implementing embodiments of the present invention.

[0073] Figure 36 This is a block diagram of another example of a video decoding system for implementing embodiments of the present invention.

[0074] Figure 37 A block diagram of an example encoding or decoding device.

[0075] Figure 38 A block diagram for another example of an encoding or decoding device. Detailed Implementation

[0076] In the following description, reference is made to the accompanying drawings, which form part of this invention, and which illustrate specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and include structural or logical variations not depicted in the drawings. Therefore, the detailed description below is not intended to be limiting, and the scope of the invention is defined by the appended claims.

[0077] It should be understood that the disclosure relating to the described methods also applies to the devices or systems corresponding to the execution of the methods, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, such as functional units, for performing the described one or more method steps (e.g., a unit performing the one or more steps, or multiple units each performing one or more of the plurality of steps), even if such one or more units are not explicitly described or illustrated in the figures. On the other hand, for example, if a particular apparatus is described according to one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., a step performing the function of the one or more units, or multiple steps each performing the function of one or more of the plurality of units), even if such one or more steps are not explicitly described or illustrated in the figures. Furthermore, it should be understood that, unless specifically indicated otherwise, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0078] Some embodiments aim to improve the quality of encoded and decoded image or video data and / or reduce the amount of data required to represent encoded image or video data. Some embodiments provide methods for efficiently selecting information indicated by the encoder to the decoder. Below, an overview of some of the technical terms and frameworks used in embodiments of the invention is provided.

[0079] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computational systems vaguely inspired by the biological neural networks that make up the animal brain. These systems "learn" to perform tasks by example, typically without being programmed using task-specific rules. For instance, in image recognition, these systems might learn to identify images containing cats—that is, by analyzing exemplary images manually labeled "cat" or "no cat" and using the results to identify cats in other images. These systems do this without prior knowledge of information such as fur, tails, whiskers, and faces. Instead, they automatically generate recognizable features from the examples they process.

[0080] ANNs are based on a set of connection units or nodes called artificial neurons, which loosely model neurons in the biological brain. Each connection, like a synapse in the biological brain, can transmit signals to other neurons. Artificial neurons receive signals, process them, and can then send signals back to the neurons connected to them.

[0081] In ANN implementations, the "signal" at a connection is a real number, and the output of each neuron is calculated as a nonlinear function of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. Weights increase or decrease the signal strength at the connection. Neurons can have thresholds, such that a signal is only sent when the aggregated signal exceeds that threshold. Typically, neurons are clustered into multiple layers. Different layers can perform different transformations on their inputs. The signal may travel from the first layer (input layer) to the last layer (output layer) after multiple traversals of the layers.

[0082] The initial goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention shifted to performing specific tasks, leading to a departure from biology. ANNs have been used for a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, board and video games, medical diagnosis, and even activities traditionally considered to be reserved for humans, such as painting.

[0083] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one layer.

[0084] Figure 1 This diagram schematically illustrates the general concept of processing in a neural network (e.g., a CNN). A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer provides input (e.g., ...). Figure 1 The hidden layers of a CNN (as shown in the image) are processed by a series of convolutional layers that perform convolutions with multiplications or other dot products. The result of each layer is one or more feature maps. Figure 1 The feature maps (f.maps) are sometimes also called channels. Subsampling may be involved in some or all layers. Therefore, the feature maps may become smaller, such as... Figure 1As shown. The activation functions in a CNN are typically rectified linear unit (ReLU) layers, followed by additional convolutions such as pooling layers, fully connected layers, and normalized layers, called hidden layers, because their inputs and outputs are masked by the activation functions and the final convolution. While these layers are colloquially referred to as convolutions, this is just by convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is significant for the indices in the matrix because it affects how weights are determined at a particular index point.

[0085] When programming a CNN to process images, such as Figure 1 As shown, the input is a tensor of shape (number of images) × (image width) × (image height) × (image depth). Then, after passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network should have the following properties: A convolutional kernel defined by width and height (hyperparameters). The number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) should be equal to the number of input feature map channels (depth).

[0086] In the past, traditional multilayer perceptron (MLP) models were used for image recognition. However, due to the full connectivity between nodes, they suffer from high dimensionality and cannot scale sufficiently in high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high for efficient large-scale processing with full connectivity. Furthermore, this network architecture does not consider the spatial structure of the data, processing distant input pixels in the same way as nearby pixels. This ignores the reference locality in image data, both computationally and semantically. Therefore, the full connectivity of neurons is wasted for purposes such as image recognition dominated by spatially local input patterns.

[0087] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons (MLPs) specifically designed to mimic the behavior of the visual cortex. These models alleviate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. The convolutional layer is the core building block of a CNN. The parameters of this layer consist of a set of learnable filters (the kernel mentioned above) that have a small receptive field but extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, the dot product between the filter entry and the input is computed, and a two-dimensional activation map of that filter is generated. Thus, the network learns the filters to activate when they detect certain types of features at a spatial location in the input.

[0088] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region of the input and shares parameters with neurons in the same activation map. A feature map, or activation map, specifies the output activation of a filter. Feature maps and activations have the same meaning. In some papers, a feature map is referred to as an activation map because it is a mapping corresponding to the activations of different parts of an image; it is also a feature map because it is a mapping that finds a certain feature in the image. High activation means that a certain feature has been found.

[0089] Another important concept in CNNs is pooling, a form of non-linear downsampling. Several non-linear functions are used to implement pooling, with max pooling being the most common. It divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0090] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to progressively reduce the size of the representation space, decreasing the number of parameters, memory footprint, and computational cost in the network, and thus also controlling overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of transformation invariance.

[0091] Pooling layers operate independently on each depth strip of the input and are spatially resized. The most common form is a pooling layer that applies a 2×2 filter with a stride of 2 at each depth strip of the input, along both width and height, discarding 75% of the activations. In this case, each max-pooling operation uses more than 4 numbers. The depth dimension remains unchanged. Besides max-pooling, pooling units can also use other functions, such as average pooling or... 2-norm pooling. Average pooling was frequently used in the past, but it has become less common recently compared to max pooling, which often performs better. Due to the significant reduction in representation size, there is a recent trend towards using smaller filters or abandoning pooling layers altogether. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a crucial component of convolutional neural networks for object detection based on the Fast R-CNN architecture.

[0092] The ReLU function, short for Corrected Linear Unit, applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to 0. This increases the non-linearity of the decision function and the overall network without affecting the receptive fields of the convolutional layers. Other functions are also used to add non-linearity, such as the saturating hyperbolic tangent and the sigmoid function. ReLU is generally more popular than other functions because it trains neural networks several times faster without significantly impacting generalization accuracy.

[0093] After several convolutional and max-pooling layers, high-level inference in the neural network is accomplished through fully connected layers. Neurons in a fully connected layer are connected to all activations in the previous layer, as shown in a conventional (non-convolutional) artificial neural network. Therefore, these activations can be computed as affine transformations, followed by bias shifts (learned or vector additions of fixed bias terms) after matrix multiplication.

[0094] The "loss layer" (including the calculation of the loss function) specifies how training penalizes the deviation between the predicted (output) label and the true label, and is usually the last layer of the neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class from K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. Euclidean loss is used to regress to real-valued labels.

[0095] In short, Figure 1 The data flow in a typical convolutional neural network is illustrated. First, the input image passes through a convolutional layer and is abstracted into a feature map comprising several channels, corresponding to multiple filters in a set of learnable filters for that layer. Then, the feature map is subsampled using a pooling layer, which reduces the dimensionality of each channel in the feature map. The data then reaches another convolutional layer, which can have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layers. To establish connections in the network, these parameters need to be synchronized between two connected layers such that the number of input channels in the current layer should equal the number of output channels in the previous layer. For the first layer processing input data such as images, the number of input channels is typically equal to the number of channels used for data representation; for example, 3 channels for RGB or YUV representation of an image or video, or 1 channel for grayscale image or video representation.

[0096] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data decoding in an unsupervised manner. (Diagram shown below) Figure 2As shown. The purpose of an autoencoder is to learn a representation (encoding) of a set of data by training the network to ignore signal "noise," often used for dimensionality reduction. Along with the simplification side, a reconstruction side is learned, where the autoencoder attempts to generate a representation from the simplified encoding that is as close as possible to its original input, hence the name.

[0097] In the simplest case, given a hidden layer, the encoder level of the autoencoder takes the input. and map it to :

[0098] This image This is commonly referred to as code, latent variable, or latent representation. Here, It is an element-wise activation function, such as the sigmoid function or the modified linear unit. It is a weight matrix. This is the bias vector. Weights and biases are typically initialized randomly and then iteratively updated during training via backpropagation. Afterwards, the decoder level of the autoencoder will... Mapping to Reconstruction of the same shape :

[0099] Among them, the decoder , and It can be corresponding to the encoder. , and Irrelevant.

[0100] Variational autoencoder models make strong assumptions about the distribution of latent variables. These models use variational methods for latent representation learning, generating additional loss components and a specific estimator for the training algorithm, called a Stochastic Gradient Variational Bayes (SGVB) estimator. It is assumed that the data is generated by a directed graphical model. The generator is producing the posterior distribution, and the encoder is learning the posterior distribution. approximation ,in, and These represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE is typically closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0101] Here, represent Kullback–Leibler divergence. The prior of latent variables is typically set as a centered isotropic multivariate Gaussian. Typically, the shapes of the variational and likelihood distributions are chosen such that they are factorized Gaussians:

[0102]

[0103] in, and It is the encoder output, and and This is the decoder output.

[0104] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression has been proposed using networks based on variational autoencoders.

[0105] Therefore, data compression is considered a fundamental and well-studied problem in engineering, typically involving designing codes with minimum entropy for a given discrete dataset. This approach relies heavily on understanding the probabilistic structure of the data, thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0106] In this context, the lossy compression problem requires a trade-off between two conflicting costs: the entropy (rate) of the discretized representation and the error (distortion) caused by quantization. Different compression applications, such as data storage or transmission over channels with limited capacity, require different balances between rate and distortion.

[0107] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable. Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, independently quantizing its elements, and then using the discrete representation encoded with lossless entropy codes. Due to the central role of the transform, this scheme is called transform coding.

[0108] For example, JPEG uses Discrete Cosine Transform (DCT) on pixel blocks, while JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform coding method (transform, quantization, and entropy coding) are optimized separately (usually by manually adjusting parameters). Modern video compression standards, such as HEVC, VVC, and EVC, also use transform representations to decode the predicted residual signal. Several transforms are used for this purpose, such as the Discrete Cosine Transform (DCT) and the Discrete Sine Transform (DST), as well as the manually optimized low-frequency non-separable manually optimized transform (LFNST).

[0109] Variational image compression The Variational Autoencoder (VAE) framework can be considered a nonlinear transform coding model. The transform process can be mainly divided into four parts. This is illustrated in the diagram of the VAE framework. Figure 3A Examples were provided in the text.

[0110] The transformation process can be mainly divided into four parts: Figure 3A An example illustrates the VAE framework. In Figure 3A In the middle, encoder 101 uses the function Input image Mapping to latent representation (by This latent representation is shown in the text. In the following text, this latent representation may also be referred to as a portion of the "latent space" or points within it. Function It is a transformation function that transforms the input signal Converted into a representation that can be further compressed. Quantizer 102 will represent the latent representation Transformation into a quantized latent representation Where the (discrete) value is , Represents the quantizer function. The entropy model, or superencoder / decoder (also known as superprior), estimates the quantized latent representation. The distribution is determined to obtain the minimum rate achievable by lossless entropy source decoding.

[0111] The latent space can be understood as a compressed representation of data, where similar data points are closer together. The latent space is extremely useful for learning data features and finding simpler representations of data for analysis. Quantifying latent representations... And edge information of super-prior 3 Arithmetic coding (AE) is used and included in bitstream 2 (which is binarized). Additionally, a decoder 104 is provided, which transforms the quantized latent representation into the reconstructed image. , .Signal It is the input image The estimate. Hope. as close as possible That is, the reconstruction quality should be as high as possible. However, and The higher the similarity between them, the greater the amount of side information required for transmission. Side information includes... Figure 3A The bitstreams 1 and 2 shown are generated by the encoder and transmitted to the decoder. Generally, the greater the amount of side information, the higher the reconstruction quality. However, a large amount of side information means a low compression ratio. Therefore, Figure 3A One objective of the system described in the text is to balance reconstruction quality and the amount of side information transmitted in the bitstream.

[0112] exist Figure 3A In this module, component AE 105 is the arithmetic coding module, which quantizes the latent representation. and edge information The samples are converted into a binary representation of the bitstream 1. For example, and Examples can include integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert sample values ​​into binary numeric strings (through a binarization process) (then the binary numbers are included in the bitstream, which may include other parts corresponding to the encoded image or other side information).

[0113] Arithmetic decoding (AD) 106 is the process of recovering the binarization process, in which the binarized numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0114] It should be noted that this invention is not limited to this specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0115] exist Figure 3A In this context, there are two cascaded subnets. A subnet is a logical division between different parts of the entire network. For example, in... Figure 3A In this context, modules 101, 102, 104, 105, and 106 are referred to as the "encoder / decoder" subnet. The "encoder / decoder" subnet is responsible for encoding (generating) and decoding (parsing) the first bitstream "bitstream1". Figure 3AThe second network, comprising modules 103, 108, 109, 110, and 107, is referred to as the "super encoder / decoder" subnet. This second subnet is responsible for generating the second bitstream, "bitstream2." These two subnets serve different purposes.

[0116] The first subnet is responsible for: Input image Transform 101 to its latent representation (This makes compression easier) ), potential representation Quantization 102 is a quantization latent representation , Arithmetic coding module 105 uses AE compressed quantization of latent representation In order to obtain the bitstream "bitstream1".

[0117] Arithmetic decoding module 106 parses bitstream 1 via AD. Reconstruct the 104 reconstructed image using the parsed data. ).

[0118] The purpose of the second subnet is to obtain the statistical properties of the "stream 1" samples (e.g., the mean, variance, and correlation among the samples of "stream 1"), making the compression of "stream 1" by the first subnet more efficient. The second subnet generates a second "stream 2", which includes the aforementioned information (e.g., the mean, variance, and correlation among the samples of "stream 1").

[0119] The second network includes an encoding part, which includes quantizing the latent representation. Transform 103 into edge information , edge information Quantized into quantized side information and the quantized side information Encode 109 (e.g., binarize) into bitstream 2. In this example, binarization is performed by arithmetic encoding (AE). The decoding part of the second network includes arithmetic decoding (AD) 110, which transforms the input bitstream 2 into decoded quantized side information. . Possibly with The same applies because arithmetic encoding and decoding are lossless compression methods. Then, the decoded quantized side information... Transformed 107 into decoded side information . express Statistical properties (e.g., (The mean of the sample, or the variance of the sample values, etc.). Then, the decoded latent representation Provided to the aforementioned arithmetic encoder 105 and arithmetic decoder 106 for control The probability model.

[0120] Figure 3A An example of a variational autoencoder (VAE) is described, the details of which may differ in different implementations. For example, in a particular implementation, additional components may exist to more efficiently obtain the statistical properties of samples from bitstream 1. In this implementation, a context modeler may be present, the purpose of which is to extract relevant information from bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and the arithmetic decoder (AD) 106 components.

[0121] Figure 3A The encoder and decoder are depicted in a single figure. Those skilled in the art will understand that the encoder and decoder can and often are embedded in different devices.

[0122] Figure 3B The encoder is described. Figure 3C The decoder component of the VAE framework is described. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as a color channel or other types of channels, such as a depth channel or motion information channel. The encoder's output (e.g., ...) Figure 3B (As shown) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the first subnet of the encoder, and bitstream 2 is the output of the second subnet of the encoder.

[0123] Similarly, in Figure 3C In this process, two bitstreams (bitstream 1 and bitstream 2) are received as input, and a reconstructed (decoded) image is generated at the output. As mentioned above, a VAE can be divided into different logical units that perform different operations. This is in Figure 3B and Figure 3C For example, Figure 3B The components involved in encoding signals such as video are described, and encoding information is provided. This encoding information is then... Figure 3C The decoder and other components in the encoder receive the data for encoding. It should be noted that the functions of the encoder and decoder components, represented by the numbers 12x and 14x, correspond to those described above. Figure 3A The component mentioned in the text is represented by the number 10x.

[0124] Specifically, such as Figure 3BAs shown, the encoder includes encoder 121, which transforms the input x into a signal y, and then provides the signal y to quantizer 322. Quantizer 122 provides information to arithmetic encoding module 125 and super encoder 123. Super encoder 123 provides the aforementioned bitstream 2 to super decoder 147, which in turn provides information to arithmetic encoding module 105 (125).

[0125] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the outputs of signal encoding, and this output is then provided (transmitted) to the decoding process. Although unit 101 (121) is called an "encoder", it can also be used to... Figure 3B The complete subnet described in the code is called an "encoder". The encoding process typically refers to the unit (module) that transforms the input into an encoded (e.g., compressed) output. From Figure 3B As can be seen, unit 121 can actually be considered the core of the entire subnet because it performs the transformation of the input x to y, which is a compressed version of x. For example, compression in encoder 121 can be achieved by applying a neural network or any processing network that typically has one or more layers. In such a network, compression can be performed through cascaded processing that includes downsampling, which reduces the size of the input and / or the number of channels. Therefore, for example, the encoder can be referred to as a neural network (NN) based encoder, etc.

[0126] The remaining parts in the diagram (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all components that improve the efficiency of the encoding process or are responsible for converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. AE 125 can perform binarization together with the super encoder 123 and super decoder 127 used to configure AE 125, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to... Figure 3B The entire subnet in the code is called the "encoder".

[0127] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into binary numbers (bits). For example, in a VAE framework, an encoder performing a non-linear transformation maps the input image x to y, where the width and height of y are smaller than x. Because y has a smaller width and height, it is smaller in size, and the (size) dimension of the signal is reduced, making it easier to compress the signal y. It's important to note that, typically, the encoder doesn't necessarily need to reduce size in two (or usually all) dimensions. Instead, some exemplary implementations can provide an encoder that reduces size in only one (or usually a subset) dimension.

[0128] In the arXiv e-printed version of J. Balle, L. Valero Lapara, and Esimoncelli (2015), “Density Modeling of Images Using a Generalized Normalization Transformation,” presented at the 4th International Conference on Learning Representations (hereinafter referred to as “Balle”) in 2016, the authors proposed an end-to-end optimization framework for image compression models based on nonlinear transformations. The authors optimized for Mean Squared Error (MSE) but used a more flexible transformation constructed from linear convolutions and nonlinear concatenations. Specifically, they employed a generalized divisive normalization (GDN) coupled with a nonlinearity inspired by neuron models in biological visual systems and proven effective for Gaussianizing image density. This concatenated transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which facilitates the parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0129] Such an example of the VAE framework is as follows: Figure 4 As shown, it utilizes six downsampling layers, labeled 401 to 406. The network architecture includes a hyperprior model. (Left side) The image autoencoder architecture is shown on the right. This corresponds to an autoencoder that implements a priori information. Factorized prior models use the same architecture for analysis and synthetic transformation. and Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder takes the input image x and inputs it to... This generates a response y (latent representation) with spatially varying standard deviation. Encoding It includes multiple convolutional layers, with subsampling and generalized divisive normalization (GDN) as the activation function.

[0130] Feed the response to In summary The standard deviation distribution in the data is then quantized, compressed, and transmitted as side information. Then, the encoder uses quantization vectors. To estimate The spatial distribution of standard deviation is used to obtain the probability (or frequency) values ​​of arithmetic coding (AE) and is used to compress and transmit quantized image representations. (or potential representation). The decoder first recovers from the compressed signal. Then, use get This provides it with the correct probability estimate for successful recovery. Then, Feed To obtain a reconstructed image.

[0131] Layers that include downsampling are indicated by a down arrow in the layer description. Layer description " This means that the layer is a convolutional layer with N channels and a kernel size of 5×5. As mentioned above, This means performing a 2x downsampling in this layer. A 2x downsampling will cause one dimension of the input signal to be reduced by half at the output. Figure 4 middle, This indicates that the width and height of the input image have been reduced by a factor of 2. Due to the presence of 6 downsampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, then the output signal... The width and height are equal to w / 64 and h / 64 respectively. The modules represented by AE and AD are the arithmetic encoder and arithmetic decoder, respectively, and will refer to... Figures 3A to 3C Let's explain. Arithmetic encoders and decoders are specific implementations of entropy decoding. AE and AD can be replaced by other entropy decoding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation; this is a reversible process. Furthermore, the "Q" in the diagram corresponds to the information mentioned above regarding... Figure 4 The quantization operation is also mentioned and further explained in the "Quantization" section above. Furthermore, the quantization operation and the corresponding quantization unit that is part of component 413 or 415 do not necessarily exist and / or can be replaced by another unit.

[0132] exist Figure 4 The diagram also shows a decoder including upsampling layers 407 to 412. Another layer 420 is provided between upsampling layers 411 and 410 in the order of input processing; this input is implemented as a convolutional layer but does not upsample the received input. A corresponding convolutional layer 430 for the decoder is also shown. Such layers can be provided in a neural network to perform operations on the input that modify specific features without changing the input size. However, it is not necessary to provide such layers.

[0133] When viewed from the processing order of bitstream 2 through the decoder, the upsampling layers operate in reverse order, from upsampling layer 412 to upsampling layer 407. Each upsampling layer is shown here to provide an upsampling ratio of 2, by... This indicates that not all upsampling layers have the same upsampling ratio, and other upsampling ratios, such as 3, 4, 8, etc., can also be used. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since they can provide the opposite operation to the encoder on the input, the upsampling layers can apply deconvolution to the received input, increasing its size by a factor corresponding to the upsampling ratio. However, the present invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by copying nearest neighbor samples, etc.

[0134] In the first subnet, after some convolutional layers (401 to 403) at the encoding end, there is generalized divisive normalization (GDN), followed by inverse GDN (IGDN) at the decoding end. In the second subnet, the activation function applied is ReLU. It should be noted that this invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLU.

[0135] End-to-end image or video compression DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations, techniques not commonly used in traditional methods. However, directly applying these techniques to build end-to-end learning systems for video compression is not common. Firstly, learning how to generate and compress motion information tailored to video compression remains an open question. Video compression methods heavily rely on motion information to reduce temporal redundancy in video sequences.

[0136] A simple technical solution is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate the flow field as accurately as possible. Accurate optical flow is often not the optimal choice for a specific video task. Furthermore, compared to motion information in traditional compression systems, optical flow data is significantly larger, and directly applying existing compression methods to compress optical flow values ​​would significantly increase the number of bits required to store motion information. Secondly, it is unclear how to construct a DNN-based video compression system with the goal of minimizing residuals and rate distortion of motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) reconstructed frames given a certain number of bits (or bit rate) for compression. RDO is crucial for video compression performance. To leverage the power of end-to-end training of a learning-based compression system, an RDO strategy is needed to optimize the entire system.

[0137] In the proceedings of the 2019 IEEE / CVF Computer Vision and Pattern Recognition (CVPR) conference, titled "DVC: An End-to-end Deep Video Compression Framework" (pages 11006-11015), Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao proposed an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual decoding.

[0138] This type of encoder is like Figure 5A shown. Specifically, Figure 5A The overall structure of the end-to-end trainable video compression framework is shown. To compress motion information, a CNN is specified to transform the optical flow into a corresponding representation suitable for better compression. Specifically, an autoencoder-style network is used to compress the optical flow. The motion vector (MV) compression network is as follows: Figure 5B As shown. Network architecture and Figure 4The ga / gs approach is somewhat similar. Specifically, the optical flow is fed into a series of convolutional operations and nonlinear transformations, including GDN and IGDN. The output channel number of the convolution (deconvolution) is 128, but the last deconvolutional layer has 2 channels. Given an optical flow size of M×N×2, the MV encoder generates a motion representation of size M / 16×N / 16×128. This motion representation is then quantized, entropy-decoded, and sent to the bitstream. The MV decoder receives the quantized representation and reconstructs the motion information using the MV encoder.

[0139] Figure 5C The structure of the motion compensation section is shown. Here, previously reconstructed frames are used. Based on the reconstructed motion information, the warping unit generates warped frames (typically using an interpolation filter, such as a bilinear interpolation filter). Then, a separate CNN with three inputs generates the predicted image. The architecture of the motion-compensated CNN is also as follows... Figure 5C As shown.

[0140] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to transform the residual into a corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better utilize the power of nonlinear transformation to achieve higher compression efficiency.

[0141] As can be seen from the overview above, considering the different parts of the video framework, including motion estimation, motion compensation, and residual coding, CNN-based architectures can be applied to image and video compression. Entropy decoding is a popular method for data compression, widely adopted in the industry, and also suitable for feature map compression in human perception or computer vision tasks.

[0142] Machine video encoding Video Coding for Machines (VCM) is another popular area of ​​computer science today. The main idea behind this approach is to transmit coded representations of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video coding designed for human perception, VCM focuses on performance characteristics for computer vision tasks, such as object detection accuracy, rather than reconstruction quality.

[0143] A recent study proposes a novel deployment paradigm called collaborative intelligence, which partitions deep models between mobile and cloud environments. Extensive experiments across various hardware configurations and wireless connectivity modes have shown that the optimal operating point in terms of energy consumption and / or computational latency involves partitioning the model, typically at some point deep within the network. The current common approach, where the model resides entirely in the cloud or entirely on the mobile device, is rarely (if ever) optimal. The concept of collaborative intelligence is also extended to model training. In this context, data flows bidirectionally: from the cloud to the mobile device during backpropagation in training, and from the mobile device to the cloud during forward propagation in training, as well as during inference.

[0144] Against the backdrop of recent deep object detection models, lossy compression of deep feature data based on HEVC intra-frame coding was investigated. As the compression level increases, detection performance degrades. Compression-enhanced training was proposed to minimize this loss by generating a model more stable to quantization noise in the feature values. However, this remains a suboptimal approach because the codec used is highly complex and optimized for natural scene compression rather than deep feature compression.

[0145] This paper investigates the trade-off between compression efficiency and recognition accuracy using a method for object detection tasks leveraging the popular YOLOv2 network, addressing the problem of deep feature compression in collaborative intelligence. Here, the term "deep features" has the same meaning as "feature map." The term "deep" derives from the idea of ​​collaborative intelligence where the output feature maps of certain hidden (deep) layers are captured and transmitted to the cloud for inference. This appears to be more efficient than sending compressed natural image data to the cloud and performing object detection using reconstructed images.

[0146] Efficient feature map compression is beneficial for image and video compression and reconstruction, whether for human perception or machine vision. The drawbacks of state-of-the-art autoencoder-based compression methods also apply to machine vision tasks.

[0147] Artificial neural networks with skip connections A residual neural network (ResNet) is a type of artificial neural network (ANN) built upon the known structure of pyramidal cells in the cerebral cortex. ResNet achieves this by utilizing skip connections or shortcuts to skip certain layers. A typical ResNet model is implemented with two or three skip layers, which include non-linearity (ReLU) and intermediate batch normalization. Additional weight matrices can be used to learn the skip weights; these models are called HighwayNets. Models with multiple parallel skips are called DenseNets. In the context of residual neural networks, non-residual networks can be described as ordinary networks.

[0148] One purpose of skipping layers is to avoid the vanishing gradient problem by reusing the activations of the previous layer until adjacent layers learn their weights. During training, weight adjustments silence upstream layers and amplify previously skipped layers. In the simplest case, only the weights of connections between adjacent layers are adjusted, without adjusting the explicit weights of upstream layers. This works best when a single non-linear layer is skipped, or when the intermediate layers are all linear. If not, explicit weight matrices should be learned for the skipped connections (HighwayNet should be used).

[0149] This skipping simplifies the network, reducing the number of layers used in the initial training phase. This speeds up learning by reducing the effects of vanishing gradients, as fewer layers need to be propagated. The network then gradually recovers the skipped layers as it learns the feature space. Near the end of training, when all layers have been expanded, it will remain closer to the manifold, thus learning faster. A neural network without residual parts can explore more of the feature space. This makes it more susceptible to perturbations that cause it to deviate from the manifold and requires additional training data to recover.

[0150] U-Net introduces longer skip connections, such as Figure 6 As shown, the U-Net architecture originates from the "fully convolutional network" first proposed by Long and Shelhamer. The main idea is to supplement the typical shrinking network with successive layers in which pooling operations are replaced by upsampling operators. Therefore, these layers improve the resolution of the output. Furthermore, successive convolutional layers can learn an accurate output based on this set of information.

[0151] A key modification in U-Net is the inclusion of numerous feature channels in the upsampling portion, allowing the network to propagate contextual information to higher-resolution layers. Consequently, the expansion path is more or less symmetrical to the contraction path, resulting in a U-shaped architecture. The network uses only the effective portion of each convolution, without any fully connected layers. To predict pixels in image boundary regions, missing context is extrapolated by mirroring the input image. This tiling strategy is crucial for applying the network to large images, as resolution would be limited by GPU memory without it.

[0152] Introducing skip connections can better capture features at different spatial resolutions, and this has been successfully applied to computer vision tasks such as object detection and segmentation. However, such skip connections in image or video compression are not a trivial task, because information from the encoding end needs to be transmitted in the communication channel, and direct connections between layers would require the transmission of large amounts of data.

[0153] Traditional hybrid video codec Neural network frameworks can also be used in combination or in traditional hybrid encoding and decoding, as illustrated later. A very brief overview of exemplary hybrid encoding and decoding is given below.

[0154] Figure 7A This is a schematic block diagram of an exemplary video encoder 20 used to implement the technology of this application. Figure 7A In the example, the video encoder 20 includes an input terminal 201 (or input interface 201), a residual calculation unit 204, a transform processing unit 206, a quantization unit 208, an inverse quantization unit 210, an inverse transform processing unit 212, a reconstruction unit 214, a loop filter unit 220, a decoded picture buffer (DPB) 230, a mode selection unit 260, an entropy coding unit 270, and an output terminal 272 (or output interface 272). The mode selection unit 260 may include an inter-frame prediction unit 244, an intra-frame prediction unit 254, and a segmentation unit 262. The inter-frame prediction unit 244 may include a motion estimation unit and a motion compensation unit (not shown). Figure 7A The video encoder 20 shown can also be called a hybrid video encoder or a video encoder based on a hybrid video codec.

[0155] Encoder 20 can be used to receive image 17 (or image data 17) via input terminal 201, for example, an image in an image sequence forming a video or video sequence. The received image or image data can also be a pre-processed image 19 (or pre-processed image data 19). For simplicity, the following description uses image 17. Image 17 can also be referred to as the current image or the image to be decoded (especially in video decoding when distinguishing the current image from other images, such as those in the same video sequence, i.e., including previously encoded and / or decoded images in the video sequence of the current image).

[0156] A (digital) image is, or can be, a two-dimensional array or matrix of samples with intensity values. Samples in an array can also be called pixels (a short form of image element). The number of samples in the horizontal and vertical directions (or axes) of the array or image defines the image size and / or resolution. To represent color, three color components are typically used; that is, the image can be represented as or comprise three sample arrays. In RGB format or color space, an image comprises corresponding red, green, and blue sample arrays. However, in video decoding, each pixel is typically represented in a luminance and chrominance format or in a color space, for example, YCbCr, comprising a luminance component represented by Y (sometimes also L) and two chrominance components represented by Cb and Cr. The luminance component Y represents luminance or grayscale intensity (e.g., as in a grayscale image), and the two chrominance components Cb and Cr represent chrominance or color information components. Accordingly, an image in YCbCr format comprises a luminance sample array of luminance sample values ​​(Y) and two chrominance sample arrays of chrominance values ​​(Cb and Cr). An RGB format image can be converted to YCbCr format and vice versa; this process is also known as color transformation or conversion. If the image is black and white, it can consist only of a luminance sample array. Accordingly, for example, an image can be a black and white luminance sample array or a luminance sample array and two corresponding chrominance sample arrays in 4:2:0, 4:2:2, and 4:4:4 color formats.

[0157] Embodiments of the video encoder 20 may include an image segmentation unit ( Figure 7A (Not shown in the image) is used to segment image 17 into multiple (typically non-overlapping) image blocks 203. These blocks may also be called root blocks, macroblocks (H.264 / AVC), coding tree blocks (CTBs), or coding tree units (CTUs) (H.265 / HEVC and VVC). Image segmentation units can be used to segment each image into corresponding blocks by applying the same block size and a corresponding grid with defined block sizes to all images in a video sequence, or by varying the block size between images, subsets of images, or groups of images. The abbreviation AVC stands for Advanced Video Coding.

[0158] In another embodiment, the video encoder may be used to directly receive blocks 203 of image 17, such as one, several, or all of the blocks that make up image 17. Image block 203 may also be referred to as the current image block or the image block to be decoded.

[0159] Similar to image 17, image block 203 is also a two-dimensional array or matrix composed of samples with intensity values ​​(sample values), but the size of image block 203 is smaller than that of image 17. That is, for example, block 203 may include, for example, a single sample array (e.g., a luminance array in the case of black and white image 17, or a luminance or chrominance array in the case of a color image) or three sample arrays (e.g., a luminance array and two chrominance arrays in the case of color image 17) or any other number and / or type of array depending on the applied color format. The number of samples in the horizontal and vertical directions (or axes) of block 203 defines the size of block 203. Therefore, a block can be an M×N (M columns × N rows) sample array, or an M×N transform coefficient array, etc.

[0160] Figure 7A The embodiment of the video encoder 20 shown can be used to encode the image 17 block by block, for example, to perform encoding and prediction for each block 203.

[0161] Figure 7A The embodiment of the video encoder 20 shown can also be used to segment and / or encode images using strips (also known as video strips), wherein one or more strips (typically non-overlapping) can be used to segment or encode images, and each strip may include one or more blocks (e.g., CTUs).

[0162] Figure 7A The embodiment of the video encoder 20 shown can also be used to segment and / or encode an image using chunk groups (also known as video chunk groups) and / or chunks (also known as video chunks), wherein one or more chunk groups (typically non-overlapping) can be used to segment or encode an image, each chunk group may include one or more blocks (e.g., CTUs) or one or more chunks, wherein each chunk may be a shape such as a rectangle and may include one or more blocks (e.g., CTUs), such as complete or partial blocks.

[0163] Figure 7B An example of a video decoder 30 for implementing the technology of this application is shown. The video decoder 30 is used to receive, for example, encoded image data 21 (e.g., encoded bitstream 21) encoded by encoder 20, to obtain a decoded image 331. The encoded image data or bitstream includes information for decoding the encoded image data, such as data representing image blocks (and / or chunks or blocks) of encoded video strips (and / or chunks) and associated syntax elements.

[0164] The entropy decoding unit 304 is used to parse the bitstream 21 (or typically encoded image data 21) and, for example, to perform entropy decoding on the encoded image data 21 to obtain quantization coefficients 309 and / or decoded decoding parameters (not shown in FIG. 3), such as any one or all of inter-frame prediction parameters (e.g., reference image index and motion vector), intra-frame prediction parameters (e.g., intra-frame prediction mode or index), transform parameters, quantization parameters, loop filter parameters, and / or other syntax elements. The entropy decoding unit 304 can be used to apply a decoding algorithm or scheme corresponding to the encoding scheme described for the entropy encoding unit 270 of the encoder 20. The entropy decoding unit 304 can also be used to provide inter-frame prediction parameters, intra-frame prediction parameters, and / or other syntax elements to the mode application unit 360, and to provide other parameters to other units of the decoder 30. The video decoder 30 can receive video strip-level and / or video block-level syntax elements. Alternatively, or as an alternative to stripes and corresponding syntax elements, it can receive or use chunk groups and / or chunks and corresponding syntax elements.

[0165] Reconstruction unit 314 (e.g., adder or summer 314) can be used to add reconstruction residual block 313 to prediction block 365 to obtain reconstruction block 315 of the sample domain, for example, by adding the sample values ​​of reconstruction residual block 313 and the sample values ​​of prediction block 365.

[0166] Figure 7B The embodiment of the video decoder 30 shown can be used to segment and / or decode an image using strips (also known as video strips), wherein one or more strips (typically non-overlapping) can be used to segment or decode an image, and each strip may include one or more blocks (e.g., CTUs).

[0167] Figure 7B The embodiment of the video decoder 30 shown can be used to segment and / or decode an image using chunk groups (also known as video chunk groups) and / or chunks (also known as video chunks), wherein one or more chunk groups (typically non-overlapping) can be used to segment or decode an image, each chunk group may include one or more blocks (e.g., CTUs) or one or more chunks, wherein each chunk may be a shape such as a rectangle and may include one or more blocks (e.g., CTUs), such as complete or partial chunks.

[0168] Other variations of the video decoder 30 can be used to decode the encoded image data 21. For example, the decoder 30 can generate an output video stream without the loop filter unit 320. For example, the non-transform-based decoder 30 can directly dequantize the residual signals of certain blocks or frames without the inverse transform processing unit 312. In another implementation, the dequantization unit 310 and the inverse transform processing unit 312 can be combined into a single unit in the video decoder 30.

[0169] It should be understood that in encoder 20 and decoder 30, the processing result of the current step can be further processed and then output to the next step. For example, after interpolation filtering, motion vector derivation, or loop filtering, the processing result of interpolation filtering, motion vector derivation, or loop filtering can be further calculated, such as clipping or shifting.

[0170] Improve decoding efficiency As mentioned above, image and video compression methods based on variational autoencoders lack the space for adaptive processing and object segmentation to capture realistic object boundaries. Therefore, content adaptability is limited. Furthermore, for certain types of video information, such as motion or residual information, sparse representation and decoding are ideal to keep indication overhead at a reasonable level.

[0171] Therefore, some embodiments of the present invention introduce segmentation information decoding and feature map decoding from different spatial resolution layers of the autoencoder to achieve content adaptation and sparse signal representation and transmission.

[0172] In some exemplary implementations, connections are introduced between encoder and decoder layers other than the lower-resolution layers (latent space), which are transmitted in the bitstream. In some exemplary implementations, only a portion of the feature maps from different resolution layers is provided in the bitstream to save bandwidth. For example, signal selection and signal feeding logic is introduced to select, transmit, and use portions of the feature maps from different resolution layers. At the receiver, tensor combination logic is introduced to combine the output of the previous resolution layer with information received from the bitstream corresponding to the current resolution layer.

[0173] Below, some detailed implementations and examples related to the encoding and decoding ends are provided.

[0174] Encoding methods and devices According to one embodiment, a method is provided for encoding data for image or video processing into a bitstream. The method includes the steps of: processing the data, including generating feature maps in multiple cascaded layers, each feature map including its own resolution, wherein at least two of the generated feature maps have different resolutions.

[0175] That is, the resolutions of two or more cascaded layers can be different from each other. Here, the layer resolution refers to the resolution of the feature map processed by that layer. In an exemplary implementation, it is the resolution of the feature map output by the layer. A feature map including resolution means that at least a portion of the feature map has said resolution. In some implementations, the entire feature map can have the same resolution. For example, the resolution of a feature map can be given by multiple feature map elements in the feature map. However, it can also be more specifically defined by the number of feature map elements in one or more dimensions (e.g., x, y; or, in addition, the number of channels can be considered).

[0176] The term "layer" here refers to a processing layer. It doesn't necessarily have to be a layer with trainable or trained parameters (weights), like the layers in some neural networks mentioned above. Instead, a layer can represent a specific process of inputting data into a layer to produce its output. In some embodiments, a layer can be trained or trainable. Here, "training" refers to machine learning or deep learning.

[0177] When cascaded layers are mentioned, it refers to layers having a certain predefined order (sequence), and the input to the first layer (in the given order) being processed sequentially by the first layer and the other layers according to the given order. That is, the output of the j-th layer is the input of the (j+1)-th layer, where j is an integer from 1 to the total number of cascaded layers. In a specific and non-limiting example, for all possible values ​​of j, the (j+1)-th layer includes (or has) the same or lower resolution as the j-th layer. That is, the resolution of the layers does not increase with the order of cascading (processing) (e.g., at the encoding end). However, it should be noted that the present invention is not limited to this specific type of cascaded layer. In some embodiments, the layers in the cascaded processing may also include layers that increase the resolution. In any case, there may be layers that do not change the resolution.

[0178] For example, a lower resolution feature map may mean fewer feature elements per feature map. Conversely, a higher resolution feature map may mean more feature elements per feature map.

[0179] The method further includes the steps of: selecting a layer from a plurality of layers that is different from the layer that generates the lowest resolution feature map, and generating a bitstream includes inserting information related to the selected layer into the bitstream.

[0180] That is, in addition to (or alternatively) outputting the processing results of all layers in the cascade to the bitstream, information is also provided to another (selected) layer. There can be one or more selected layers. The information associated with the selected layer can be of any type, such as the layer's output or some segmentation information of the layer (described below), or other information also related to the feature map processed by the layer and / or the processing performed by the layer. That is, in some examples, the information can be elements of the feature map and / or the positions of elements within the feature map (within the layer).

[0181] The input to the cascaded processing is data used for image or video processing. For example, this data may be related to predictive decoding such as inter-frame or intra-frame prediction. It can be motion vectors or other parameters of the prediction, such as the prediction mode or reference image or orientation, or other decoding components besides prediction, such as transforms, filtering, entropy decoding, or quantization. Bitstream generation can include any transformation that converts values ​​into bits including fixed codewords, variable-length codes, or arithmetic codes (binarization).

[0182] Here, an image can be a still image or a video image. An image refers to one or more samples, such as samples captured by a camera or generated by computer graphics. An image may include samples representing brightness levels in grayscale, or may have multiple channels including one or more of a luminance channel, a chroma channel, a depth channel, or other channels. Image or video encoding can be any of the hybrid decoding methods described above (e.g., similar to HEVC or VVC) or automatic encoders.

[0183] Figure 8 The flowchart above illustrates the method. Therefore, the method includes step 810, which processes the input data. In the selection step 820, a portion is selected from the processed data, and in the generation step 830, this data portion is included in the bitstream. Not all data generated in the processing steps must be included in the bitstream.

[0184] According to an exemplary implementation, the process further includes downsampling by one or more cascaded layers. Figure 9 An exemplary network 900 is shown that implements (performs in operation) such processing.

[0185] Specifically, Figure 9 The diagram shows input data entering network 900 for image or video processing 901. The input data for image or video processing can be any type of data used for such processing, such as samples of images or videos directly as described above, prediction patterns, motion vectors, etc. Figure 9The processing applied to input 901 is performed by multiple processing layers 911 to 913, each reducing the resolution of each motion vector array. That is, the cascaded layers 911 to 913 are downsampling layers. It should be noted that when a layer is called a downsampling layer, it performs downsampling. There are embodiments where downsampling layers 911 to 913 perform downsampling as their sole task; and there may be embodiments where downsampling layers 911 to 913 do not perform downsampling as their sole task. Conversely, downsampling layers can typically also perform other types of processing.

[0186] As in Figure 9 As can be seen, downsampling layers 911 to 913, in addition to processing data inputs and outputs, also have additional selection outputs that cause signal selection logic 920. It should be noted that the term "logic" here refers to any circuitry that implements this function (in this case, signal selection). Signal selection logic 920 selects information from the selection outputs of any layer to include in the bitstream 930. Figure 9 In the example, each layer 911 to 913 downsamples the layer input. However, layers that do not apply downsampling can also be added between downsampled layers. For example, these layers can process the input through filtering or other operations.

[0187] exist Figure 9 In the example shown, signal selection logic 920 selects information to be included in the bitstream from the outputs of layers 911 to 913. The goal of this selection could be to choose information relevant to the reconstructed image or video from multiple feature maps output from different layers. That is, the downsampling layer and signal selection logic can be implemented as part of the encoder (image or video encoder). For example, the encoder could be... Figure 3A The encoder 101 shown Figure 3B Encoder 121, MV Encoder Network ( Figure 5A Part of mid-to-end compression Figure 5B MV encoder, or Figure 7A Some parts of the encoder (e.g., loop filter 220, mode selection unit 260, or part of prediction unit 244, 254), etc.

[0188] Figure 9 It also includes a decoding section (which may be referred to as the extended path), comprising signal feed logic 940 and upsampling layers 951 to 953. The input to the encoding section is the bitstream 930. For example, output 911 is the reconstruction input 901. The decoding section is described later below.

[0189] For example, downsampling can be accomplished through max pooling, average pooling, or any other operation that results in downsampling. Another example of such an operation is convolution. Figure 10AAn example of max pooling is shown. In this example, every four elements (adjacent 2×2 squares) of array 1010 are grouped and used to determine one element in array 1020. Arrays 1020 and 1010 may correspond to feature maps in some embodiments of the invention. However, the arrays may also correspond to portions of the feature map in this embodiment. Fields (elements) in arrays 1020 and 1010 may correspond to elements of the feature map. In this image, feature map 1020 is determined by downsampling feature map 1010. The numbers in the fields of arrays 1010 and 1020 are merely exemplary. For example, fields may also include motion vectors instead of numbers. Figure 10A In the max-pooling example shown, the four fields in the top-left corner of array 1010 are grouped, and the maximum value among them is selected. This group of values ​​determines the top-left field of array 1020 by assigning the maximum value to the field. That is, the largest of the four top-left values ​​of array 1010 is inserted into the top-left field of array 1020.

[0190] Alternatively, in some implementations, min-pooling can be used. Min-pooling selects the field with the smallest value instead of the field with the largest value. However, these downsampling techniques are just examples, and various downsampling strategies can be used in different implementations. Some implementations may use different downsampling techniques in different layers, different regions within the feature map, and / or for different types of input data.

[0191] In some implementations, downsampling is performed using average pooling. In average pooling, the average of a set of feature map elements is calculated and associated with the corresponding field in the feature map of the downsampled feature map.

[0192] Figure 10B An example of average pooling is shown. In this example, the top-left element of feature map 1050 is averaged, and the top-left element of feature map 1060 is also averaged. Figure 10B The three groups in the upper right, lower right, and lower left of the image show the same situation.

[0193] In another embodiment, convolution operations are used for downsampling in some or all layers. In convolution, a filter kernel is applied to groups or blocks of elements in the input feature map. The kernel itself can be an array of elements with the same size as the input element block, where each element of the kernel stores the weights of the filter operation. In downsampling, the sum of the elements in the input block is computed, with each element weighted by the corresponding value taken from the kernel. If the weights of all elements in the kernel are fixed, such a convolution can correspond to the filter operation described above. For example, a convolution with kernels of the same, fixed weights and kernel size stride corresponds to an average pooling operation. However, the stride of the convolution used in this embodiment can be different from the kernel size, and the weights can be different. In one example, the kernel weights can be such that certain features in the input feature map can be enhanced or distinguished from each feature. Furthermore, the kernel weights can be learnable or learned in advance.

[0194] According to one embodiment, the information associated with the selected layer includes element 1120 of the feature map of that layer. For example, this information may convey feature map information. Typically, the feature map may include any features associated with the moving image.

[0195] Figure 11 Feature map 1110 is shown as an exemplary implementation of dense optical flow of motion vectors with width W and height H. Motion segmentation mesh 1140 includes three downsampling layers (e.g., corresponding to...). Figure 9 The downsampling layers 911 to 913 and the signal selection circuit (logic) 1100 (e.g., corresponding to signal selection logic 920) are included. Figure 11 Examples of the outputs (L1 to L3) of different layers in the right-hand contraction path are shown.

[0196] In this example, the output of each layer (L1 to L3) is a feature map with progressively decreasing resolution. The input to L1 is a dense optical flow 1110. In this example, one element of the feature map output from L1 is determined from sixteen (4×4) elements of the dense optical flow 1110. Each square in the L1 output ( Figure 11 The lower right corner corresponds to the motion vector obtained by downsampling (downspl4) from the 16 motion vectors of the dense optical flow. Such downsampling can be, for example, average pooling or another operation, as described above. In this exemplary implementation, information 1120 includes only a portion of the feature map L1 of that layer. Layer L1 is selected, and the portion corresponding to the four motion vectors (feature map elements) associated with the selected layer is sent within the selected information 1120.

[0197] The output L1 of the first layer is then fed into the second layer (downspl2). The output L2 feature map element of the second layer is determined from the four elements of L1. However, in other examples, each element of the feature map with the lower resolution can also be determined by a group of any other number of elements that determine the next higher resolution feature map. For example, the number of elements in the group determining one element in the next layer can also be any power of 2. In this example, the output L2 feature map corresponds to three motion vectors, which are also included in the selected information 1120, so the second layer is also a selected layer. The third layer (downspl2) downsamples the output L2 of the second layer by 2 in each of the two dimensions. Accordingly, one feature map element of the third layer output L3 is obtained from the four elements of L2. No elements are sent in the feature map L3, i.e., in this example, the third layer is not a selected layer.

[0198] The signal selection module 1100 of the motion segmentation network 1140 selects the aforementioned motion vectors (elements from the feature maps output by the first and second layers) and provides them to the bitstream 1150. This specification may be a simple binarization, and may, but does not necessarily, include entropy decoding.

[0199] Element groups can be arranged in a square shape, such as in Figure 11 The examples are shown below. However, these groups can also be arranged in any other shape, such as a rectangle, where the long side of the rectangle can be arranged horizontally or vertically. These shapes are just examples. In one implementation, any shape can be used. The shape can also be indicated in the bitstream 1150. The indication can be implemented by a marker diagram indicating which feature elements belong to the shape and which do not. Alternatively, the indication can be done using a more abstract description of the shape.

[0200] In this exemplary implementation, feature map elements are grouped such that each element belongs to exactly one element group that determines an element of the next layer's feature map. That is, the feature map element groups are non-overlapping, with only one group contributing to the feature map elements of higher (later in the cascaded processing order) layers. However, it is conceivable that an element from one layer can be used for multiple elements of the next layer. Specifically, in processing 810, when generating a new layer output, such as layer output L2, based on the layer output L1 with higher resolution, filter operations can be used.

[0201] In this embodiment, selection 820 (e.g., by means of signal selection 1100) selects elements to be included in the bitstream from multiple output feature maps (L1 to L3). This selection minimizes the amount of data required to transmit the selected data while maximizing the amount of information relevant to decoding. For example, rate distortion optimization or other optimizations may be employed.

[0202] The example above illustrates a three-layer processing method. However, this method is not typically limited to this. Any number (or more) of processing layers can be used. That is, according to a more general example, the method includes acquiring the data to be encoded. This could be a dense stream of motion vectors 1110, as shown above. However, the invention is not limited to this, and alternatively, or in addition to motion vectors, other data can be processed, such as prediction patterns, prediction directions, filtering parameters, or even spatial image information (samples) or depth information.

[0203] The processing 810 of the data to be encoded includes processing by each j-th layer in this example of multiple (N) cascaded layers. The processing of the j-th layer includes: If j=1, the data to be encoded is obtained as the layer input; otherwise, the feature map processed by the (j-1)th layer is obtained as the layer input (i.e., if the i-th layer is the current processing layer, then the (j-1)th layer is the previous layer). The acquired layer input is processed, and the processing includes downsampling; Output the downsampled feature map.

[0204] In this example, the layer j=1 is the highest resolution layer among the N processing layers. It should be noted that the input to this layer can be dense optical flow (generally, it can also be considered a feature map). Therefore, in some specific embodiments, the layer j=1 can be the input layer. However, this is not always the case, as it is conceivable that there are some preprocessing layers before the N processing layers. A typical characteristic of encoders is that earlier processing layers have higher resolution than later processing layers (contraction path). The corresponding process is the opposite at the decoding end. It is conceivable that some processing layers do not change the resolution, or even enhance the resolution—this invention can still be applied.

[0205] In the example above, bitstream 1150 carries selected information 1120. For example, this information could be a motion vector or any other feature. That is, bitstream 1150 carries feature map elements from at least one output layer that is not a processing network (encoding-side processing network). Figure 11 In the example, only a portion of the selected feature map is transmitted in the bitstream. This portion has one or more feature elements. To enable the decoder to determine which portion of the feature map to transmit, rules can be defined for this determination. In some embodiments, segmentation information can be transmitted in bitstream 1150 to configure which portions of the feature map are transmitted. These exemplary embodiments will be described below. However, it should be noted that the above embodiments are merely exemplary and such additional signaling is generally not required, as rules may exist for deriving information and depending on other known or indicative parameters.

[0206] In an exemplary embodiment related to segmentation information, the information related to the selected layer includes (in addition or alternatively, selection information 1120) information 1130 indicating from which layer and / or from which part of the feature map of the layer the elements are selected.

[0207] exist Figure 11 In the example shown, segmentation information is indicated by binarization flags. For example, on the right, each lower-resolution feature map or feature map portion is assigned a value of 0 or 1. For example, L3 is assigned a value of zero (0) because it is not selected and there is no indication of motion vector (feature element) signal for L3. Feature map L2 has four portions. Layer processing L2 is the selected layer. Feature map elements (motion vectors) for three of the four portions are indicated, and the flag is set to 1 accordingly. The remaining portion of feature map L2 does not contain motion vectors, so the flag is set to 0 because the motion vector corresponding to this portion is represented by feature map L1. Since L1 is the first layer, the remaining motion vectors are implicitly provided in this layer. It should be noted that the binarization flag here takes a first value (e.g., 1) when the corresponding feature map portion is partially selected information, and a second value (e.g., 0) when the corresponding feature map portion is not partially selected information. Since it is a binarization flag, it can only accept one of these two values.

[0208] This segmentation information can be provided in the bitstream. Figure 11 The left side shows the processing of segmentation information 1130. It should be noted that segmentation information 1130 can also be processed by layers of the motion segmentation network 1140. It can be processed in the same layer as the feature map, or in a separate layer. Segmentation information 1130 can also be interpreted as follows: A superpixel of the layer with the lowest resolution covers 16×16 cells of the feature map obtained by downsampling downspl4 of the dense optical flow 1110. Since the flag assigned to the superpixel covering the 16×16 cells is set to 0, this means that the feature map elements—here, motion vectors—are not indicated for this layer (this layer is not selected). Therefore, feature map elements can be indicated in a region corresponding to the next layer of 16×16 cells, represented by four equal-sized superpixels, each covering 8×8 feature element cells. Each of these four superpixels is associated with a flag. For those superpixels associated with a flag having a value of 1, the feature map elements (motion vectors) are indicated. For superpixels with a flag set to 0, no motion vector signal is indicated. Unindicated motion vectors are layer indicators for superpixels covering 4×4 element units.

[0209] More broadly, the method for encoding data for image / video decoding may also include selection (segmentation) information to be inserted into the bitstream. This information relates to a first region (superpixel) in a feature map processed by layers j > 1. The first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j that includes multiple elements. The method further includes the step of: when selecting feature maps processed by k layers, not selecting the region corresponding to the first region, where k is an integer equal to or greater than 1 and k < j. The correspondence between regions of different layers here refers to the fact that the corresponding regions (superpixels) spatially cover the same feature elements (initial data elements) in the feature map (initial data) to be encoded. Figure 11 In the example, the initial data to be segmented is L1 data. However, this correspondence can also be referenced to dense optical flow 1110.

[0210] pass Figure 11 A specific arrangement in the layer ensures that each feature element of the initial feature map (e.g., L1) is included in only one superpixel of N layers. This arrangement offers the advantage of efficient decoding of feature maps and segmentation information. The cascaded layer processing framework corresponds to a neural network processing framework, which can segment data in this way and provide data for each segment at different resolutions. Specifically, the advantage of downsampling in some layers can include reducing the amount of data required to indicate the initial feature map representation. Specifically, in the example of indicating motion vectors, groups of similar motion vectors can be indicated by a common motion vector due to downsampling. However, for good inter-frame prediction, the prediction error caused by grouped motion vectors should be small. This may mean that different grouping levels of motion vectors for different regions of the image may be the optimal level for achieving the desired prediction quality, while requiring a small amount of data to indicate the motion vector signal. This can be achieved using multiple layers with different resolutions.

[0211] In embodiments where the feature map elements are motion vectors, for downsampling purposes, the length and direction of the motion vectors can be averaged, and the averaged motion vector is associated with the corresponding feature map element of the downsampled feature map. In a typical averaging, all elements in a group corresponding to an element in the downsampled feature map have the same weight. This corresponds to applying filters with equal weights to a group or block of elements to compute the downsampled feature map element. However, in other implementations, such filters can have different weights for different elements in the layer input. In other implementations, instead of calculating the average of the group or block of elements in the downsampled feature map, the median of the corresponding group of elements can be computed.

[0212] exist Figure 11In the example, the downsampling filter operation uses a square filter with an input element size of 2×2 and computes a filter output that maps to an element in the downsampled feature map, depending on the selected filter operation. The filter operation uses a step size equal to the edge length or 2 of the square filter. That is, between two filtering operations, the filter will move by a step size equal to the filter size. Therefore, in downsampling, the downsampled element is computed based on the non-overlapping blocks in the layer to which the downsampling filter is applied.

[0213] However, in some further conceivable embodiments, the step size can differ from the edge length of the filter. For example, the step size might be smaller than the length of the filter edge. Therefore, the filter blocks used to determine elements in the downsampling layer can overlap; that is, one element in the feature map to be downsampled helps in calculating multiple elements in the downsampled feature map.

[0214] Typically, data associated with a selected layer includes indications of the location of feature map elements within the feature map of that selected layer. Here, similar to... Figure 11 The concept of (feature maps L1 to L3) refers to the feature map of a selected layer, which is the output from the selected layer, i.e. the feature map processed by the selected layer.

[0215] For example, the positions of selected and non-selected feature map elements are indicated by multiple binarized flags, which are based on the positions of these flags in the bitstream. (See above reference...) Figure 11 In the description, the binarized flags are included in the bitstream 1150 as segmentation information 1130. To enable the decoder to parse and correctly interpret the segmentation information, the allocation between flags and layers and / or regions in the feature maps processed by the layers should be defined. This can be accomplished by defining a flag binarization order known to both the encoder and decoder.

[0216] The above examples have provided data for encoding images / videos as motion vectors. However, the invention is not limited to this embodiment. In one embodiment, the data to be encoded includes image information and / or prediction residual information and / or prediction information. Here, image information refers to sample values ​​of the original image (or the image to be encoded). Sample values ​​can be samples of one or more colors or other channels.

[0217] The information associated with a selected layer is not necessarily a motion vector or a superpixel's motion vector. Additionally or alternatively, in some embodiments, the information includes prediction information. Prediction information may include a reference index and / or a prediction mode. For example, a reference index may indicate which specific image from a set of reference images is applied to inter-frame prediction. This index may be relative to the current image containing the current block to be predicted. For example, a prediction mode may indicate whether a single or multiple reference frames and / or a combination of different predictions are used, such as joint inter-frame and intra-frame prediction, etc.

[0218] However, when the data to be encoded is a motion vector field, efficient motion vector field decoding and reconstruction can be achieved. Figure 12 A corresponding general block scheme for a device capable of performing motion field encoding and decoding is shown. At the encoding end, motion information is obtained using some motion estimation or optical flow estimation modules (units) 1210. The input for motion vector (optical flow) estimation is the current image and one or more reference images (stored in a reference image buffer). Figure 12 In this context, the image is referred to as a "frame," a term sometimes used for images in video. The optical flow estimation unit 1210 outputs optical flow 1215. In different implementations, the motion estimation unit may output motion information with different spatial resolutions; for example, for some N×N blocks or for each pixel at the original resolution, this can be referred to as dense optical flow. Motion vector information is intended to be transmitted (embedded into the bitstream 1250) to the decoder and used for motion compensation. To obtain the motion-compensated region, each pixel in that region should have a defined motion vector. Transmitting motion vector information for each pixel at the original resolution may be too costly. To reduce indication overhead, a motion specification (or segmentation) module 1220 is used. The corresponding module 1270 at the decoder performs a motion generation (dense) task to reconstruct the motion vector field 1275. The motion specification (or segmentation) module 1220 outputs motion information (e.g., motion vectors, and / or possibly a reference image) and segmentation information. This information is added (encoded) into the bitstream.

[0219] In this embodiment, the motion segmentation unit 1220 and the motion generation unit 1270 only include a downsampling layer dwnspl and a corresponding upsampling layer uppl, such as Figure 13 As shown. The nearest neighbor method can be used for both downsampling and upsampling; average pooling can be used for downsampling. Feature map data from layers with different spatial resolutions are selected by the encoder and transmitted in the bitstream as selection information 1120 along with segmentation information 1130, which instructs the decoder how to interpret and utilize the selection information 1120. The motion segmentation (sparser) network 1220 is in Figure 13 The network is shown as 1310. Therefore, a dense optical flow 1215 is inserted into the motion segmentation (sparserization) network 1310. Network 1310 includes three downsampling layers and signal selection logic 1320, which selects the information to be included in the bitstream 1350. This function is related to... Figure 9 (More generally) similar to what is described.

[0220] In the above embodiments, the indication of information related to layers different from the output layer improves the scalability of the system. Such information can be related to hidden layers. Below, embodiments and examples are given that utilize the provided scalability and flexibility. That is, some methods are provided regarding how to select layers and how to display information.

[0221] Some embodiments in this document illustrate image or video compression systems using an autoencoder architecture that includes one or more dimensional (or spatial resolution) reduction steps in the encoding section (implemented by layers that incorporate downsampling operations). Along with the simplification (encoding) end, a learning reconstruction (decoding) end is performed, where the autoencoder attempts to generate a representation from the simplified encoding that is as close as possible to its original input. This typically implies one or more resolution upsampling steps in the decoding end (implemented by layers that include upsampling operations).

[0222] Here and thereafter, under the action of the encoder, the encoding part of the autoencoder generates the latent signal representation included in the bitstream. For example, such an encoder is encoder 101 or 121 mentioned above. Under the action of the decoder, the generating part of the autoencoder senses the latent signal representation obtained from the bitstream. For example, such a decoder is decoder 104 or 144 mentioned above.

[0223] For reference Figure 11 As described above, the encoder selects a portion (or multiple portions) of feature map information (selection information 1120) from layers with different spatial resolutions according to signal selection logic 1100, and transmits the selection information 1120 in the bitstream 1150. Segmentation information 1130 indicates which layer and which portion of the feature map of the corresponding layer the selection information is obtained from.

[0224] According to one embodiment, the processing of the j-th layer in a plurality (N) cascaded layers includes: Determine the first cost incurred in reconstructing a portion of the image using feature map elements output from the j-th layer; Determine the second cost incurred in reconstructing the portion of the reconstructed image using feature map elements output from the (j-1)th layer; If the first cost is higher than the second cost, then the (j-1)th layer is selected, and information related to the portion in the (j-1)th layer is selected.

[0225] The selection of which layer to use can be based on the distortion or the distortion function. For example, for motion vector field decoding, the reconstructed image (or image portion) could be a motion-compensated image (or image portion).

[0226] In order to select selected information in this exemplary implementation, the encoder includes a cost calculation unit (module) that estimates the cost of transmitting motion information from a specific resolution layer at a specific location. The cost is calculated using the distortion resulting from motion compensation of the selected motion vector, combined with an estimate of the number of bits required to transmit the motion information, multiplied by a Lagrange multiplier. That is, according to one embodiment, rate-distortion optimization (RDO) is performed.

[0227] That is, in some embodiments, the first and second costs include data volume and / or distortion. For example, the data volume includes the amount of data required to transmit data associated with the selected layer. This can be motion information or other information. It can also be or include the overhead generated by residual decoding. Distortion is calculated by comparing the reconstructed image with the target image (the original image to be encoded or a portion of the original image to be encoded). It should be noted that RDO is only one possibility. The invention is not limited to this method. Furthermore, complexity or other factors may be included in the cost function.

[0228] Figure 14 The first part of the cost calculation is shown. Specifically, the cost calculation (or estimation) unit 1400 obtains the optical flow L1 downsampled by the downsampling layer (downspl4) of the motion segmentation unit 1140. Then, the cost calculation unit 1400 upsamples the optical flow 1415 to its original resolution, for example, in this case, upsampling 4 in each of the two directions (x and y). Then, motion compensation 1420 is performed using the upsampled motion vector output from 1410 and the reference image 1405 to obtain the motion of the compensated frame (image) or a portion of the motion compensated frame (image) 1420. Then, distortion 1430 is calculated by comparing the motion compensated image (part) 1420 with the target image 1408. For example, the target image 1408 may be the image to be encoded (the original image). In some exemplary implementations, the comparison can be performed by calculating the mean squared error (MSE) or the sum of absolute differences (SAD) between the target image 1408 and the motion compensated image 1420. However, other types of measurements / metrics can be used instead or alternatively, such as more advanced metrics for subjective perception, such as MS-SSIM or VMAF. The calculated distortion 1430 is then provided to the cost calculation module 1460.

[0229] Furthermore, the rate estimation module 1440 calculates an estimate of the number of bits for each motion vector. The rate estimate may include not only bits indicating the motion vector signal but also bits indicating segmentation information (in some embodiments). For example, the number of bits thus obtained can be normalized pixel-by-pixel (feature map element) 1450. The resulting rate (number of bits) is provided to the cost calculation module 1460. To obtain the rate (number of bits) estimate, for example, a motion information decoding module (e.g., by performing decoding and recording the resulting number of bits) or, in some simplified implementations, the length of the motion vector with its x or y component can be used as a rough estimate. Another estimation technique can be applied. To account for segmentation information, it can be evaluated by a segmentation information decoding module (e.g., by generating and decoding segmentation information and counting the number of resulting bits) or, in a simpler implementation, by adding the bits to the total number of bits.

[0230] In this example, the next step in cost calculation is cost calculation 1460, followed by downsampling by 4 1470 (downspl4) to achieve the resolution of the corresponding downsampling layer of motion segmentation unit 1100. Only one motion vector can be transmitted per point (image sample value). Therefore, the resulting cost tensor can have a corresponding size (dimension). Thus, the bit evaluation value can be normalized by the square of the downsampling filter shape (e.g., 4×4).

[0231] Then, using Lagrange multipliers, cost estimation unit 1460 calculates the cost using the following formula: .

[0232] in, This indicates distortion (calculated by 1430). This represents the bit estimate (the rate estimate output by 1440 or 1450). and It is a Lagrange multiplier. Downsampling to 1470 outputs a cost tensor of 1480. As is known in the art, the Lagrange multiplier and... and It can be learned through experience.

[0233] This yields a tensor of 1480 with cost estimates for each location in the feature map (in this case, the W×H location of the dense optical flow). It should be noted that using sequential average pooling and upsampling with the nearest neighbor method results in averaging the motion vector over an N×N (e.g., 4×4) region, where N×N is the average pooling filter shape and scaling factor of the upsampling operation. During upsampling using the nearest neighbor method, the values ​​of the lower-resolution layer are copied (repeated) across all points in the higher-resolution layer corresponding to the filter shape. This corresponds to a translational motion model.

[0234] Various implementations of the cost selection unit are possible. For example, Figure 15 Another exemplary implementation is shown. In this example, with... Figure 14 The difference is that the motion vector field obtained after downsampling the dense optical flow by 4 in each of the x and y dimensions 1501 is not upsampled 1415. Instead, the motion vector field is directly provided to motion compensation 1510 and rate estimation 1540. Conversely, before motion compensation 1510 and distortion assessment 1530, the reference image 1505 and the target image 1505 can be downsampled 1515, 1518 to the corresponding resolution. This eliminates the need for performing... Figure 14 The initial motion field is upsampled to the original resolution by 1415, and no steps are performed. Figure 14 The final cost downsampling step 1470. This also makes bit normalization 1450 unnecessary. This implementation requires less memory to store tensors during processing, but may provide less accurate results. It should be noted that, to speed up or reduce the complexity of RDO, one can imagine downsampling dense optical flow as well as the reference and target images, even more than L1. However, the accuracy of such RDO may be further reduced.

[0235] Cost estimation units (1400, 1500) are applied to each downsampling layer of motion segmentation units (1220, 1310) to obtain costs with different motion vector averaging levels (different spatial resolutions). As a next step, signal selection logic 1100 uses the cost information from each downsampling layer to select motion information at different spatial resolutions. To enable signal selection logic 1100 to perform pairwise comparisons of costs from sequential (cascaded) downsampling layers, signal selection logic 1100 selects the minimum cost at each spatial location and propagates it to the next (in processing order) downsampling layer. Figure 16 An exemplary architecture of the signal selection unit 1600 is shown.

[0236] The dense optical flow 610 enters three downsampling layers downspl4, downspl2, and downspl2, similar to... Figure 11 The downsampling layer shown in the image. Figure 16 The signal selection logic 1600 in the middle is Figure 11 An exemplary implementation of the signal selection logic 1100 is shown below. Specifically, the LayerMv tensor 611 is a subsampled motion vector field (feature map) that enters the cost calculation unit 613. The LayerMv tensor 611 also enters the layer information selection unit 614 of the first layer. If a selected motion vector exists on this (first) layer, the layer information selection unit 614 provides the selected motion vector to the bitstream. Its function will be further described below.

[0237] For example, cost calculation unit 613 calculates costs, as referenced Figure 14 This is described for cost calculation unit 1400. The cost calculation unit outputs a cost tensor, which is then downsampled by 2 to match the resolution of the second layer's operation. After processing by the second downsampling layer downspl2, the LayerMV tensor 621 is provided to the next (third) layer and the cost calculation unit 623 of the second layer. Cost calculation unit 623 operates in a similar manner to cost calculation unit 1400. As will be apparent to those skilled in the art, with reference to... Figure 14 The example described uses different upsampling / downsampling for 4, with downsampling of 2 applied in each direction.

[0238] To perform a pairwise comparison of the cost tensors from cost calculation units 613 and 623, the cost tensor from the previous (first) downsampling layer has been downsampled (2) to the current resolution layer (second layer). Then, a pooling operation 625 is performed between the two cost tensors. That is, the pooling operation 625 retains the lower cost in each element's cost tensor. The layer with the lower cost is selected based on the element index of the pooling operation result. For example, if the cost of the first tensor is lower than the cost of the corresponding element in the second tensor within a given tensor element, the index is equal to 0; otherwise, the index is equal to 1.

[0239] To ensure gradient propagation for training purposes, a soft arg max can be used to obtain pooling indices with gradients. If gradient propagation is not required, regular pooling with indices can be used. As a result of pooling operation 625, 622 indicates whether the motion vector (LayerFlag tensor) from the current resolution layer or the previous resolution layer has been selected, and the index of the motion vector from the corresponding downsampled layer of the motion segmentation unit (LayerMv tensor) is transmitted to the layer information selection unit 624 of the current (here, the second) layer. The optimal pooling cost tensor is propagated to the next downsampling level (downspl2), and then the operation is repeated for the third layer.

[0240] Specifically, the output LayerMv 621 of the second layer is further downsampled (downspl2) by the third layer, and the resulting motion vector field LayerMv 631 is provided to the cost calculation unit 633 of the third layer. The calculated cost tensor is compared element-wise with the downsampled cost tensor propagated from the second layer and provided by the MinCost pooling unit 625. After the MinCost pooling 635 processing, the index indicating whether the motion vector (LayerFlag tensor) from the current (third) resolution layer or the previous (second) resolution layer is selected, and the index of the motion vector from the corresponding downsampled layer of the motion segmentation unit (LayerMv tensor) is transmitted to the layer information selection unit 634 of the current (here, the third) layer. In this example, only three layers are shown as an example. However, typically, there may be more than three layers, where the other layers and signal selection logic of these layers have functions similar to those shown in the second and third layers.

[0241] To collect pooling information from each spatial resolution layer, the next process is performed in reverse order—using layer information selection units 634, 624, and 614 from lower resolution layers to higher resolution layers. First, the TakeFromPrev tensor, the same size as the lowest resolution layer (here, the third layer), is initialized to 0 at 601. Then, the same operation is repeated for layers of different resolutions, as follows. Where the value of the tensor (NOT TakeFromPrev) is equal to 1, the value of the LayerFlag tensor (in the current layer) is selected as the segmentation information and transmitted in the bitstream. The (NOT TakeFromPrev) tensor is the element-wise negation of the TakeFromPrev tensor. Therefore, in the third layer (here, the last layer), the (NOT TakeFromPrev) tensor sets all values ​​to 1 (the negation of 0 values ​​set by 601). Thus, the segmentation information 1130 (LayerFlag) of the last (here, the third) layer is always transmitted.

[0242] The TakeFromCurrent tensor is obtained using the logical operation TakeFromCurrent = (NOT TakeFromPrev) AND LayerFlag. The TakeFromCurrent flag in this tensor indicates whether motion vector information is selected to be transmitted in the bitstream of the current resolution layer. Layer information selection units (634, 624, 614) select motion vector information from the corresponding downsampling layer of the motion segmentation unit by taking the value of the LayerMv tensor, where the TakeFromCurrent tensor is equal to 1. This information is transmitted in the bitstream as selection information 1120.

[0243] For the third (first in reverse order) processing layer corresponding to the lowest resolution, all flags will be transmitted because TakeFromPrev is initialized to 0, and then all values ​​of (NOT TakeFromPrev) are equal to 1. For the last processing layer corresponding to the highest resolution layer, the LayerFlag flag does not need to be transmitted. It is assumed that for all locations where motion information was not selected from the previous layer, these locations should be selected from the current or next (highest resolution) layer.

[0244] It should be noted that, as Figure 16 The cost calculation shown is a parallelizable scheme that can run on a GPU / NPU. This scheme is also trainable because it is transmitting gradients, and can be used in end-to-end trainable video decoding solutions.

[0245] It should be noted that the reverse order processing is similar to the processing performed by the decoder when parsing segmentation information and motion vector information, as shown in the following discussion of decoder functions.

[0246] Another exemplary implementation of the signal selection logic 1700 is in Figure 17 As shown in the image. (and) Figure 16 compared to, Figure 17 The block diagram introduces multiple decoding options at the same resolution layer. This is illustrated by options 1 to N in the first layer cost calculation unit 710. It should be noted that typically one or more or all layers can include more options. That is, any of the cost calculation units 613, 623, and 633 can provide more options. For example, these options can be one or more or all of the following: different reference images for motion estimation / compensation, one-way prediction, two-way prediction or multi-hypothesis prediction, direct decoding where different prediction methods (e.g., inter-frame prediction or intra-frame prediction) do not predict, multi-hypothesis prediction, presence or absence of residual information, quantization level of residuals, etc. In the cost calculation unit 710, a cost is calculated for each decoding option. Then, the best option is selected using minimum cost pooling 720. An indicator (e.g., index) 705 for the best selected option is transmitted to the layer information selection module 730, and then, if the corresponding point of the current layer will be selected to transmit information, the indicator BestOpt is transmitted in the bitstream. In the given example, the options are shown only for the first layer, but it should be understood that similar option selection logic can also be applied to other layers at different resolutions, or to all layers.

[0247] The above method is also applicable to the segmentation and transmission of logical information, such as flags or switches controlling the image reconstruction process, and information that remains unchanged after decoding and identical to that at the encoding end. That is, instead of in Figure 16The motion vector field (dense optical flow) processed in the exemplary implementation can be encoded in a similar manner, including segmentation, for any one or more other parameters. Parameters can be one or more of the following: indicators indicating different reference images, etc., used for motion estimation / compensation; unidirectional, bidirectional, or multi-hypothesis prediction indicators; different prediction methods, such as inter-frame or intra-frame prediction; direct decoding indicators without prediction; multi-hypothesis prediction; the presence or absence of residual information; the quantization level of the residual; parameters of the in-loop filter, etc.

[0248] Further modifications to the above embodiments and examples According to the first modification, the downsampling layer of the motion segmentation unit 1310 and / or the upsampling layer of the motion generation unit 1360 include convolution operations. This in Figure 18 As shown in the image. Figure 18 As shown, with Figure 13 In contrast, the downsampling layer "dwnspl" and the upsampling layer "upspl" are respectively oversampling convolutional layers in motion segmentation unit 1810. "Replacement, and upsampling convolutional layers in motion generation unit 1860" "Replacement. Some advantages of convolutional rescaling (downsampling, upsampling) layers are that they enable learnable downsampling and upsampling processes. For example, in cases where motion information is dense, it helps to find the optimal upsampling transform and thus can reduce block artifacts caused by motion compensation using block-averaged motion vector information, as described in the embodiments and examples above. The same applies to texture restoration processes, such as for generating raw image intensity values ​​or prediction residuals processed by cascaded layers."

[0249] The above text Figure 18 In the example, all downsampling and upsampling layers are convolutional layers. However, this invention is not typically limited to this implementation. Generally, within the segmentation units (1310, 1810) and / or the generation units (1360, 1860), subsets (one or more) of the downsampling and corresponding upsampling operations can be implemented as convolutions.

[0250] Please recall that although the examples described herein are for dense optical flow / motion vector field processing, and thus refer to motion segmentation units (1310, 1810) and / or motion generation units (1360, 1860), the invention is not limited to such data / feature maps. Instead, in any embodiments and examples herein, any decoding parameter or even texture, such as samples of an image or prediction residuals (prediction errors), can be processed. Or, in addition to motion vector fields, any decoding parameter or even texture can be processed.

[0251] It should be noted that, for example, an encoder with motion information averaged in downsampling can be used in conjunction with a decoder that includes convolutional upsampling layers. Furthermore, an encoder with convolutional layers, aiming to find better latent representations, can be combined with a motion generation network (decoder) that implements nearest-neighbor-based upsampling layers. Other combinations are possible. That is, upsampling and downsampling layers do not necessarily have to be similar types.

[0252] According to a second modification that can be combined with any of the above embodiments and examples (and the first modification), the network processing includes one or more additional convolutional layers between cascaded layers having the different resolutions described above. For example, the motion segmentation unit 1310 and / or the motion generation unit 1360 also include one or more intermediate convolutional layers between partially or entirely downsampling layers and upsampling layers. This in Figure 19 As shown in the figure, Figure 19 Exemplary implementations of such motion segmentation network (module) 1910 and motion generation network (module) 1860 are shown. It should be noted that the terms "module" and "unit" are used interchangeably here to refer to functional units. In this specific embodiment, units 1910 and 1960 are more specifically network structures with multiple cascaded layers.

[0253] For example, compared to motion segmentation unit 1310, motion segmentation unit 1910 has a lower resolution at each downsampling layer. (This could also be other types of downsampling) has an additional convolutional layer "conv" before it. In addition, compared with the motion generation unit 1360, the motion generation unit 1960 has an additional convolutional layer "conv" before each upsampling layer "conv" (which could also be other types of upsampling).

[0254] This can further reduce clogging artifacts caused by motion information sparsity and increase the generalization effect in finding better potential representations. Similar to the first modification mentioned above, encoders and decoders from the different embodiments / modifications described above can be combined in a single compression system. For example, there can be only an encoder with additional layers between downsampling layers and a decoder without such additional layers, or vice versa. Alternatively or additionally, there can be different numbers and locations of such additional layers at the encoder and decoder.

[0255] According to the third amendment, such as Figure 19 As shown, a direct connection between the input and output signals is provided. It should be noted that, even though shown in the same figure here, the second and third modifications are independent. They can be applied together or separately to the previously described embodiments and examples, as well as other modifications. Direct connections are indicated by dashed lines.

[0256] In addition to bottleneck information from the autoencoder's potential representation (the output of the lowest resolution layer), in some embodiments, information from higher resolution layers is added to the bitstream. To optimize indication overhead, only a portion of the information from different resolution layers is inserted into the bitstream, controlled by signal selection logic. On the receiver (decoder) side, corresponding signal feeding logic feeds information from the bitstream to layers of different spatial resolutions, as described in detail below. Furthermore, information from the input signal prior to the downsampling layer can be added to the bitstream, further increasing variability and flexibility. For example, decoding can be aligned to real object boundaries and segmentations with higher spatial resolution and adjusted according to the characteristics of a specific sequence.

[0257] According to the fourth modification, the shapes of the downsampling and upsampling filters can be shapes other than squares, such as rectangles, with horizontal or vertical orientations, asymmetrical shapes, or other more arbitrary shapes through masking operations. This helps to further increase the variability of the segmentation process to better capture the boundaries of real objects. This modification is in... Figure 20 As shown in the diagram. In the motion segmentation unit 2010, after the first downsampling layer, and after a first downsampling layer that may be the same as any of the above embodiments, two additional downsampling layers use selected filters from the set of filter shapes. This modification is not limited to the processing of motion vector information.

[0258] Typically, when downsampling at a layer, a first feature map is obtained by downsampling the input feature map using a first filter, and a second feature map is obtained by downsampling the input feature map using a second filter. Cost calculation includes determining a third cost incurred in reconstructing a portion of the reconstructed image using the first feature map, and a fourth cost incurred in reconstructing a portion of the reconstructed image using the second feature map. Then, during selection, the first feature map is selected if the third cost is less than the fourth cost, and the second feature map is selected if the third cost is greater than the fourth cost. In this example, selection is made from two filters. However, the invention is not limited to two filters, but can be made from a predefined number of filters in a similar manner, for example, by estimating the cost of all selectable filters and minimizing the cost by selecting a filter.

[0259] The shapes of the first and second filters can be any of squares, horizontal rectangles, and vertical rectangles. However, the invention is not limited to these shapes. Generally, any arbitrary filter shape can be designed. Filters can also include filters that can be defined with any desired shape. Such a shape can be represented by obtaining a mask, wherein the mask consists of flags that represent an arbitrary filter shape, and one of the first and second filters (typically from any selectable filters in the filter set) has an arbitrary filter shape.

[0260] In an exemplary implementation, to provide variability, the encoder also includes pooling between cost tensors obtained using filters with mutually different shapes. The index of the selected filter shape is indicated in the bitstream as part of the segmentation information, similar to the motion vector described above. For example, for selecting between rectangular shapes in the horizontal and vertical directions, a corresponding flag can be indicated in the bitstream. For example, refer to... Figure 17 The described method for selecting multiple encoding options can be used to select different filter shapes at the same resolution layer.

[0261] According to the fifth modification, motion models from a predefined set of different motion models can be selected within the same resolution layer. In previous embodiments, specific cases of downsampling and / or upsampling filters have been described. In this case, motion information can be averaged over the blocks representing the translational motion model. In the fifth modification, in addition to the translational motion model, other motion models can also be employed. Such other motion models may include one or more of the following: Affine motion model Higher-order motion models, or CNN layers are specifically trained to represent specific motion models such as scaling, rotation, affine, and perspective.

[0262] In the exemplary implementation of the fifth modification, the autoencoder also includes a set of CNN layers and / or "handcrafted" layers representing the translational motion model. This autoencoder (and decoder) is as follows: Figure 21 As shown. In Figure 21 In this context, the autoencoder includes layers provided at both the encoding and decoding ends, comprising a set of filters denoted as a "convolutional fit set".

[0263] For example, at each spatial layer, the encoder selects the appropriate filter from the filter set corresponding to a specific motion model and inserts the indicator into the bitstream. At the receiver, the signal feed logic interprets the indicator and performs convolution at the specific layer using the corresponding filter from the set.

[0264] The examples of the methods described above use motion information, specifically motion vectors, as exemplary input for encoding. It should be noted again that these methods are also applicable to the compression of different types of image or video information, such as direct image sample values, prediction residual information, intra-frame prediction parameters, and inter-frame prediction parameters.

[0265] According to the sixth amendment, the above references Figure 16 or Figure 17 The RDO example can be applied to traditional block-based codecs.

[0266] Traditional video coding methods, such as state-of-the-art video coding standards like AVC, HEVC, VVC, or EVC, use a block-based coding concept, according to which images are recursively divided into square or rectangular blocks. For these blocks, signal reconstruction parameters are estimated or evaluated at the encoder and transmitted to the decoder in the bitstream. Typically, the encoder aims to find the optimal reconstruction parameters for the set of blocks representing the image in terms of rate distortion cost, attempting to maximize reconstruction quality (i.e., minimize distortion of the original image) and minimize the number of bits required to transmit the reconstruction process parameters. Parameter selection (or decoding mode decision) is a complex and resource-intensive task, and a major source of encoder complexity. For example, in real-time applications, the encoder may sacrifice the quality of mode decisions, which in turn affect the quality of the reconstructed signal. Optimizing the mode decision process has been a desirable technological improvement.

[0267] One of the decoding mode decisions is whether to divide the current block (or coding unit (CU)) into multiple blocks according to the segmentation method.

[0268] According to the sixth modification, the motion segmentation unit 1310 (or 1810) described above is adapted to make a segmentation pattern decision based on minimizing cost (e.g., rate distortion optimization criterion). Figure 22 An example of this optimization is shown.

[0269] like Figure 22 As shown, the block segmentation structure is used to represent information at different spatial resolutions, rather than as a downsampling layer. For each block of a given size N×N (considering squares) of an image or a portion of an image, the cost computation unit computes the distortion tensor and further downsamples the resolution by a factor of 16 (to match the original resolution). Figure 22 In the given example above, the first block size is 16×16 (e.g., downsampling is performed via average pooling) to obtain a tensor, where each element represents the average distortion in each 16×16 block. In the first layer, the image is segmented into 16×16 blocks 2201 at the initial highest resolution. In the second layer, the resolution is reduced so that the block size 2202 in the image is 32×32 (corresponding to the four blocks connected to the previous layer). In the third layer, the resolution is reduced again so that the block size 2203 is 64×64 (corresponding to the four blocks connected to the previous layer). It should be noted that, in this case, connecting four blocks from the previous layer can be considered as subsampling of block-related information. This is because in the first layer, block-related information is provided for each 16×16 block, while in the second layer, block-related information is only provided for 32×32 blocks, i.e., four fewer parameters are provided. Similarly, in the third layer, block-related information is only provided for 64×64 blocks, i.e., four fewer timing parameters are provided than in the second layer, and 16 times fewer than in the first layer.

[0270] In this context, block-related information is any information decoded for each block, such as the prediction mode; prediction mode-specific information, such as motion vectors, prediction directions, reference images, etc.; filtering parameters; quantization parameters; transform parameters or other settings that may change at the block (coding unit) level.

[0271] Then, the corresponding cost calculation units 2211, 2212 and 2213 of the first, second and third layers calculate the cost based on the block reconstruction parameters for the corresponding block sizes 2201, 2202 and 2203 and based on the input image of size W×H.

[0272] The output cost tensor is obtained as the average distortion in each block and combined with the bit estimate required to transmit the decoding parameters of an N×N (e.g., 16×16 for the first layer) block using Lagrange multipliers. Figure 23 An exemplary structure for a cost calculation unit 2300 for a block N×N is shown (which may correspond to each or any one of cost calculation units 2211, 2212 and 2213).

[0273] Figure 23 An exemplary block diagram of a cost calculation unit 2300 with a typical block size of N×N 230x is shown. The cost calculation unit 2300 acquires 2310 block reconstruction parameters (block-related parameters) associated with blocks of size N×N. This acquisition can correspond to acquiring parameters (parameter values) from memory, etc. For example, the block-related parameters can be a specific prediction mode, such as an inter-frame prediction mode. In unit 2310, the block reconstruction parameters are acquired, and in reconstruction unit 2320, these parameters are used to reconstruct a portion of the image (in this example, all blocks are reconstructed using the inter-frame prediction mode). The distortion calculation unit 2330 calculates the distortion of the reconstructed portion of the image by comparing it to a corresponding portion of a target image, which can be the original image to be encoded. Since the distortion can be calculated per sample, downsampling 2340 (one value per N×N block) can be performed on the distortion to acquire the distortion on a block-by-block basis. In a lower branch, the rate or number of bits required to decode the image is estimated 2360. Specifically, the bit estimation unit 2360 can estimate the number of bits to be indicated for each N×N block. For example, it can calculate the number of bits per block required for inter-frame prediction modes. With the estimated distortion and bit quantity (or rate), the cost can be calculated, for example, according to the Lagrangian optimization mentioned above. The output is a cost tensor.

[0274] It should be noted that throughout this description, if only a 2D image of the sample is observed, such as a grayscale image, the term "tensor" here can refer to a matrix. However, images may have multiple channels, such as color or depth channels, so the output may also have more dimensions. Feature maps can generally also exceed 2 or 3 dimensions.

[0275] For the first level (with 16×16 block granularity) and the next level quadtree divided into blocks of size 32×32, the same cost evaluation process is performed. To determine whether it is better to use a single 32×32 block as the reconstruction parameter (block-related parameter) or four 16×16 blocks, the cost tensor for the 16×16 block evaluation is downsampled by a factor of 2 (see [link to cost tensor]). Figure 22 Then, the minimum cost pooling operation 2222 provides the optimal decision for each 32×32 block. The index of the pooling cost is passed to the layer information selection unit 2232 to be transmitted in the bitstream as split_flags. The reconstruction parameters blk_rec_params of the best selected block based on the pooling index are also passed to the layer information selection unit 2231. The pooling cost tensor is further passed (downsampled by 2) to the next quadtree aggregation level for a block of size 64×64, i.e., to the MinCost pooling 2223. The MinCost pooling 2223 also receives the cost calculated in the cost calculation unit 2213 for the 64×64 block resolution 2203. It passes the index of the pooling cost as split_flags to the layer information selection unit 2233 to indicate this in the bitstream. The reconstruction parameters blk_rec_params of the best selected block based on the pooling index are also passed to the layer information selection unit 2233.

[0276] To collect pooling information from each block aggregation level, use layer information selection units 2233, 2232, and 2231 as referenced above. Figure 16 The processing is performed in reverse order from the higher (highest in this example) aggregation level (64×64 samples) to the lower (lowest in this example) aggregation level (16×16 samples).

[0277] This yields the bitstream partitioned by the quadtree obtained through optimization, along with the encoded values ​​and further decoding parameters for possible resulting blocks. The above method can make decisions regarding the partitioning criteria for blocks. To obtain the reconstruction parameters for each block, traditional methods based on evaluating each or some possible decoding modes can be used.

[0278] Figure 24 An example of the seventh modification is shown. The seventh modification is referenced above. Figure 22 and Figure 23The sixth modification described is an evolution of the previous one. The seventh modification represents a scheme in which the evaluation of decoding modes is incorporated into the design. Specifically, as shown in the figure, the cost calculation unit 710 can evaluate N options. It should be noted that the term "N" here is a placeholder for an integer. The "N" representing the number of options is not necessarily the same as the "N" in "N×N" representing the general block size. In the cost calculation unit 710, for block partitions of the same level, for example, for blocks of size 16×16 samples (as in the first layer), the encoder iterates over all possible (or a finite set thereof) decoding modes for each block.

[0279] There need to consider N options for decoding each 16×16 block, denoted as blk_rec_params 0, blk_rec_params 1, ..., blk_rec_params N. For example, the parameter combination blk_rec_params k (where k is an integer from 0 to N) can be a combination of certain prediction modes (e.g., inter-frame prediction or intra-frame prediction), certain transforms (e.g., DCT or KLT), certain filter orders, or filter coefficient sets (in a predefined filter), etc. In some implementations, blk_rec_params k can be the value k of a single parameter if only one parameter is being optimized. Those skilled in the art will understand that any one or more parameters can be optimized by examining their usage costs.

[0280] For each given set of block reconstruction parameters (blk_rec_params k), cost calculation unit 2410 computes a tensor representing the cost of each block. Then, using minimum cost pooling 2420, the optimal decoding mode for each block is selected and passed to layer information selection unit 2430. The optimal pooling cost tensor is further downsampled by a factor of 2 and passed to the next quadtree aggregation level (in this example, the second level corresponds to an aggregation with a block size of 32×32). Then, a partitioning (splitting) decision is made in the same manner as the sixth modification described above. Figure 24 In this method, options 0...N are evaluated only at the first level (aggregation level 16×16). However, the invention is not limited to this method. Instead, the evaluation of options 0...N can be performed at each aggregation level.

[0281] For example, at the next quadtree aggregation level (32×32, 64×64), the encoder evaluates (by calculating the cost in the corresponding cost unit) and pools (by the corresponding MinCost pooling unit) the optimal decoding mode for each block (not shown in the image for ease of understanding), comparing it with the previous aggregation level. The decision regarding the optimal mode and the corresponding set of reconstruction parameters is provided to the layer information selection unit (e.g., layer information selection unit 2430 shown for the first layer). To collect pooling information from each block aggregation level, the layer information selection unit performs processing in reverse order from the higher aggregation level (64×64) to the lower aggregation level (16×16), as described in the sixth modification.

[0282] Different block shapes can be used to represent more advanced partitioning methods, such as binary trees, ternary trees, asymmetric partitions, and geometric partitions. Figure 25 This example illustrates this type of block partitioning. That is, optimization is not necessarily performed only on different block sizes; it can also be performed on different partition types (e.g., through corresponding options). Figure 25 The following example is shown: Quadtree partitioning 2510: In quadtree partitioning, a block is divided (split) into four blocks of the same size.

[0283] (Symmetric) Binary Tree Partition 2520: In a symmetric binary tree partition, a block is divided into two blocks of the same size. The partition can be vertical or horizontal. Vertical or horizontal are additional parameters for the partition.

[0284] (Asymmetric) Binary Tree Partitioning 2530: In asymmetric binary tree partitioning, a block is divided into two blocks of different sizes. The size ratio can be fixed (to save on overhead caused by indices) or variable (in which case some ratio options can also be optimized, i.e., configurable).

[0285] Tritree partitioning 2540: In ternary tree partitioning, a block is divided into three sub-blocks by two vertical lines or two horizontal lines. Vertical or horizontal are additional parameters for partitioning.

[0286] This invention is not limited to these exemplary partitioning patterns. Tritree partitioning or any other type of partitioning can be used.

[0287] In the seventh amendment, hybrid architectures applicable to popular video coding standards are supported and licensed with a method based on powerful (neural) networks. The technical advantages of the described method provide a highly parallelizable GPU / NPU-friendly solution that can help accelerate the computational speed of the pattern decision process. The method enables global image optimization because multiple blocks are considered at the same decision level, thus incorporating learnable parts to speed up decision-making, for example, the number of bits required to decode reconstruction parameters.

[0288] In summary, according to the sixth or seventh modification, the processing of the cascaded layer structure includes processing data of the same image that is segmented (i.e., partitioned / segmented) into blocks with different block sizes and / or shapes in different layers. Layer selection includes choosing layers based on the computational cost of a predetermined set of decoding patterns.

[0289] That is, different layers can process image data with different block sizes. Therefore, a cascaded layer consists of at least two layers that process each other with different block sizes. When "block" is mentioned here, it means a unit, that is, a portion of the image to which decoding is performed. This block can also be called a decoding unit or a processing unit, etc.

[0290] A predetermined set of decoding modes corresponds to combinations of decoding parameter values. Different block sizes can be evaluated within a single set of decoding modes (a combination of one or more decoding parameter values). Alternatively, the evaluation can include various combinations of block size and segmentation shape (e.g., Figure 25 (Combinations of [various parameters]). However, the invention is not limited thereto, and, as specifically mentioned in the seventh modification, there may be several predetermined sets of decoding modes (combinations of decoding parameter values), which may also include, for example, decoding modes such as intra / inter-frame prediction types for each block, intra-frame prediction modes, residual skipping, residual data, etc.

[0291] For example, the process includes determining the cost (a combination of decoding parameter values) of different sets of decoding modes for at least one layer, and selecting one of the sets of decoding modes based on the determined cost. Figure 24 The diagram illustrates the case where only the first layer performs this selection. However, this is not a limitation. Quickly, each cost calculation unit can have the same structure as the first cost calculation unit 2410, including options 0…N. This is not shown in the diagram for simplicity.

[0292] As mentioned above, this is a GPU-friendly RDO that can be executed by the codec and selects the optimal decoding mode for each block. Figure 24In this approach, the input image (picture) is the same in every layer. However, the decoding (computational cost) of the image is performed in each layer with different block sizes. In addition to the block size, other decoding parameters can be selected based on RDO testing and for one or more block sizes.

[0293] Specifically, in these modifications, the indication of data associated with the selected layer includes the selected set of decoding modes (e.g., blk_rec_params).

[0294] In summary, in some embodiments, an encoder may be provided whose structure corresponds to a neural network autoencoder for decoding video or image information. Such an encoder can be used to: analyze input image or video information using a neural network comprising layers of different spatial resolutions; transmit a latent representation in the bitstream corresponding to the output of the lowest resolution layer; and transmit outputs other than the lowest resolution layer in the bitstream.

[0295] decoding The encoder described above provides a bitstream that includes feature data and / or segmentation information for a selected layer. Accordingly, the decoder performs multi-level processing on the data received from the bitstream. Furthermore, the selected layer receives additional (direct) input from the bitstream. This input may be some feature data information and / or segmentation information.

[0296] Accordingly, embodiments described below focus on information related to a selected layer as feature data. Other described embodiments focus on information related to a selected layer as segmentation information. Hybrid embodiments of bitstream carrying and layer processing feature data and segmentation information also exist.

[0297] As a simple example, a decoder for a neural network autoencoder can be provided for decoding video or image information. The decoder can be used to: read the latent representation corresponding to the input of a lower-resolution layer from the bitstream; obtain the layer input information based on the corresponding information read from the bitstream other than the lower-resolution layer; obtain the combined input of the layer based on the layer information obtained from the bitstream and the output of the previous layer; feed the combined input to the layer; and synthesize an image based on the output of the layer.

[0298] Here, the term "lower resolution" refers to a layer that processes feature maps with a lower resolution, such as feature maps of the latent space provided by the bitstream. Lower resolution can actually be the lowest resolution of the network.

[0299] The decoder can also be used to: obtain segmentation information based on corresponding information read from the bitstream; and obtain combined inputs for layers based on the segmentation information. The segmentation information can be a quadtree, binary tree, or ternary tree data structure, or a combination thereof. For example, layer input information can correspond to motion information, image information, and / or prediction residual information, etc.

[0300] In some examples, a hyperprior neural network is used to decode information corresponding to the layer input information obtained from the bitstream. A hyperprior neural network can be used to decode information corresponding to segmentation information obtained from the bitstream.

[0301] The decoder can be readily applied to decoding motion vectors (e.g., motion vector fields or optical flow). Some of these motion vectors can be similar or related. For example, in a video displaying an object moving against a constant background, there could be two sets of similar motion vectors. The first set could be vectors used to predict pixels of the displayed object, and the second set could be vectors used to predict pixels of the background. Therefore, instead of indicating all motion vectors in the encoded data, it is more efficient to indicate groups of motion vectors to reduce the amount of data representing the encoded video. This can be helpful in indicating representations of motion vector fields that require less data.

[0302] Figure 9 The diagram illustrates a bitstream 930 generated by the encoder described above, received at the decoding end. At the decoding end, the decoder portion of system 900 includes signal feed logic 940, which in some embodiments interprets segmentation information obtained from bitstream 930. Based on the segmentation information, signal feed logic 940 identifies a specific (selected) layer, spatial size (resolution), and location of a feature map portion, and the corresponding selected information (also obtained from the bitstream) should be placed within this information.

[0303] It should be noted that in some embodiments, the segmentation information is not necessarily processed by the cascaded network. The segmentation information can be provided independently or derived from other parameters in the bitstream. In other embodiments, the feature data is not necessarily processed in the cascaded network, but the segmentation information is processed in the cascaded network. Therefore, the sections "Decoding using Feature Information" and "Decoding using Segmentation Information" describe examples of these embodiments, as well as combinations of these embodiments.

[0304] It should be noted that, for the two-part embodiments, the above-described modifications to the encoding end (first to seventh modifications) are correspondingly applied to the decoding end. For ease of understanding, other modified features will not be repeated in both parts. However, as will be apparent to those skilled in the art, they can be applied alternatively or in combination to the decoding methods of both parts.

[0305] Decoding using feature information In this embodiment of the invention, a method for decoding data for image or video processing from a bitstream is provided, such as... Figure 33 As shown. Accordingly, an apparatus is provided for decoding data from a bitstream for image or video processing. The apparatus may include processing circuitry for performing the method steps.

[0306] The method includes obtaining two or more feature map element sets from a bitstream, wherein each feature map element set is associated with a (respective) feature map. The acquisition can be performed by parsing the bitstream. In some exemplary implementations, bitstream parsing may also include entropy decoding. This invention is not limited to any particular method of obtaining data from a bitstream.

[0307] The method further includes the step of inputting each feature map element set from two or more feature map element sets into two or more feature map processing layers in a plurality of cascaded layers.

[0308] Cascaded layers can form part of a processing network. In this invention, the term "cascade" means that the output of one layer is later processed by another layer. Cascaded layers do not need to be closely adjacent (the output of one cascaded layer directly enters the input of the second cascaded layer). When referring to... Figure 9 At this time, data from the bitstream 930 is input to the signal feed logic 940, which feeds the feature map element set to the appropriate layers (indicated by arrows) 953, 952, and / or 951. For example, the first feature element set is inserted into the first layer 953 (the first in the processing sequence), and the second feature element set is inserted into the third layer 951. It is not necessary to insert a set in the second layer. The number and position of layers (within the processing order) can vary, and the invention is not limited to any particular number and position.

[0309] The method also includes acquiring the decoded data (3330) for image or video processing as the processing result of multiple cascaded layers. For example, the first set is the set of latent feature map elements processed by all layers of the network. The second set is an additional set provided to another layer. When referencing... Figure 9 At that time, the decoded data 911 was obtained after the first set was processed by three layers 953, 952 and 951 (in this order).

[0310] In an exemplary implementation, feature maps are processed in each of two or more feature map processing layers, wherein the resolutions of the feature maps processed in the two or more feature map processing layers are different. For example, the resolution of the first feature map processed by the first layer is different from the resolution of the second feature map processed by the second layer.

[0311] Specifically, the processing of feature maps in two or more feature map processing layers includes upsampling. Figure 9 The decoding section is shown to consist of a network of three (directly) cascaded upsampling layers 953, 952, and 951.

[0312] In an exemplary implementation, the decoder includes only upsampling layers with different spatial resolutions, and the nearest neighbor method is used for upsampling. The nearest neighbor method repeats lower-resolution values ​​in higher-resolution regions corresponding to a given shape. For example, if a lower-resolution element corresponds to four higher-resolution elements, the value of that element is repeated four times in the higher-resolution region. In this context, the term "corresponds" refers to the same region describing the highest-resolution data (initial feature map, initial data). This upsampling method facilitates the transfer of information from lower-resolution layers to higher-resolution layers without modification. This approach can be applied to various types of data, such as logical flags or indicator information, or information where it is desirable to maintain the same information as acquired at the encoding end, such as modifications to some convolutional layers. An example of such data is prediction information, which may include, for example, motion information of motion vectors estimated at the encoding end, a reference index indicating which specific image from the reference image set should be used, a prediction pattern indicating whether a single or multiple reference frames are used, or a combination of different predictions (such as inter-frame and intra-frame joint prediction), the presence or absence of residual information, etc.

[0313] However, this invention is not limited to upsampling performed via the nearest neighbor method. Alternatively, upsampling can be performed by applying some interpolation or extrapolation, or by applying convolution, etc. These methods may be particularly suitable for upsampling data that is expected to have smooth characteristics (e.g., motion vectors or residuals or other sample-related data).

[0314] exist Figure 9 In this implementation, the encoder (e.g., symbols 911 to 920) and decoder (e.g., symbols 940 to 951) have the same number of downsampling and upsampling layers, respectively. Nearest neighbor methods can be used for upsampling, and average pooling can be used for downsampling. The shape and size of the pooling layers are aligned with the scaling factor of the upsampling layers. In some other possible implementations, an alternative pooling method, such as max pooling, can be used.

[0315] As illustrated in several encoder embodiments, the data used for image or video processing may include motion vector fields. For example, Figure 12The encoding and decoding ends are shown. At the decoding end, the bitstream 1250 is parsed, and motion information 1260 (which may include segmentation information discussed below) is obtained from the bitstream. The obtained motion information is then provided to a motion generation network 1270. The motion generation network can increase the resolution of the motion information, i.e., dense motion information. The reconstructed motion vector field (e.g., dense optical flow) 1275 is then provided to a motion compensation unit 1280. The motion compensation unit 1280 uses the reconstructed motion vector field to obtain predicted image / video data based on a reference frame and reconstructs a motion-compensated frame based on the reference frame (e.g., by adding decoding residuals, such as...). Figure 5A As shown, the decoder section of the encoder, or Figure 7B (Middle, Reconstruction Unit 314).

[0316] Figure 13 Also shown is a motion generation (dense) network 1360 at the decoding end. Network 1360 includes signal feed logic 1370, which functions similarly to... Figure 9 The signal feed logic 940 functions, and includes three upsampling (processing) layers. (Refer to the above reference.) Figure 9 The main difference in the described embodiments is that, Figure 13 In this context, Network 1360 is specifically used for motion vector information processing, outputting a motion vector field.

[0317] As described above, according to one embodiment, the method further includes obtaining segmentation information associated with two or more layers from the bitstream. Then, feature map elements are obtained from the bitstream based on the segmentation information. Based on the segmentation information, the set of feature map elements is input into two or more feature map processing layers respectively. The following sections related to decoding using segmentation information provide some detailed examples related to the use of segmentation information in parsing and processing. For example, Figure 28 and Figure 29 Very specific (and merely exemplary) layer processing options are provided.

[0318] In some embodiments, the cascaded layers further include multiple segmentation information processing layers. The method also includes processing segmentation information in the multiple segmentation information processing layers. For example, processing segmentation information in at least one of the multiple segmentation information processing layers includes upsampling. In some embodiments, such upsampling of segmentation information and / or upsampling of feature maps includes nearest-neighbor upsampling. Typically, the upsampling applied to feature map information and the upsampling applied to segmentation information can be different. Furthermore, upsampling within the same network can be different, so a network (segmentation information processing or feature map processing) can include different types of upsampling layers. For example, in Figure 20 or Figure 21The example shown is an illustration of this. It should be noted that upsampling types other than nearest neighbor can include interpolation methods such as polynomial methods, such as bilinear, cubic, etc.

[0319] According to an exemplary implementation, the upsampling of the segmentation information and / or the upsampling of the feature map includes (transposed) convolution. This corresponds to the first modification of the encoder discussed above. Figure 18 The motion generation unit 1869 is shown at the decoding end, including convolution operations. Instead of nearest-neighbor upsampling, the motion generation unit can perform a learnable upsampling process, which, for example, helps find the optimal upsampling transform in motion-intensive cases, and can reduce block artifacts caused by motion compensation by using block-averaged motion vector information as described in the reference encoder above. The same applies to texture restoration processes, for example, for generating original image intensity values ​​or prediction residuals. The motion generation unit 1869 also includes signal feed logic, the function of which is similar to... Figure 9 Signal feed logic 940 and Figure 13 The signal feed logic has the same function as 1370.

[0320] Figure 30 A block diagram of layer processing according to an exemplary decoding end based on the first modification is shown. Specifically, the bitstream 3030 is parsed, and signal feed logic 3040 (functionally identical to signal feed logic 940 or 1370) provides selection instructions to the convolutional upsampling filter 300. In some embodiments, a convolutional filter can be selected from a set of N filters (denoted as filter 1 to N). The filter can be selected based on information representing the selected filter and parsed from the bitstream. The indication of the selected filter can be provided by the encoder according to optimization methods such as RDO (generated and inserted into the bitstream). Specifically, it can be applied... Figure 17 or Figure 24 The example illustrates RDO (which treats filter size / shape / order as one of the options, i.e., the decoding parameters to be optimized). However, the invention is not limited to this, and generally, filters can be derived based on other indicator parameters (e.g., decoding mode, interpolation direction, etc.).

[0321] In summary, the signal feed logic unit controls the inputs of different layers with different filter shapes and selectively bypasses the output to the next layer based on the segmentation and motion information obtained from the bitstream. The convolution filter unit 3000 corresponds to the convolution performed by one layer. Several such convolutional flood filters can be cascaded, such as... Figure 18 As shown. It should be noted that this invention is not limited to variable or trainable filter settings. Typically, convolutional upsampling can also be performed using fixed convolution operations.

[0322] Aspects of this embodiment can be combined with aspects of other embodiments. For example, an encoder with motion information averaged in downsampling layers can be combined with a decoder including convolutional upsampling layers. The encoder with convolutional layers aims to find better latent representations and can be combined with a motion generation network including nearest-neighbor-based upsampling layers. Other combinations are possible. That is, the implementation of the encoder and decoder does not have to be symmetrical.

[0323] Figure 32A Two examples of reconstruction using the nearest neighbor method are shown. Specifically, Example 1 shows the case where the segmentation information of the lowest resolution layer has a flag set value (1). Accordingly, the motion information represents a motion vector. Since the motion vector is already represented in the lowest resolution layer, there are no other motion vectors or other segmentation information in the bitstream. The network generates motion vector fields with higher resolution (2×2) and highest resolution (4×4) respectively from an indicated motion vector by copying an indicated motion vector during nearest neighbor upsampling. A 4×4 region is obtained where all 16 motion vectors are identical and equal to the indicated motion vector.

[0324] Figure 32B Two examples of reconstruction using convolutional layer-based methods are shown. Example 1 has the same characteristics as... Figure 32A The input is the same as in Example 1. Specifically, the segmentation information from the lowest resolution layer has values ​​of a flag set (which is 1). Correspondingly, the motion information represents a motion vector. However, after applying convolutional layers (which may have been trained), the motion vectors in the higher and highest layers are not exactly the same, rather than simply a copy of a motion vector.

[0325] Similarly, Figure 32A Example 2 shows the segmentation information for 0 in the lowest resolution layer, and the segmentation information 0101 in the next (higher resolution) layer. Accordingly, the two motion vectors representing the positions indicated by the segmentation information are indicated as motion information in the bitstream. These are shown in the intermediate layers. From the bottom layer, it can be seen that the indicated motion vectors are copied, each copied four times to include the highest resolution region. The remaining eight motion vectors of the highest resolution (bottom) layer are indicated in the bitstream. Figure 32B Example 2 applies convolution instead of nearest neighbor copying. Motion vectors are no longer copied. Figure 32A The transitions between copied motion vectors are now smoother, thus reducing block artifacts.

[0326] Similar to the second modification of the encoder discussed above, at the decoding end, multiple cascaded layers include convolutional layers, instead of upsampling between layers with different resolutions. It should be noted that the encoder and decoder are not necessarily symmetrical in this respect: the encoder can have these additional layers while the decoder does not, and vice versa. Of course, the encoder and decoder can also be designed symmetrically, with additional layers between the corresponding downsampling and upsampling layers in the encoder and decoder.

[0327] Regarding the combination of segmentation information processing and feature map processing, the extraction of feature map elements from the bitstream is based on processed segmentation information processed by at least one of multiple segmentation information processing layers. The segmentation layers can parse and interpret the segmentation information, as described in more detail below in the section on using segmentation information for decoding. It should be noted that the embodiments and examples described in this section can be used in conjunction with the embodiments in this section. Specifically, refer to... Figures 26 to 32B The layer processing of the segmentation information described can also be performed in conjunction with the feature map processing described in this paper.

[0328] For example, inputting each set of two or more feature map element sets into two or more feature map processing layers is performed based on processed segmentation information processed by at least one of multiple segmentation information processing layers. The acquired segmentation information is represented by a set of syntax elements, where the position of an element in the set of syntax elements indicates which feature map element position the syntax element is associated with. For example, the set of syntax elements is a portion of the bitstream, which can be binarized using fixed code, entropy code (e.g., variable-length code, or arithmetic code), any of which can be context-adaptive. If the invention has a predefined structure known at both the encoding and decoding ends, the invention is not limited to any particular decoding form of the bitstream. Thus, parsing and processing segmentation information and feature map information can be performed in association. For example, for each syntax element, the processing of the feature map includes: (i) when the syntax element has a first value, parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream, and (ii) otherwise (or more generally, when the syntax element has a second value), not parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream. Syntax elements can be binary tokens, which are arranged in order in the bitstream at the encoding end and parsed out from the decoder in the correct order by the specific layer structure of the processing network.

[0329] It should be noted that options (i) and (ii) can also be provided to unbinded syntax elements. In this case, the first value indicates parsing, and the second value indicates no parsing. Syntax elements can take some other values ​​besides the first and second values. These can also result in parsing or no parsing, or can represent a specific type of parsing, etc. The number of parsed feature map elements can correspond to the number of syntax elements equal to the first value.

[0330] According to an exemplary implementation, the processing of the feature map by each of the multiple (N) feature map processing layers (1 < j < N) further includes: parsing the segmentation information elements of the j-th feature map processing layer from the bitstream; obtaining the feature map processed by the previous feature map processing layer; and parsing the feature map elements from the bitstream and associating the parsed feature map elements with the obtained feature map, wherein the position of the feature map element in the processed feature map is represented by the parsed segmentation information elements, which are processed by the previous segmentation information processing layer. For example, the association can be a replacement or combination of previously processed feature map elements, such as addition, subtraction, or multiplication. Some exemplary implementations are provided below. Parsing can depend on the previously processed segmentation information and can provide a very compact and efficient syntax.

[0331] For example, the method may include: parsing elements of the feature map from the bitstream when the syntax element has a first value; and not parsing elements of the feature map from the bitstream when the syntax element has a second value or when the segmentation information processed by the previous segmentation information processing layer has a first value. That is, if the relevant part is parsed in the previous layer, then no parsing is performed. For example, the syntax element parsed from the bitstream representing segmentation information is a binarization flag. As mentioned above, it is beneficial for the processed segmentation information to be represented by a set of binarization flags. The set of binarization flags is a sequence of binarization flags, each with a value of 1 or 0 (corresponding to the first and second values ​​mentioned above).

[0332] In some embodiments, upsampling of the segmentation information in each segmentation information processing layer j further includes: for each p-th position in the acquired feature map represented by the input segmentation information, determining an indication of the feature map position included in the region corresponding to the p-th position in the reconstructed image, as the upsampled segmentation information. This provides a spatial relationship between the reconstructed image (or reconstructed feature map or general data), the position in the subsampled feature map, and the corresponding segmentation marker.

[0333] As described above, and in the encoder embodiment, the data used for image or video processing may include image data (e.g., image samples) and / or prediction residual data and / or prediction information data. When referring to "residual" in this invention, it should be noted that these residuals may be pixel-domain residuals or transform (spectral) coefficients (i.e., transform residuals, residuals represented in a domain different from the sample / pixel domain).

[0334] Similar to the fourth modification described above for the encoder, in the exemplary implementation, the filter is used for upsampling of the feature map, and the shape of the filter can be any of a square, a horizontal rectangle, or a vertical rectangle. It should be noted that the shape of the filter can be... Figure 25 The shapes of the blocks shown are similar.

[0335] Figure 20 The diagram illustrates layer processing at an exemplary decoding end. The motion generation network (unit) 2060 includes signal feed logic and one or more (here, two) upsampling layers using filters (upsampling filters), which can be selected from a predetermined or predefined set of filters. Selection can be performed at the encoding end, for example, via RDO or other settings, and indicated in the bitstream. At the decoding end, the filter selection indication is parsed from the bitstream and applied. Alternatively, filters can be selected at the decoding end without indicating the filters using other decoding parameters derived from the bitstream. These parameters can be any content-related parameters, such as prediction type, direction, motion information, residuals, loop filtering characteristics, etc.

[0336] Figure 31 A block diagram of an upsampling filter unit 3100 supporting selection of one filter from N filters 1 to N is shown. The filter selection indication can directly include the index of one of the N filters. The index may include filter direction, filter order, filter shape, and / or coefficients. At the decoding end, the signal feed logic interprets the filter selection flag (e.g., a direction flag to distinguish between vertical and horizontal filters, or other directions) and feeds the feature map values ​​to a layer with the corresponding set of filter shapes. Figure 31 In this context, a direct connection from the signal feed logic to the selective bypass logic can be made without selecting any filters. The corresponding values ​​of the filter selection indicators can also be indicated or derived from the bitstream.

[0337] Typically, feature map upsampling uses filters, and the input information from the bitstream also includes: information representing the filter shape and / or filter orientation and / or filter coefficients from the bitstream. There can be implementations where each layer has a set of filters to select, or where each layer is a filter, and the signal feed logic determines which layers to select and which to bypass based on filter selection flags (indicators).

[0338] In some embodiments, a flexible filter shape can be provided, wherein the information representing the filter shape represents a mask composed of flags, and the mask represents the filter shape because a flag with a third value indicates a non-zero filter coefficient, and a flag with a fourth value different from the third value indicates that the filter coefficient is 0. That is, as already described for the encoding end, the filter shape can be defined by representing the positions of non-zero coefficients. Non-zero coefficients can be derived according to predefined rules or indicated.

[0339] The decoder embodiments described above can be implemented as a computer program product stored in a non-transient medium, which, when executed on one or more processors, performs the steps of any of the methods described above. Similarly, the decoder embodiments described above can be implemented as an apparatus for decoding images or videos, including processing circuitry for performing the steps of any of the methods described above. Specifically, an apparatus for decoding data for image or video processing from a bitstream can be provided, the apparatus comprising: an acquisition unit for acquiring two or more feature map element sets from the bitstream, wherein each feature map element set is associated with a feature map; an input unit for inputting each feature map element set from the two or more feature map element sets into two or more feature map processing layers in a plurality of cascaded layers; and a decoded data acquisition unit for acquiring the decoded data for image or video processing as the processing result of the plurality of cascaded layers. These units can be implemented in software or hardware or a combination of both, as discussed in more detail below.

[0340] Decoding using segmentation information At the receiving end, the decoder in this embodiment performs parsing and decoding of the segmented information. Therefore, as Figure 34 As shown, a method for decoding data for image or video processing from a bitstream is provided. Correspondingly, an apparatus for decoding data for image or video processing from a bitstream is provided. The apparatus may include processing circuitry for performing the method steps.

[0341] The method includes obtaining two or more segmentation information element sets 3410 from the bitstream. This acquisition can be performed by parsing the bitstream. In some exemplary implementations, bitstream parsing may also include entropy decoding. This invention is not limited to any particular method of obtaining data from the bitstream. The method also includes inputting each of the two or more segmentation information element sets 3420 into two or more segmentation information processing layers in a plurality of cascaded layers. It should be noted that the segmentation information processing layer can be the same layer as the feature map processing layer or a different layer. That is, a layer can have one or more functions.

[0342] Furthermore, in each of two or more segmentation information processing layers, the method includes processing a corresponding set of segmentation information. The decoded data 3430 used for image or video processing is obtained based on segmentation information processed by multiple cascaded layers.

[0343] Figure 26 Exemplary segmentation information for three-layer decoding is shown. The segmentation information can be viewed as a selection (see encoder description) of the layers whose feature map elements will be parsed or otherwise acquired. Feature map element 2610 is not selected. Therefore, the encoder sets flag 2611 to 0. That is, the bitstream does not include feature map element 2610 with the lowest resolution. However, the bitstream includes flag 2611 indicating that a feature map element is not selected. For example, if the feature map element is a motion vector, the motion vector 2610 of the largest block is not selected and is not included in the bitstream.

[0344] exist Figure 26 In the example shown, in feature map 2620, three feature map elements (represented by flags 2621, 2622, and 2624) are selected from the four feature map elements used to determine the feature map elements of feature map 2610 for indication, while one feature map element 2623 is not selected. In the example using motion vectors, three motion vectors are selected from feature map 2620 and their respective flags are set to 1, while no feature map element is selected and its respective flag 2623 is set to 0.

[0345] The bitstream can then include all four flags 2621 to 2624 and three selected motion vectors. Typically, the bitstream can include four flags 2621 to 2624 and three selected feature map elements. In feature map 2630, one or more elements that determine the unselected feature map elements of feature map 2620 can be selected.

[0346] In this example, when selecting feature map elements, no elements of the higher resolution feature map are selected. In this example, no feature map elements of feature map 2630, used to determine the feature map elements represented by flags 2621, 2622, and 2624, are selected. In one embodiment, none of the flags for these feature map elements are included in the bitstream. Instead, the bitstream includes only the flags of the feature map elements of feature map 2630, which identify the feature map element with flag 2623.

[0347] In the example where the feature map elements are motion vectors, feature map elements 2621, 2622, and 2624 can be determined by each group of four motion vectors in feature map 2630. In each group that determines the motion vectors with flags 2621, 2622, and 2624, the motion vectors can have more similarity to each other than the four motion vectors (feature map elements) that determine the motion vectors in feature map 2620 in the unselected feature map 2630 (indicated by flag 2623).

[0348] Figure 26 This is described above through the characteristics of the bitstream. It should be noted that the decoder accordingly decodes (parses) these bitstreams: the decoder determines which information is included (indicates) based on the values ​​of the flags as described above, and parses / interprets the parsed information accordingly.

[0349] In an exemplary implementation, the segmented information is organized as follows: Figure 27 As shown. For 2D information such as images or videos that are considered as image sequences, feature maps of some layers can be represented in two-dimensional space. Segmentation information includes indicators (binarization flags) for 2D spatial locations, indicating whether the feature map value corresponding to that location is presented in the bitstream.

[0350] exist Figure 27 In this context, there exists a starting layer (layer 0) for decoding the segmentation information, for example, the lowest resolution layer, i.e., the latent representation layer. For this starting layer, each 2D location includes a binarized flag. If this flag equals 1, the selected information includes the feature map value of this location on this particular layer. On the other hand, if this flag equals 0, there is no information for this location on this particular layer. This set of flags (or typically a flag tensor, here a flag matrix) is called TakeFromCurrent. The TakeFromCurrent tensor is upsampled to the next layer resolution, for example, using a nearest neighbor method. Let's denote this tensor as TakeFromPrev. The flags in this tensor indicate whether the corresponding sample location was filled in the previous layer (here, layer 0).

[0351] As a next step, the signal feed logic reads the LayerFlag of the current resolution layer position. In this exemplary implementation, it only indicates positions that were not filled in the previous layer (not set to 1, meaning no feature map element values ​​were filled). Using logical operations, this can be represented as TakeFromPrev==0 or !TakeFromPrev==1, where "!" represents the logical NOT operation (inversion).

[0352] The required flags for this layer can be calculated as the number of 0 (logical false) elements in the TakeFromPrev tensor, or the number of 1 (logical true) elements in the inverted (!TakeFromPrev) tensor. No flags are needed in the bitstream for non-zero elements in the TakeFromPrev tensor. This is indicated in the diagram by displaying "" at locations where reading is not required. This is represented by "". From an implementation perspective, it might be easier to compute the sum of the elements on the inverted tensor as sum(!TakeFromPrev). The signal feed logic can use this algorithm to identify how many flags need to be parsed from the bitstream. The read flag is placed where !TakeFromPrev is equal to 1 to obtain the LayerFlag tensor. Then, by keeping the flag at the position read from the bitstream of the current resolution layer, and marking the previous resolution layer (LayerFlag) as "", the signal feed logic can be used to determine the layer flags. The value at the position read from the previous resolution layer (here, layer 1) is zeroed, and the TakeFromCurrent tensor is obtained as a combination of the TakeFromPrev and LayerFlag tensors. This can be represented and implemented using the logical AND operator, as follows: TakeFromCurrent = !TakeFromPrev AND LayerFlag. Then, to take into account the position read from the previous resolution layer, the TakeFromCurrent tensor is obtained using logical OR operations such as TakeFromCurrent = TakeFromCurrent OR TakeFromPrev. It should be understood that Boolean operations can be implemented using regular mathematical operations, such as AND multiplication and OR summation. This will provide the benefit of preserving and transferring gradients, and the above method can be used in end-to-end training.

[0353] Then, the acquired TakeFromCurrent tensor is upsampled to the next resolution layer (here, the 2nd layer), and the above operation is repeated.

[0354] For the sake of generality and simplification of implementation, it is beneficial to unify the processing of all resolution layers without specifically considering the first resolution layer, where all flags are parsed from the bitstream for the first resolution layer. This can be achieved by initializing TakeFromPrev to 0 before processing in the first (lower resolution) layer (layer 0) and repeating the above steps for each resolution layer.

[0355] To further reduce indication overhead, in some other implementations, the LayerFlags of the last resolution layer (here, the third layer, i.e., the second layer) do not need to be transmitted (including at the encoder and parsed at the decoder) to the bitstream. That is, for the last resolution layer, the feature map values ​​serve as selection information for all locations in the last resolution layer (see...). Figure 11 The positions 1120 in the bitstream are not acquired in (all) previous resolution layers (or any previous resolution layers). That is, for the last resolution layer, TakeFromCurrent = !TakeFromPrev, meaning TakeFromCurrent corresponds to the inverted TakeFromPrev. Similarly, to maintain the generality of the last resolution layer processing, LayerFlag can be initialized to 1 and can use the same formula: TakeFromCurrent = !TakeFromPrev AND LayerFlag.

[0356] In some other possible implementations, the last resolution layer has the same resolution as the original image. If the last resolution layer has no further processing steps, some values ​​of the original tensor are transmitted without compression by the autoencoder.

[0357] Below, for reference Figure 28 An example describing the signal feed logic 2800. In Figure 28 In this process, the decoder's signal feed logic 2800 uses segmentation information (LayerFlag) to obtain and utilize selected information (LayerMv) transmitted in the bitstream. Specifically, at each layer, the bitstream is parsed to obtain the segmentation information (LayerFlag) in the corresponding syntax decoding units 2823, 2822, and 2821, as well as the possible selected information (LayerMv) (in this order). As mentioned above, to enable the same operation as other layers in the first layer (syntax decoding 2823), the TakeFromPrev tensor is initialized to 0 in 2820. The TakeFromPrev tensor is propagated in the processing order from the syntax decoding of earlier layers (e.g., from 2823) to the syntax decoding of later layers (e.g., 2822). This propagation includes an upsampling of 2, as referenced above. Figure 27 The explanation given.

[0358] When interpreting the segmentation information (LayerFlag) for each resolution layer, a tensor TakeFromCurrent is obtained (generated). This tensor TakeFromCurrent includes flags indicating whether feature map information (LayerMv) exists in the bitstream at each specific location of the current resolution layer. The decoder reads the values ​​of the feature map LayerMv from the bitstream and places these values ​​in the positions where the flags in the TakeFromCurrent tensor are equal to 1. The total number of feature map values ​​included in the bitstream of the current resolution layer can be calculated either based on the number of non-zero elements in TakeFromCurrent or as sum(TakeFromCurrent) (the sum of all elements in the TakeFromCurrent tensor). In the next step, the tensors in each layer (e.g., in 2812) are combined by combining the outputs of the previous resolution layers (e.g., generated by 2813 and upsampled by 2801 to match the outputs of subsequent layer processing 2812) in a manner that replaces the values ​​of the feature maps at the locations where the TakeFromCurrent tensor is equal to 1 with the feature map values ​​transmitted in the bitstream as selected information (LayerMv). As described above, in order to enable the same operation as other layers in the first layer (tensor combination 2813), the combined tensor is initialized to 0 in 2810. After processing the LayerFlags from all layers and generating the output tensor of the last layer (in 2811), the combined tensor is upsampled by 4 in 2801 to obtain the original size of the dense optical flow, i.e., W×H.

[0359] Figure 28 The exemplary implementation provides a fully parallelizable scheme that can run on GPUs / NPUs and leverage parallelism. This fully trainable scheme for transporting gradients can be used in end-to-end trainable video decoding solutions.

[0360] Figure 29 Another possible and exemplary implementation of the signal feed logic 2900 is shown. This implementation generates a LayerIdx tensor ( Figure 29The layer Idx (referred to as LayerIdxUp) contains indices for different resolution layers, indicating which layer should be used to capture motion information transmitted in the bitstream (including in the encoder and parsing at the decoder). In each parsing block (2923, 2922, 2923), the LayerIdx tensor is updated by multiplying the TakeFromCurrent tensor by the upsampling layer index numbered from highest to lowest resolution. The LayerIdx tensor is then upsampled and passed (transferred) to the next layer in processing order, e.g., from 2923 to 2922, from 2922 to 2921. To ensure similar processing across all layers, the LayerIdx tensor is initialized to 0 in 2920 and passed to the first layer's parsing block 2923.

[0361] After the last layer (here, the third layer), the LayerIdx tensor is upsampled to the original resolution (upsampled from 4 to 2995). Therefore, each location in LayerIdx includes the index of the layer from which motion information is to be extracted. The location of LayerIdx corresponds to the original resolution of the feature map data (here, the dense optical flow), which is 2D (matrix) in this example, at the same resolution. Thus, for each location in the reconstructed optical flow, LayerIdx represents the location from which motion information is extracted (which layer's MayerMV).

[0362] Motion information (LayerMv, in) Figure 29The layer (also known as LayerMvUp) is generated as follows. At each spatial resolution layer, tensor combination blocks (2913, 2912, 2911) combine the LayerMv obtained from the bitstream (after corresponding syntax interpretation units 2923, 2922, 2921) with intermediate tensors and intermediate TakeFromCurrent boolean tensors, according to the method described above and based on the segmentation information (LayerFlag) obtained from the bitstream. Intermediate tensors can be initialized to 0 (see initialization units 2910, 2919, 2918) or to any other value. The initialization values ​​are not important because, ultimately, after all steps are completed, these values ​​will not be selected for dense optical flow reconstruction 2990 according to this method. The combined tensors, including motion information (from each of the outputs in 2913, 2912, 2911), are upsampled and concatenated with the combined tensors from the previous spatial resolution layer (2902, 2901). The concatenation process is performed along additional dimensions corresponding to the motion information obtained from layers of different resolutions (i.e., the 2D tensor before concatenation 2902 becomes a 3D tensor after concatenation; the 3D tensor before concatenation 2901 remains a 43D tensor after concatenation, but its size increases). Finally, after completing all upsampling steps in LayerIdxUp and LayerMvUp, motion information is selected from LayerMvUp using the value of LayerIdxUp as an index on the central axis of LayerMvUp to obtain the reconstructed dense optical flow, where the axis is the dimension added during the LayerMvUp concatenation step. That is, the dimension added in LayerMvUp is the layer-by-layer dimension, and LayerIdxUp selects the appropriate layer for each location.

[0363] The specific exemplary implementations described above do not limit the invention. Typically, segmentation can be performed and indicated in a variety of conceivable ways within the bitstream. Generally, obtaining the set of segmentation information elements is based on segmentation information processed by at least one segmentation information processing layer among multiple cascaded layers. For example... Figure 28 As shown, such a layer may include syntax interpretation units (2823, 2822, 2821), which parse / interpret the meaning (semantics) of the parsed segmentation information LayerFlag.

[0364] More specifically, the input segmentation information element set is based on processed segmentation information output from at least one of multiple cascaded layers. For example, in Figure 28 This is illustrated by passing the TakeFromPrev tensor between the syntax interpretation units (2823, 2822, 2821). As explained in the description at the encoding end, in some exemplary implementations, the resolution of the segmentation information processed in two or more segmentation information processing layers is different.

[0365] Furthermore, the processing of segmentation information in two or more segmentation information processing layers includes upsampling, such as referencing... Figure 9 , Figure 13 Examples of other diagrams illustrate this. For instance, the upsampling of segmentation information includes nearest-neighbor upsampling. It should be noted that, in this embodiment and in the embodiments described above, the invention is not limited by the application of nearest-neighbor upsampling. Upsampling can include interpolation, rather than simply copying neighboring sample (element) values. Interpolation can be any known interpolation, such as linear or polynomial interpolation, such as cubic upsampling, etc. Regarding copying, it should be noted that the copying performed by nearest neighbors is copying element values ​​from predefined (available) nearest neighbors (e.g., top or left). If there is a neighbor at the same distance as the location to be filled, it may be necessary to copy the predefined neighbor.

[0366] As described above regarding the first modification, in some exemplary implementations, the upsampling includes transposed convolution. Besides, or alternatively, applying convolutional upsampling to feature map information, convolutional upsampling can also be applied to segmentation information. It should be noted that the type of upsampling performed on segmentation information is not necessarily the same type of upsampling applied to feature map elements.

[0367] Typically, for each segmentation information processing layer j in multiple (N) segmentation information processing layers in a cascaded layer, the input includes: If j=1, the initial segmentation information is input from the bitstream (and / or based on initialization, e.g., initialized to 0 in 2820); otherwise, the segmentation information is input by the (j–1)th segmentation information processing layer. Output the processed segmented information.

[0368] This is segmentation information related to the input layer, and may not (but still could) come from the entire segmentation information of the bitstream. The upsampled segmentation information in the j-th layer is the upsampled segmentation information in the j-th layer, i.e., output by the j-th layer. Typically, the processing of the segmentation layer includes upsampling (TakeFromPrev) and including new elements from the bitstream (LayerFlag).

[0369] For example, the processing of input segmentation information by each layer j < N in multiple (N) segmentation information processing layers also includes parsing segmentation information elements (LayerFlag) from the bitstream, and associating the parsed segmentation information elements with the segmentation information output by the previous layer (TakeFromPrev) (e.g., in...). Figure 28 (In the syntax interpretation unit 282x). The position of the parsed segmentation information element (LayerFlag) in the associated segmentation information is determined based on the segmentation information output by the previous layer. As in Figure 28and Figure 29 As can be seen, there are various ways to associate and propagate location information. This invention is not limited to any particular implementation.

[0370] For example, the number of segmentation information elements to be parsed from the bitstream is determined based on the segmentation information output from the previous layer. Specifically, if some regions have already been covered by the segmentation information from the previous layer, it is not necessary to cover them again in subsequent layers. It should be noted that this design provides an efficient parsing method. Each position in the reconstructed feature map data corresponding to the position of the reconstructed segmentation information is associated only with the segmentation information belonging to a single layer (in N processing layers). Therefore, there is no information duplication. However, the invention is not limited to this method. It is conceivable that segmentation information is repeated, even if this may lead to some redundant information.

[0371] like Figure 27 As shown in the diagram, in some embodiments, the parsed segmented information elements are represented by a set of binary flags. The order of the flags in the bitstream (syntax) can convey the association between the flags and their respective layers. The order (sequence) can be given by a predefined processing order at the encoding end, and correspondingly, by a predefined processing order at the decoding end. For example, Figure 16 and Figure 28 The example provided illustrates this.

[0372] In some exemplary embodiments, such as those mentioned above with reference to the seventh modification, obtaining decoded data for image or video processing includes determining at least one of the following parameters based on segmentation information. The segmentation information can be used to determine the parsing of additional information such as decoding parameters in a manner similar to motion information. These decoding parameters may include: intra-frame or inter-frame prediction mode; image reference index; single-reference or multi-reference prediction (including bidirectional prediction); presence or absence of prediction residual information; quantization step size; motion information prediction type; motion vector length; motion vector resolution; motion vector prediction index; motion vector difference magnitude; motion vector difference resolution; motion interpolation filter; in-loop filter parameters; and / or post-filter parameters, etc. That is, when processed by the segmentation information processing layer, the segmentation information can indicate which processing layer can obtain the decoding parameters from. For example, in... Figure 22 or Figure 23 In the encoder method described above, the reconstruction (decoding) parameters can be received from the bitstream, rather than (or except) from the motion information (LayerMv). These reconstruction (decoding) parameters, blk_rec_params, can be used at the decoding end in conjunction with... Figure 28 and Figure 29 The motion information shown is parsed in the same way.

[0373] Typically, segmentation information is used for parsing and inputting feature map elements (motion information or any of the aforementioned reconstruction parameters or sample-related data). The method may further include obtaining a set of feature map elements from the bitstream and inputting the set of feature map elements into feature map processing layers in multiple layers, based on the segmentation information processed by the segmentation information processing layer. Furthermore, the method also includes obtaining decoded data for image or video processing based on the feature maps processed by multiple cascaded layers. Specifically, in some embodiments, at least one of the multiple cascaded layers is a segmentation information processing layer and a feature map processing layer. As mentioned above, the network can be designed with separate segmentation information processing layers and feature map processing layers, or with a combination of layers with both functions. In some implementations, each of the multiple layers is either a segmentation information processing layer or a feature map processing layer.

[0374] The methods described above can be embodied as a computer program product stored in a non-transient medium, which, when executed on one or more processors, causes the processors to perform the steps of any of these methods. Similarly, an apparatus for decoding images or videos is provided, including processing circuitry for performing the method steps of any of the methods discussed above. The functional structure of the apparatus provided by the present invention can correspond to the functions provided by the above embodiments and steps. For example, an apparatus for decoding data for image or video processing from a bitstream is provided, wherein the apparatus includes: an acquisition unit for acquiring two or more sets of segmentation information elements from the bitstream; an input unit for inputting each of the two or more sets of segmentation information elements into two or more segmentation information processing layers in a plurality of cascaded layers; a processing unit for processing a respective set of segmentation information in each of the two or more segmentation information processing layers; and a decoded data acquisition unit for acquiring the decoded data for image or video processing based on the segmentation information processed in the plurality of cascaded layers. These units and other units can perform all the functions of the methods described above.

[0375] Brief overview of some embodiments Examples of encoding using feature information or segmentation information According to one aspect of the present invention, a method is provided for encoding data for image or video processing into a bitstream. The method includes: processing the data, the processing including generating feature maps in a plurality of cascaded layers, each feature map including its own resolution, wherein at least two feature maps in the generated feature maps have different resolutions; selecting from the plurality of layers a layer different from the layer that generated the lowest resolution feature map; and generating the bitstream, including inserting information associated with the selected layer into the bitstream.

[0376] This method can improve the efficiency of this encoding because it can encode data from different layers, thereby including features of different resolutions or other types of layer-related information in the bitstream.

[0377] According to one aspect of the present invention, an apparatus is provided for encoding data for image or video processing into a bitstream. The apparatus includes: a processing unit for processing the data, wherein the processing includes generating feature maps with mutually different resolutions in a plurality of cascaded layers, each feature map including its own resolution; a selection unit for selecting from the plurality of layers a layer different from the layer that generated the lowest resolution feature map; and a generation unit for generating the bitstream, including inserting an indication of data associated with the selected layer into the bitstream. The processing unit, the selection unit, and the generation unit may be implemented by processing circuitry, for example, one or more processors or any combination of software and hardware.

[0378] This device can improve decoding efficiency because it can decode data from different layers and use it for reconstruction, thereby enabling the use of features or other types of layer-related information at different resolutions.

[0379] In an exemplary implementation, the processing further includes downsampling by one or more cascaded layers. Applying downsampling reduces both the complexity of the processing and the amount of data provided in the bitstream. Furthermore, layers processing different resolutions can thus focus on features at different scales. Therefore, networks processing images (still or video) can operate efficiently.

[0380] For example, one or more downsampling layers include average pooling or max pooling for downsampling. Average pooling and max pooling operations are part of several frameworks that provide efficient methods for downsampling with low complexity.

[0381] In another example, convolution is used for downsampling. Convolution can provide more sophisticated downsampling methods, allowing for the appropriate selection of kernels for specific applications, and even the training of kernels. This enables learnable downsampling processes to find more suitable latent representations of motion information while maintaining the advantages of representing and transmitting information at different spatial resolutions, thereby improving adaptability.

[0382] In one exemplary implementation, the information associated with the selected layer includes elements of the layer's feature map.

[0383] By providing features with different resolutions, the scalability of encoding / decoding is improved, and the resulting bitstream can offer greater flexibility to meet optimization criteria such as rate, distortion, and complexity, ultimately improving decoding efficiency.

[0384] In any of the above examples, for example, the information associated with the selected layer includes information indicating from which layer and / or from which part of the feature map of that layer the elements of that layer were selected.

[0385] Signal segmentation information can efficiently decode feature maps from different layers, allowing each region of the original (to-be-decoded) feature map (data) to be included in the information of only one layer. While this is not a limitation of the invention, in some cases, the inter-layer information of specific regions in the feature map (data) to be encoded in the invention may have some overlap.

[0386] In an exemplary implementation, the above method includes a step of acquiring data to be encoded, wherein the processing of the data to be encoded includes processing by each j-th layer in the plurality (N) cascaded layers, including: if j=1, acquiring the data to be encoded as layer input, otherwise acquiring the feature map processed by the (j-1)-th layer as layer input; processing the acquired layer input, the processing including downsampling; and outputting the downsampled feature map.

[0387] Accordingly, the above-described apparatus has a processing unit in an exemplary implementation, wherein the processing unit is used to acquire the data to be encoded and to perform processing on the data to be encoded, including: the processing of each j-th layer of the plurality (N) cascaded layers includes: if j=1, acquiring the data to be encoded as the layer input, otherwise acquiring the feature map processed by the (j-1)-th layer as the layer input; processing the acquired layer input, the processing including downsampling; and outputting the downsampled feature map.

[0388] The method according to any of the above examples includes: in some embodiments, selecting information for insertion into the bitstream, the information being associated with a first region in a feature map processed by a layer j > 1, wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j comprising a plurality of elements; when selecting feature maps processed by k layers, not selecting the region corresponding to the first region, wherein k is an integer equal to or greater than 1 and k < j.

[0389] The apparatus according to any of the above examples includes: in some embodiments, the processing circuitry is further configured to select information for insertion into the bitstream, the information being associated with a first region in a feature map processed by a layer j > 1, wherein the first region corresponds to a region in the feature map or initial data to be encoded in a layer less than j comprising a plurality of elements; when selecting feature maps processed by k layers, the region corresponding to the first region is not selected, wherein k is an integer equal to or greater than 1 and k < j.

[0390] This choice in some layers excludes regions of the original feature map included by other layers, making it particularly efficient in terms of decoding overhead.

[0391] In any of the above examples, for instance, the data to be encoded includes image information and / or prediction residual information and / or prediction information.

[0392] Alternatively, the information associated with the selected layer may include prediction information.

[0393] In any of the above examples, for instance, the data associated with the selected layer includes an indication of the position of the feature map element in the feature map of the selected layer.

[0394] This instruction enables the correct association of feature map elements of different resolutions with the input data region.

[0395] In any of the above examples, for instance, the positions of selected and unselected feature map elements are represented by a plurality of binarized flags, which are based on the positions of the flags in the bitstream.

[0396] The binarization flag provides a particularly efficient way to decode the segmented information.

[0397] According to one embodiment, in the above method or apparatus, the processing of the j-th layer among the plurality (N) cascaded layers includes: determining a first cost incurred in reconstructing a portion of a reconstructed image using feature map elements output by the j-th layer; determining a second cost incurred in reconstructing the portion of the reconstructed image using feature map elements output by the (j-1)-th layer; if the first cost is higher than the second cost, then selecting the (j-1)-th layer and selecting information related to the portion in the (j-1)-th layer.

[0398] It provides optimizations, including distortion, to achieve the desired quality in a more efficient way.

[0399] For example, the first and second costs include data volume and / or distortion. By optimizing for both the rate (the amount of data generated by the encoder) and the distortion of the reconstructed image, the requirements of various applications or users can be flexibly met.

[0400] Alternatively, or additionally, the data to be encoded is a motion vector field. The methods described above are suitable for compressing motion vector fields, such as dense optical flow or subsampled optical flow. The application of these methods enables efficient decoding of motion vectors (in terms of rate and distortion or other standards) and can further reduce the bitstream size of the encoded image or video data.

[0401] In some embodiments, the prediction information includes a reference index and / or a prediction pattern. In addition to or alternatively, other information related to the prediction can be processed. The reference index and prediction pattern can be similar to the motion vector field and related to the content of the image; therefore, encoding feature map elements with different resolutions can improve efficiency.

[0402] For example, the data volume includes the amount of data required to transmit data related to the selected layer. Thus, during optimization, the overhead incurred by providing layer-related information that differs from the output layer can be calculated.

[0403] Alternatively, distortion can be calculated by comparing the reconstructed image with the target image. This end-to-end quality comparison ensures that distortion in the reconstructed image is properly accounted for. Therefore, optimization enables the efficient selection of decoding methods and more accurately meets the quality requirements of applications or users.

[0404] In any of the above examples, for instance, the processing includes additional convolutional layers between cascaded layers with different resolutions.

[0405] Providing such an additional layer in the cascaded network can introduce additional processing, such as various types of filtering, to improve the quality or efficiency of decoding.

[0406] According to an exemplary implementation, the processing circuit of the method or apparatus described in the above embodiments includes: when downsampling at a layer, using a first filter to downsample an input feature map to obtain a first feature map, using a second filter to downsample the input feature map to obtain a second feature map, determining a third cost incurred in reconstructing a portion of a reconstructed image using the first feature map, and determining a fourth cost incurred in reconstructing the portion of the reconstructed image using the second feature map; when making a selection, if the third cost is lower than the fourth cost, then selecting the first feature map.

[0407] Applying different downsampling filters can help adapt to the different characteristics of the content.

[0408] For example, the shapes of the first filter and the second filter can be any one of a square, a horizontal rectangle, or a vertical rectangle.

[0409] These filters still have simple shapes, but can be improved in other ways, such as adapting to object boundaries.

[0410] The method steps or the steps performed by the processing circuit of the device may further include: obtaining a mask, wherein the mask consists of flags, the mask represents an arbitrary filter shape, and the first filter or the second filter has an arbitrary filter shape.

[0411] It allows for more flexible design of filters of any shape.

[0412] The method steps or the steps performed by the processing circuitry of the device may further include: processing data associated with the same image that is segmented into blocks of different block sizes and shapes in the different layers, the selection including: selecting a layer based on the computational cost of a predetermined set of decoding modes.

[0413] In some exemplary implementations, the process includes determining the cost of different sets of decoding modes for at least one layer, and selecting one of the sets of decoding modes based on the determined cost.

[0414] Applying optimization to the decoding mode can achieve efficient rate distortion optimization, thereby improving decoding efficiency.

[0415] For example, the indication of the data associated with the selected layer includes the selected set of decoding modes.

[0416] According to one aspect of the invention, a computer program stored in a non-transitory medium is provided, the computer program comprising code that, when executed on one or more processors, performs the steps of any of the methods described above.

[0417] According to one aspect of the present invention, an apparatus for encoding images or videos is provided, including processing circuitry for performing the method according to any of the above examples.

[0418] Any of the above-described devices can be implemented on an integrated chip. This invention can be implemented in hardware (HW) and / or software (SW). Furthermore, hardware-based implementations can be combined with software-based implementations.

[0419] It should be noted that this invention is not limited to a specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0420] For clarity, any of the above embodiments can be combined with any one or more of the other embodiments described above to create a new embodiment within the scope of the present invention.

[0421] Examples of decoding using feature map elements According to one embodiment, a method for decoding from a bitstream for image or video processing is provided, wherein the method includes: obtaining two or more feature map element sets from the bitstream, wherein each feature map element set is associated with a feature map; inputting each feature map element set from the two or more feature map element sets into two or more feature map processing layers in a plurality of cascaded layers; and obtaining the decoded data for image or video processing as the processing result of the plurality of cascaded layers.

[0422] This method can improve efficiency because it enables data from different layers to be used for decoding, thereby parsing features or other types of layer-related information from the bitstream.

[0423] For example, feature maps are processed in each of two or more feature map processing layers, wherein the feature maps processed in the two or more feature map processing layers have different resolutions.

[0424] In some embodiments, the processing of the feature map in two or more feature map processing layers includes upsampling.

[0425] Applying upsampling reduces processing complexity (because the first layer has a lower resolution) and also reduces the amount of data provided in the bitstream and parsed at the decoder. Furthermore, processing layers at different resolutions allows attention to features at different scales. Therefore, networks processing images (static or video) can operate efficiently.

[0426] In an exemplary implementation, the method further includes the following steps: obtaining segmentation information related to two or more layers from the bitstream, wherein obtaining feature map elements from the bitstream is performed based on the segmentation information, and inputting the feature map element set into two or more feature map processing layers respectively is performed based on the segmentation information.

[0427] Using segmentation information allows for efficient decoding of feature maps from different layers, enabling each region (to be reconstructed) to be included in the information of only one layer. While this does not limit the invention, in some cases, the invention can also provide overlap between layers for specific regions in the feature map (data). For example, multiple cascaded layers may also include multiple segmentation information processing layers, and the method further includes processing segmentation information within multiple segmentation information processing layers. This approach provides the possibility of controlling the parsing of feature elements from different layers.

[0428] In some embodiments, processing segmentation information in at least one of a plurality of segmentation information processing layers includes upsampling. A hierarchical structure of segmentation information can provide a small amount of side information in the bitstream to be inserted, thereby improving efficiency and / or reducing processing time.

[0429] For example, this upsampling of segmentation information and / or feature maps includes nearest neighbor upsampling. Nearest neighbor upsampling has low computational complexity and can be easily implemented. Nevertheless, nearest neighbor upsampling is still efficient, especially for logical indicators such as flags.

[0430] In some embodiments and examples, the upsampling of the segmentation information and / or the upsampling of the feature map includes transposed convolution. The use of convolution helps reduce block artifacts and enables trainable schemes, where the upsampling filter is optional.

[0431] In an exemplary implementation, feature map elements are obtained from the bitstream based on processed segmentation information processed by at least one of a plurality of segmentation information processing layers.

[0432] In an exemplary implementation, inputting each set of two or more feature map element sets into two or more feature map processing layers is performed based on processed segmentation information processed by at least one of multiple segmentation information processing layers.

[0433] According to one embodiment, the acquired segmentation information is represented by a set of syntax elements, wherein the position of an element in the set of syntax elements indicates which feature map element position the syntax element is associated with. For each syntax element, the feature map processing includes: when the syntax element has a first value, parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream; otherwise, not parsing the element of the feature map from the bitstream at the position indicated by the position of the syntax element in the bitstream.

[0434] This relationship between segmentation information and feature map information enables efficient decoding of frequency information and allows for parsing within a hierarchical structure by considering different resolutions.

[0435] For example, the processing of feature maps by each of the multiple (N) feature map processing layers, where 1 < j < N, further includes: parsing the segmentation information elements of the j-th feature map processing layer from the bitstream; obtaining the feature map processed by the previous feature map processing layer; parsing the feature map elements from the bitstream and associating the parsed feature map elements with the obtained feature map, wherein the position of the feature map element in the processed feature map is represented by the parsed segmentation information elements, and the segmentation information is processed by the previous segmentation information processing layer.

[0436] Specifically, the method may include: parsing elements of the feature map from the bitstream when the syntax element has a first value; and not parsing elements of the feature map from the bitstream when the syntax element has a second value or when the segmentation information processed by the previous segmentation information processing layer has a first value.

[0437] For example, the syntax elements parsed from the bitstream representing segmentation information are binarization flags. Specifically, the processed segmentation information is represented by a set of binarization flags.

[0438] Providing binary flags enables efficient decoding. At the decoding end, the processing of logical flags can be performed with low complexity.

[0439] According to an exemplary implementation, the upsampling of segmentation information in each segmentation information processing layer j further includes: for each p-th position in the acquired feature map represented by the input segmentation information, determining an indication of the feature map position included in the region that is the same as the p-th position in the reconstructed image, as the upsampled segmentation information.

[0440] For example, data used for image or video processing includes motion vector fields. Since dense optical flow or motion vector fields with resolutions similar to those of the image are required to model motion, the hierarchical structure of this invention is suitable for efficiently reconstructing such motion information. Through layered processing and instruction, a good trade-off between rate and distortion can be achieved.

[0441] For example, the data used for image or video processing includes image data and / or prediction residual data and / or prediction information data. This invention can be used with a variety of different parameters. However, image data and / or prediction residual data and / or prediction information data can still have some redundancy in the spatial domain, and the hierarchical method described herein can provide efficient decoding from the bitstream using different resolutions.

[0442] In some embodiments and examples, filters are used for upsampling of feature maps, and the shape of the filters is any one of square, horizontal rectangle, and vertical rectangle.

[0443] Applying different upsampling filters helps to adapt to different features of the content. For example, upsampling of feature maps uses filters, and the input information from the bitstream also includes: obtaining information from the bitstream representing the filter shape and / or filter coefficients.

[0444] Accordingly, the decoder can provide better reconstruction quality based on the information from the encoder sent in the bitstream.

[0445] For example, the information representing the filter shape is represented by a mask composed of flags, and the mask represents the filter shape because a flag with a third value indicates a non-zero filter coefficient, and a flag with a fourth value different from the third value indicates that the filter coefficient is 0. This allows for more flexible design of filters of any shape.

[0446] For example, multiple cascaded layers include convolutional layers, instead of upsampling between layers with different resolutions.

[0447] Providing such an additional layer in the cascaded network can introduce additional processing, such as various types of filtering, to improve the quality or efficiency of decoding.

[0448] According to one embodiment, a computer program product stored in a non-transitory medium is provided that, when executed on one or more processors, performs the method described in any of the above methods.

[0449] According to one embodiment, an apparatus for decoding images or videos is provided, including processing circuitry for performing the method according to any of the above embodiments and examples.

[0450] According to one aspect, an apparatus for decoding data for image or video processing from a bitstream can be provided, the apparatus comprising: an acquisition unit for acquiring two or more feature map element sets from the bitstream, wherein each feature map element set is associated with a feature map; an input unit for inputting each feature map element set from the two or more feature map element sets into two or more feature map processing layers in a plurality of cascaded layers; and a decoded data acquisition unit for acquiring the decoded data for image or video processing as the processing result of the plurality of cascaded layers.

[0451] Any of the above-described devices can be implemented on an integrated chip. This invention can be implemented in hardware (HW) and / or software (SW). Furthermore, hardware-based implementations can be combined with software-based implementations.

[0452] It should be noted that this invention is not limited to a specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0453] Some exemplary implementations in hardware and software Figure 35 A corresponding system is shown that can be deployed with the above encoder-decoder processing chain. Figure 35 This is a schematic block diagram of an exemplary decoding system, such as a video, image, audio, and / or other decoding system (or simply a decoding system) that can utilize the techniques of this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform the various techniques according to the various examples described in this application. For example, video encoding and decoding can use neural networks or the processing networks typically described in the above embodiments and examples.

[0454] like Figure 35As shown, the decoding system 10 includes a source device 12, for example, the source device 12 is used to provide encoded image data 21 to a destination device 14 for decoding encoded image data 13.

[0455] The source device 12 includes an encoder 20, and optionally may include an image source 16, an image preprocessor 18 or other preprocessing unit 18, a communication interface or communication unit 22.

[0456] Image source 16 may include or may be any type of image capture device, such as a camera for capturing real-world images, and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images, or any other type of device for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory / storage for storing any of the aforementioned images.

[0457] Unlike the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 can also be referred to as the raw image or raw image data 17.

[0458] The preprocessor 18 receives (raw) image data 17 and preprocesses the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include trimming, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It should be understood that the preprocessing unit 18 may be an optional component. It should also be noted that preprocessing may employ a neural network.

[0459] The video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0460] The communication interface 22 in the source device 12 can be used to: receive encoded image data 21 and send the encoded image data 21 (or any other processed version) to another device such as the destination device 14 or any other device via the communication channel 13 for storage or direct reconstruction.

[0461] Destination device 14 includes decoder 30 (e.g., video decoder 30), and optionally may include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0462] The communication interface 28 in the destination device 14 is used to receive encoded image data 21 (or any other processed version) directly from the source device 12 or from any other source device such as a storage device, for example, an encoded image data storage device, and to provide the encoded image data 21 to the decoder 30.

[0463] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired or wireless networks or any combination thereof, or any type of private and public network), or any combination thereof.

[0464] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing for transmission on a communication link or communication network.

[0465] For example, the communication interface 28 corresponding to the communication interface 22 can be used to receive the transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain the encoded image data 21.

[0466] Both communication interface 22 and communication interface 28 can be configured as Figure 35 The communication channel 13, indicated by the arrow pointing from source device 12 to destination device 14, is a one-way communication interface, or configured as a two-way communication interface, and can be used to send and receive messages, etc., to establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). Decoder 30 is used to receive encoded image data 21 and provide decoded image data 31 or decoded image 31 (e.g., using the neural network described in the embodiments or examples above).

[0467] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also referred to as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). The post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing to provide the decoded image data 31 for display by the display device 34, etc.

[0468] The display device 34 in the destination device 14 is used to receive post-processed image data 33 to display the image to a user or viewer. The display device 34 can be or can include any type of display for representing the reconstructed image, such as an integrated or external display or screen. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0469] although Figure 35 Source device 12 and destination device 14 are depicted as separate devices; however, embodiments of the devices may also include source device 12 and destination device 14, or simultaneously include corresponding functions of source device 12 and corresponding functions of destination device 14. In such embodiments, the source device 12 or its corresponding functions and the destination device 14 or its corresponding functions may be implemented using the same hardware and / or software, using separate hardware and / or software, or any combination thereof.

[0470] Based on the description, it is obvious to the technicians that... Figure 35 The presence and (precise) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0471] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, can be implemented using processing circuitry, such as one or more microprocessors, one or more digital signal processors (DSPs), one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), one or more discrete logic devices, one or more hardware devices, one or more dedicated video decoding processors, or any combination thereof. Encoder 20 can be implemented using processing circuitry 46 to encompass various modules, including neural networks. Decoder 30 can be implemented using processing circuitry 46 to include the various modules discussed in the above embodiments and examples. The processing circuitry can be used to perform various operations, which will be discussed later. When the technology is implemented in part in software, the device may store software instructions in a suitable non-transitory computer-readable storage medium, and may execute the instructions in hardware using one or more processors to perform the technology of the invention. Video encoder 20 or video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (encoder-decoder), such as... Figure 36 As shown.

[0472] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or tablet computer, mobile phone, smartphone, tablet / tabletcomputer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (such as a content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 may be equipped with components for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0473] In some cases, Figure 35The video decoding system 10 shown is merely exemplary, and the technology provided in this application can be applied to video decoding setups (e.g., video encoding or video decoding) that do not necessarily include any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, sent over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode it.

[0474] Figure 37 This is a schematic diagram of a video decoding device 3700 provided for an embodiment of the present invention. The video decoding device 3700 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 3700 may be a decoder (such as...) Figure 35 Video decoder 30) or encoder (such as Figure 35 (video encoder 20).

[0475] The video decoding device 3700 includes an input port 3710 (or input port 3710) for receiving data and a receiver unit (Rx) 3720, a processor, logic unit, or central processing unit (CPU) 3730 for processing data, a transmitter unit (Tx) 3740 and an output port 3750 (or output port 3750) for transmitting data, and a memory 3760 for storing data. The video decoding device 3700 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 3710, receiver unit 3720, transmitter unit 3740, and output port 3750 for the entry or exit of optical or electrical signals.

[0476] Processor 3730 is implemented in both hardware and software. Processor 3730 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 3730 communicates with ingress port 3710, receiver unit 3720, transmitter unit 3740, egress port 3750, and memory 3760. Processor 3730 includes decoding module 3770. Decoding module 3770 implements the embodiments disclosed above. For example, decoding module 3770 performs, processes, prepares, or provides various decoding operations. Therefore, decoding module 3770 provides a substantial improvement to the functionality of video decoding device 3700 and affects the transitions of video decoding device 3700 to different states. Alternatively, decoding module 3770 may be implemented with instructions stored in memory 3760 and executed by processor 3730.

[0477] The memory 3760 may include one or more disks, tape drives, and solid-state drives, which can be used as overflow data storage devices to store programs when they are selected for execution, and to store instructions and data read during program execution. For example, the memory 3760 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0478] Figure 38 A simplified block diagram of a device 3800 provided for an exemplary embodiment, the device 3800 being used as Figure 35 The source device 12 and the destination device 14 are either or both of them.

[0479] The processor 3802 in device 3800 may be a central processing unit. Alternatively, the processor 3802 may be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation may be implemented using a single processor such as the processor 3802 shown in the figure, using more than one processor can improve speed and efficiency.

[0480] In one implementation, the memory 3804 in device 1100 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 3804. Memory 3804 may include code and data 3806 accessed by processor 3802 via bus 3812. Memory 3804 may also include an operating system 3808 and an application program 3810, which includes at least one program that causes processor 3802 to perform the methods described herein. For example, application program 3810 may include applications 1 to N, and also includes an image decoding (encoding or decoding) application that performs the methods described herein.

[0481] The device 3800 may also include one or more output devices, such as a display 3818. In one example, the display 3818 may be a touch-sensitive display that combines a display with a touch-sensitive element that can be used to sense touch input. The display 3818 may be coupled to the processor 3802 via a bus 3812.

[0482] Although bus 3812 in device 3800 is described herein as a single bus, bus 3812 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components of device 3800 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 3800 can have a wide variety of configurations.

[0483] In general, the present invention relates to methods and apparatus for encoding data (for still image or video processing) into a bitstream. Specifically, the data is processed by a network comprising multiple cascaded layers. In the processing, feature maps are generated by the layers. The feature maps processed (output) by at least two different layers have different resolutions. In the processing, a layer is selected from the cascaded layers that is different from the layer that generated the lowest resolution feature map (e.g., the latent space). The bitstream includes information associated with the selected layer. This method provides scalable processing that can be applied to different resolutions so that the bitstream can transmit information associated with such different resolutions. Therefore, the data can be efficiently decoded in the bitstream according to the resolution, which can vary depending on the content of the encoded image data.

[0484] The present invention also relates to a method and apparatus for decoding data from a bitstream (for still image or video processing). Specifically, two or more feature map element sets are obtained from the bitstream. Each feature map element set is associated with a feature map. Each feature map element set is input into two or more feature map processing layers in a plurality of cascaded layers. Then, the decoded data for image or video processing is obtained as the processing result of the plurality of cascaded layers. Therefore, data can be decoded from the bitstream in a hierarchical structure in an efficient manner.

[0485] The present invention also relates to a method and apparatus for decoding data from a bitstream (for still image or video processing). Two or more sets of segmentation information elements are obtained from the bitstream. Then, each of the two or more sets of segmentation information elements is input into two or more segmentation information processing layers in a plurality of cascaded layers. In each of the two or more segmentation information processing layers, its respective set of segmentation information is processed. Based on the segmentation information processed by the plurality of cascaded layers, decoded data for image or video processing is obtained. Therefore, data can be decoded from the bitstream in an efficient manner within a hierarchical structure.

Claims

1. A method for decoding data from a bitstream for image or video processing, characterized in that, The method includes: Obtaining two or more sets of segmentation information elements from the bitstream; Respectively inputting each of the two or more sets of segmentation information elements in the two or more sets of segmentation information elements into two or more segmentation information processing layers in multiple cascaded layers; Processing the respective segmentation information sets in each of the two or more segmentation information processing layers; Wherein, obtaining the decoded data for image or video processing is performed according to the segmentation information processed by the multiple cascaded layers; Wherein, the resolutions of the segmentation information respectively processed in the two or more segmentation information processing layers are different; Wherein, the processing of the segmentation information in the two or more segmentation information processing layers includes upsampling, and the upsampling of the segmentation information includes nearest neighbor upsampling.

2. The method according to claim 1, wherein: Obtaining the set of segmentation information elements is performed according to the segmentation information processed by at least one segmentation information processing layer in the multiple cascaded layers.

3. The method according to claim 1 or 2, wherein: Inputting the set of segmentation information elements is performed according to the processed segmentation information output by at least one cascaded layer in the multiple cascaded layers.

4. The method according to claim 1, characterized in that, The upsampling of the segmentation information includes transposed convolution.

5. The method according to claim 1, wherein: For each segmentation information processing layer j in N segmentation information processing layers in the multiple cascaded layers, The input includes: if j = 1, input the initial segmentation information from the bitstream, otherwise input the segmentation information processed by the (j - 1)th segmentation information processing layer; Output the processed segmentation information.

6. The method according to claim 5, characterized in that, The processing performed by each layer j < N in the N segmentation information processing layers on the input segmentation information further includes: Parsing segmentation information elements from the bitstream, and associating the parsed segmentation information elements with the segmentation information output by the previous layer, wherein the position of the parsed segmentation information elements in the associated segmentation information is determined according to the segmentation information output by the previous layer.

7. The method according to claim 6, wherein: The number of segmentation information elements parsed from the bitstream is determined according to the segmentation information output by the previous layer.

8. The method according to claim 6 or 7, wherein: The parsed segmentation information elements are represented by a set of binarization flags.

9. The method according to any one of claims 1, 2, 4 to 7, characterized in that, Obtaining the decoded data for image or video processing includes determining at least one of the following according to the segmentation information: Intra-frame or inter-frame prediction mode; Image reference index; Single-reference or multi-reference prediction; Existence or non-existence of prediction residual information; Quantization step size; Motion information prediction type; Motion vector length; `` Motion vector resolution; Motion vector prediction index; Motion vector difference magnitude; Motion vector difference resolution; Motion interpolation filter; In-loop filter parameters; Post-filter parameters.

10. The method according to any one of claims 1, 2, 4 to 7, characterized in that, It further includes: The feature map element set is obtained from the bitstream, and according to the segmentation information processed by the segmentation information processing layer, the feature map element set is respectively input to the feature map processing layer in the multiple cascaded layers; Based on the feature maps of the multiple cascaded layers, the decoded data for image or video processing is obtained.

11. The method according to claim 10, characterized in that, At least one of the multiple cascaded layers is a segmentation information processing layer and a feature map processing layer.

12. The method according to claim 10, characterized in that, Each of the multiple cascaded layers is a segmentation information processing layer or a feature map processing layer.

13. A computer program product stored in a non-transient medium, characterized in that, When the computer program product is executed on one or more processors, it performs a method for decoding data from a bitstream for image or video processing, the method comprising: Obtain two or more sets of segmentation information elements from the bitstream; Each of the two or more segmentation information element sets is input into two or more segmentation information processing layers in multiple cascaded layers; In each of the two or more segmentation information processing layers, a respective segmentation information set is processed; The acquisition of decoded data for image or video processing is performed based on the segmentation information processed by the multiple cascaded layers. The resolutions of the segmentation information processed in the two or more segmentation information processing layers are different. The processing of the segmented information in the two or more segmented information processing layers includes upsampling, and the upsampling of the segmented information includes nearest neighbor upsampling.

14. A device for decoding images or videos, characterized in that, The processing circuitry includes a method for performing a method of decoding data from a bitstream for image or video processing, the method comprising: Obtain two or more sets of segmentation information elements from the bitstream; Each of the two or more segmentation information element sets is input into two or more segmentation information processing layers in multiple cascaded layers; In each of the two or more segmentation information processing layers, a respective segmentation information set is processed; The acquisition of decoded data for image or video processing is performed based on the segmentation information processed by the multiple cascaded layers. The resolutions of the segmentation information processed in the two or more segmentation information processing layers are different. The processing of the segmented information in the two or more segmented information processing layers includes upsampling, and the upsampling of the segmented information includes nearest neighbor upsampling.

15. An apparatus for decoding data from a bitstream for image or video processing, characterized in that, The device includes: An acquisition unit is used to acquire two or more sets of segmentation information elements from the bitstream; The input unit is used to input each of two or more segmentation information element sets into two or more segmentation information processing layers in multiple cascaded layers; The processing unit is used to process the respective segmentation information set in each of two or more segmentation information processing layers; The decoded data acquisition unit is used to acquire the decoded data for image or video processing based on the segmentation information processed in the plurality of cascaded layers; The resolutions of the segmentation information processed in the two or more segmentation information processing layers are different. The processing of the segmented information in the two or more segmented information processing layers includes upsampling, and the upsampling of the segmented information includes nearest neighbor upsampling.

Images