Method and apparatus for indicating block parameters for multiple image coding operations

Abstract

An image processing device for reconstructing an image from an image data unit, the device comprising one or more processors configured to: process the image data unit to identify a first field of a first predetermined type and a second field of a second predetermined type therein, and determine, from a value of the first field, a block size for a second block-level operation, and determine, from a value of the second field, a block size for a first block-level operation; perform the first block-level operation on data derived from the image data unit to generate intermediate data; perform the second block-level operation on the intermediate data. An accompanying method and instruction syntax are provided.

Description

Technical Field

[0001] Embodiments of the present invention generally relate to the field of encoding and decoding data based on neural network architectures. Specifically, some embodiments relate to methods and apparatus for encoding and decoding images and / or videos from bitstreams using multiple processing layers. Background Technology

[0002] In recent decades, hybrid image and video codecs have been used to compress image and video data. In such codecs, the signal is typically encoded block-by-block by predicting blocks and then further decoded using only the difference between the original block and its prediction. Specifically, this decoding can include transform, quantization, and bitstream generation, often involving some entropy decoding. Typically, the three components of a hybrid coding approach—transform, quantization, and entropy coding—are optimized separately. Modern video compression standards such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use the transformed representation to encode the residual signal after prediction.

[0003] Recently, neural network architectures have been applied to image and / or video decoding. Typically, these neural network (NN)-based methods can be applied to image and video decoding in various ways. For example, some end-to-end optimized image or video decoding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize certain parts of the end-to-end decoding framework, such as the selection of prediction parameters or compression. Additionally, some neural network-based methods have been discussed for hybrid image and video decoding frameworks, such as trained deep learning models implemented for intra-frame or inter-frame prediction in image or video decoding.

[0004] The common thread among the end-to-end optimized image or video decoding applications discussed above is that they generate some feature map data, which is then transmitted between the encoder and decoder.

[0005] Neural networks are machine learning models that use one or more layers of nonlinear units to predict the output of a received input. Some neural networks include one or more hidden layers in addition to the output layer. Corresponding feature maps can be provided as the output of each hidden layer. These corresponding feature maps of each hidden layer can be used as input to subsequent layers in the network (i.e., subsequent hidden layers or output layers). Each layer of the network generates an output from the received input based on the current values ​​of its corresponding parameter set. In neural networks partitioned between devices (e.g., between encoders and decoders, between devices and the cloud, or between different devices), the feature map on the output side of the partition location (e.g., the first device) is compressed and transmitted to the remaining layers of the neural network (e.g., the second device).

[0006] Further improvements to encoding and decoding may be needed using a trained network architecture. Summary of the Invention

[0007] The present invention provides a method and apparatus for improving image compression.

[0008] The above and other objectives are achieved through the subject matter claimed in the independent claims. Other implementations are apparent in the dependent claims, the specification, and the drawings.

[0009] Specific embodiments are set forth in the appended independent claims, and other embodiments are set forth in the dependent claims.

[0010] According to a first aspect, the present invention relates to an image processing apparatus for reconstructing an image from an image data unit, the apparatus comprising one or more processors configured to: process the image data unit to identify therein a first field of a first predetermined type and a second field of a second predetermined type, and determine, based on the value of the first field, a tile size for a second tilewise operation, and based on the value of the second field, a tile size for a first tilewise operation; perform the first tilewise operation on data derived from the image data unit to generate intermediate data; and perform the second tilewise operation on the intermediate data.

[0011] The above and other embodiments may each optionally include one or more of the following individual or combined features.

[0012] In one possible implementation, the one or more processors process the image data unit to identify a third field of a third predetermined type therein, and determine block overlap for the second block-level operation based on the value of the third field, and determine block overlap for the first block-level operation based on the value of the second field or the value of a fourth field of a fourth predetermined type. This can support specifying overlap in a flexible and / or efficient manner.

[0013] The value of the second field can include an explicit value for the chunk size used in the first chunk-level operation. This allows for specifying the chunk size in a flexible manner.

[0014] The value of the fourth field can include an explicit value for the chunk overlap used in the first chunk-level operation. This allows for specifying overlap in a flexible way.

[0015] The second field may include a flag that, when set to a value or state (e.g., '1'), indicates that the first and second block-level operations include juxtaposed blocks, and when set to another value or state (e.g., '0'), indicates that the first and second block-level operations include hierarchical blocks. Juxtaposed blocks may have the same size and / or position, and / or may be defined such that each block in a subsequent operation includes or only includes samples generated by processing the corresponding or corresponding block in a previous operation. Hierarchical blocks may be shifted and / or have different sizes between previous and subsequent block-level operations. Specifying the type of block in this way helps to efficiently specify block parameters.

[0016] The value of a second field having a predetermined property (e.g., '1') can indicate that the block size used for the first block-level operation can be determined based on the value of the first field. One or more processors of the device can determine the block size in this manner in response to the second field having this property. This allows for efficient specification of the block size.

[0017] The value of a second field with predetermined properties (e.g., '1') can indicate that the chunk overlap used for the first chunk-level operation can be determined based on the value of the third field. This allows for the efficient specification of chunk overlap.

[0018] The first block-level operation can be a prediction operation performed on blocks in the hyperlatent space. Intermediate data may include the output of the prediction operation in the latent space. The second block-level operation can be a synthesis operation performed on blocks in the latent space. This type of pipeline can provide efficient image reconstruction.

[0019] First-level and second-level block operations can be performed on square blocks. The block size can be represented as the number of samples along any single edge of the block. The block size can be either the length or width of the block. The block size can also be represented as the length or width divided by a predetermined factor. These features support efficient representation of block size.

[0020] Block overlap can be represented as the number of samples spanned from the edge inwards by its overlap with adjacent blocks. This allows for efficient representation of block overlap.

[0021] The block size processed by the first block-level operation can be represented in a superlatent space requiring up to 7 bits. Block overlap of blocks processed by the first block-level operation can be represented in a superlatent space of up to 3 bits. Explicit values ​​for the second block-level operation can be represented in up to 9 bits for block size and up to 4 bits for block overlap. This feature provides an efficient range of size and / or overlap options.

[0022] The block size used for the first block-level operation can be determined as a fraction of the value of the first field. One or more processors of the device can be configured to determine the block size in this way. This allows for efficient specification of the block size.

[0023] Block overlap processed through the first block-level operation can be determined as a fraction of the value of the third field. One or more processors of the device can be configured to determine block overlap in this manner. This allows for efficient specification of block overlap.

[0024] The block size and block overlap used for prediction operations can be represented in a hyperlatent space and can be transformed for multi-stage context modeling by multiplying the explicit values ​​of the second or fourth field by 4. One or more processors can be used to perform this transformation. This allows for efficient specification of size and overlap.

[0025] According to a second aspect, the present invention relates to a data carrier for storing image data units, the image data units comprising data, the data being reconstructed to form an image by: performing a first block-level operation on the data derived from the image data units to generate intermediate data and performing a second block-level composite operation on the intermediate data; the image data units comprising a first field of a first predetermined type and a second field of a second predetermined type, the first field of the first predetermined type specifying the block size for the first block-level operation, and the second field of the second predetermined type specifying the block size for the second block-level operation.

[0026] The data carrier can be configured to provide input to one or more processors of the first aspect.

[0027] The above and other embodiments may each optionally include one or more of the following individual or combined features.

[0028] According to a third aspect, the present invention relates to a method for processing an image tensor representing image data via a neural network including a first subnet and a second subnet, the processing including indicating in block header data the size of block portions of the image tensor processed by each subnet of the neural network, the method comprising: processing the block header data to identify therein a first field of a first predetermined type and a second field of a second predetermined type, and determining, based on the value of the first field, a block size for applying the second subnet, and based on the value of the second field, a block size for applying the first subnet; applying the first subnet to block portions derived from the image tensor to generate intermediate tensor data; and applying the second subnet to block portions derived from the intermediate tensor data. This method can be performed by an apparatus described in the first or fourth aspect. Modules of the apparatus can perform the operations.

[0029] The above and other embodiments may each optionally include one or more of the following individual or combined features.

[0030] In one possible implementation, the method may include: processing the chunk header data to identify a third field of a third predetermined type therein, determining chunk overlap for applying the second subnet based on the value of the third field, and determining chunk overlap for applying the first subnet based on the value of the second field or the value of a fourth field of a fourth predetermined type.

[0031] The value of the second field can include an explicit value for the chunk size used to apply the first subnet. This allows for efficient specification of the chunk size.

[0032] The method may include: determining a chunk overlap for applying the first subnet based on the value of the fourth field, wherein the fourth field includes an explicit value for the chunk overlap. This can support efficient specification of chunk overlap.

[0033] The method may include a flag that, when set to a first predetermined value (e.g., "1"), indicates that the first block-level operation and the second block-level operation include juxtaposed blocks, and when set to a second predetermined value (e.g., "0"), indicates that the first block-level operation and the second block-level operation include hierarchical blocks. Specifying the block type in this way helps to efficiently specify block parameters.

[0034] The value of the second field can indicate that the chunk size used to apply the first subnet can be determined based on the value of the first field. This allows for efficient specification of the chunk size.

[0035] The value of the second field can indicate the tile overlap used to apply the first subnet, which can be determined based on the value of the third field. This allows for efficient specification of tile overlap.

[0036] Applying the first subnet may involve performing prediction operations on blocks within the hyperlatent space. Intermediate tensor data may include the output of the prediction operations in the latent space, and applying the second subnet may involve performing synthesis operations on the blocks within the latent space. This type of pipeline can provide efficient image reconstruction.

[0037] The first and second subnets can be applied to square blocks, and the block size can be represented as the number of samples along any single edge of the block. This allows for efficient specification of the block size.

[0038] Block overlap can be represented as the number of samples spanned from the edge inwards by its overlap with adjacent blocks. This allows for efficient specification of block overlap.

[0039] The block size of a block processed by the first subnet can be represented in a hyperlatent space requiring up to 7 bits. Block overlap of blocks processed by the first subnet can be represented in a hyperlatent space of up to 3 bits. Explicit values ​​for applying the second subnet can be represented in up to 9 bits for block size and up to 4 bits for block overlap. Valid block size and / or overlap ranges can be provided.

[0040] The block size used for the first subnet can be determined as a fraction of the value of the first field. The block overlap of blocks processed by the first subnet can be determined as a fraction of the value of the third field. Each of these methods is an efficient method for specifying the appropriate number.

[0041] The chunk size and chunk overlap used for applying the first subnet can be represented in the hyperlatent space. This can be transformed for multi-stage context modeling by multiplying the explicit values ​​of the second and fourth fields by 4. This allows for efficient bandwidth utilization for transmitting the second and / or fourth fields.

[0042] According to a fourth aspect, the present invention relates to an image processing apparatus for processing an image tensor representing image data via a neural network, the processing comprising: indicating the size of a block portion of the image tensor, the apparatus being configured to indicate the size in the block header data of a plurality of subnetworks of the neural network, and formatting the indication according to the following syntax: tile_header (comp) { tile_y_enable[comp] if (tile_y_enable[comp]) { tile_y_size[comp] tile_y_overlap[comp] } tile_z_enable[comp] if (tile_z_enable[comp]) { tile_z_is_derived[comp] if (!tile_z_is_derived) { tile_z_size[comp] tile_z_overlap[comp] } } } The syntax described above is functionally equivalent to the syntax shown above. For example, the same concept can be given different names without changing the syntax functionally. This syntax can provide an efficient way to transmit chunk size and / or overlap information.

[0043] In block-level operations, multiple chunks or blocks of adjacent samples can be processed. Samples in each chunk or block can be processed together, but separately from samples in other chunks or blocks. Chunks or blocks can be processed serially or in parallel. Attached Figure Description

[0044] The embodiments of the present invention will be described in detail below with reference to the accompanying drawings. In the drawings: Figure 1 This is a schematic diagram showing the channels processed by the various layers of a neural network; Figure 2 This is a schematic diagram illustrating the type of autoencoder in a neural network; Figure 3A This is a schematic diagram illustrating an exemplary network architecture that includes an encoder and decoder with a priori model. Figure 3B This is a schematic diagram illustrating a general network architecture that includes an encoding end with a super-prior model; Figure 3C This is a schematic diagram illustrating a general network architecture that includes a decoding end with a priori model; Figure 4 This is a schematic diagram illustrating an exemplary network architecture that includes an encoder and decoder with a priori model. Figure 5 This is a block diagram illustrating the structure of a cloud-based solution for machine-based tasks such as machine vision. Figure 6AThis is a block diagram illustrating an end-to-end video compression framework based on neural networks; Figure 6B This is a block diagram illustrating some exemplary details of a neural network application for sports field compression; Figure 6C This is a block diagram illustrating some exemplary details of neural network applications for motion compression; Figure 7 This is a block diagram illustrating an example of an encoding or decoding device; Figure 8 This is a block diagram illustrating another example of an encoding or decoding device; Figure 9 This is a schematic diagram illustrating an embodiment of a network architecture that includes a component of the encoding end of a priori model; Figure 10 This is a schematic diagram illustrating an embodiment of a network architecture that includes a component of the decoding end of a priori model; Figure 11 It is a block diagram of an encoder and decoder with corresponding modules, the corresponding modules including NN-based subnets for processing a first plurality of blocks and a second plurality of blocks; Figure 12 An example is shown of dividing a first tensor and / or a second tensor into overlapping regions Li (i.e., a first block and a second block), subsequently cropping samples within the overlapping regions, and then stitching the cropped regions together. Each Li includes the total receptive field; Figure 13 It shows the relationship with Figure 17 Another example similar to this is to divide the first tensor and / or the second tensor into overlapping regions Li (i.e., the first block and the second block), except that clipping is removed; Figure 14 An example is shown where a first tensor and / or a second tensor are divided into overlapping regions Li (i.e., a first block and a second block), samples within the overlapping regions are then cropped, and the cropped regions are stitched together. Each Li comprises a subset of the total receptive field; Figure 15 An example is shown of dividing a first tensor and / or a second tensor into non-overlapping regions Li (i.e., a first block and a second block), followed by sample cropping and stitching of the cropped regions. Each Li comprises a subset of the total receptive field; Figure 16 Various parameters, such as the sizes of regions Li, Ri, and overlapping regions of the first and / or second blocks, are shown. These parameters can be included in (and parsed from) the bitstream. Any one of these parameters can be included in (and parsed from) the bitstream as a representation. Figure 17The diagram illustrates a portion of the JPEG AI decoding scheme and two variations of the block-based scheme that can be used in the image decoding process; Figure 18 Showing more details Figure 17 Two variations of the block partitioning scheme; Figure 19 This is a flowchart illustrating an exemplary method for decoding blocks across multiple neural network subnets; Figure 20 A device for decoding to be processed by a neural network-based unit is shown; Figure 21 This is a block diagram illustrating an example of a video decoding system for implementing embodiments of the present invention; Figure 22 This is a block diagram illustrating another example of a video decoding system for implementing embodiments of the present invention; Figure 23 This is a block diagram illustrating an example of an encoding or decoding device; Figure 24 This is a block diagram illustrating another example of an encoding or decoding device; Figure 25 This is a block diagram illustrating another example of an encoding or decoding device.

[0045] Similar reference numbers and names in different figures may represent similar elements. Detailed Implementation

[0046] In the following description, reference is made to the accompanying drawings, which form part of this invention, illustrating by way of description specific aspects of embodiments of the invention or aspects in which embodiments of the invention may be used. It should be understood that embodiments of the invention can be used in other aspects and may include structural or logical variations not described in the drawings. Therefore, the following detailed description should not be construed in a limiting sense, and the scope of the invention is defined by the appended claims.

[0047] For example, it should be understood that the disclosure of the described method is equally applicable to corresponding devices or systems for performing the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units (e.g., functional units) to perform the described one or more method steps (e.g., one unit performs one or more steps, or multiple units perform one or more of multiple steps respectively), even if the one or more units are not explicitly described or illustrated in the drawings. On the other hand, for example, if a specific apparatus is described based on one or more units (e.g., functional units), the corresponding method may include a step to perform the function of one or more units (e.g., one step performs the function of one or more units, or multiple steps perform the function of one or more of multiple units respectively), even if the one or more steps are not explicitly described or shown in the drawings. Furthermore, it should be understood that, unless otherwise stated, features of the various exemplary embodiments and / or aspects described herein may be combined with each other.

[0048] The following provides an overview of some of the technical terms and frameworks used within which embodiments of the present invention can be used.

[0049] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computational systems fuzzily inspired by the biological neural networks that make up the animal brain. These systems "learn" to perform tasks by example, typically without being programmed using task-specific rules. For instance, in image recognition, these systems might learn to recognize images containing cats by analyzing exemplary images manually labeled "cat" or "no cat" and using the results to identify cats in other images. These systems do this without any prior knowledge about cats, such as that they have fur, tails, whiskers, and cat-like faces. Instead, these systems automatically generate recognizable features from the examples they process.

[0050] ANNs are collections of connection units or nodes called artificial neurons, which loosely model neurons in a biological brain. Each connection, like a synapse in a biological brain, can transmit signals to other neurons. The artificial neuron that receives the signal then processes it and can transmit signals to the neurons connected to it.

[0051] In ANN implementations, the "signals" at connections are real numbers, and the output of each neuron is calculated using a non-linear function of the sum of its inputs. These connections are called edges. Neurons and edges typically have weights that adjust as they learn. These weights increase or decrease the strength of the signal at the connection. Neurons can have thresholds, such that a signal is only sent when the aggregate signal exceeds that threshold. Typically, neurons are clustered into layers. Different layers can perform different transformations on their inputs. The signal may travel from the first layer (input layer) to the last layer (output layer) after multiple traversals of the layers.

[0052] The initial goal of ANN methods was to solve problems in the same way as the human brain. Over time, the focus shifted to performing specific tasks, leading to a departure from biology. ANNs have been applied to a wide variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, board games and video games, medical diagnosis, and even activities traditionally considered to be solely human, such as painting.

[0053] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a specialized linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0054] Figure 1 This diagram illustrates the general concepts of processing in a neural network (such as a CNN). A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer provides input (e.g., ... Figure 1 The input image shown is a portion 11) processed by the layers. Hidden layers in a CNN typically consist of a series of convolutional layers that perform convolutions through multiplication or other dot products. The result of a layer is one or more feature maps (represented by empty solid-line rectangles), sometimes also called channels. Resampling (e.g., subsampling) may be involved in some or all layers. Therefore, the feature maps may become smaller, such as Figure 1 As shown. It's worth noting that convolutions with strides can also reduce the size of the input feature map (resampling). The activation functions in CNNs are typically rectified linear unit (ReLU) layers or Leaky ReLU, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers, called hidden layers, because their inputs and outputs are masked by the activation function and the final convolution. While these layers are colloquially called convolutional layers, this is just a convention. Mathematically, convolution is technically a sliding dot product or cross-correlation. This is important for indexing in a matrix because the sliding dot product or cross-correlation affects how weights are determined at a particular index point.

[0055] When programming a CNN to process images, such as Figure 1 As shown, the input is a tensor of shape (number of images) × (image width) × (image height) × (image depth). It should be understood that image depth can be composed of the image's channels. After passing through the convolutional layer, the image is abstracted into a feature map, with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer within the neural network should have the following properties: a convolutional kernel defined by width and height (hyperparameters); the number of input and output channels (hyperparameters); and the depth of the convolutional filter (input channels) should be equal to the number of channels (depth) of the input feature map.

[0056] In the past, traditional multilayer perceptron (MLP) models were used for image recognition. However, due to the full connectivity between nodes, MLP models are affected by high dimensionality and cannot scale well when processing high-resolution images. A 1000×1000 pixel image with RGB color channels has 3 million weights, which is too high for efficient and feasible large-scale processing with full connectivity. This network architecture also does not consider the spatial structure of the data, causing input pixels that are far apart to be processed in the same way as pixels that are very close together. This ignores the reference locality in image data both computationally and semantically. Therefore, the full connectivity of neurons is wasted for purposes such as image recognition dominated by spatially local input patterns.

[0057] Convolutional Neural Networks (CNNs) are biologically inspired variants of multilayer perceptrons (MLPs) specifically designed to mimic the behavior of the visual cortex. These models mitigate the challenges of MLP architectures by leveraging the strong spatial local correlations present in natural images. The convolutional layer is a core building block in CNNs. The parameters of this layer consist of a set of learnable filters (the kernels mentioned above) that have a small receptive field but extend across the entire depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, the dot product between the filter entry and the input is computed, and a two-dimensional activation map of that filter is generated. Thus, the network learns the filters that are activated when the network detects a specific type of feature at a certain spatial location in the input.

[0058] The activation maps of all filters stacked along the depth dimension form the complete output volume of the convolutional layer. Therefore, each entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation of a given filter. Feature maps and activations have the same meaning. In some papers, it's called an activation map because it's a mapping corresponding to the activations of different parts of an image; it's also called a feature map because it's a mapping that finds a certain feature in the image. High activation indicates that a certain feature has been found.

[0059] Another important concept in CNNs is pooling, a form of non-linear downsampling. Several non-linear functions are used to implement pooling, the most common being max pooling. Max pooling divides the input image into a set of non-overlapping rectangles and outputs the maximum value for each sub-region.

[0060] Intuitively, the exact location of a feature is less important than its approximate location relative to other features. This is the idea behind using pooling in convolutional neural networks. Pooling layers are used to progressively reduce the size of the representation space, decreasing the number of parameters, memory footprint, and computational cost in the network, and thus also controlling overfitting. In CNN architectures, pooling layers are typically inserted periodically between consecutive convolutional layers. Pooling operations provide another form of translation invariance.

[0061] Pooling layers operate independently on each depth strip in the input and spatially resize it. The most common form is a pooling layer with a 2×2 filter that applies a stride of 2 along both the width and height of each depth strip in the input, discarding 75% of the activations. In this case, each max operation is performed on four numbers. The depth dimension remains unchanged. Besides max pooling, pooling units can also use other functions, such as average pooling or... 2-norm pooling. Average pooling was frequently used in the past, but it has become less common recently compared to max pooling, which generally performs better in practice. Due to the significant reduction in representation size, there is a recent trend towards using smaller filters or abandoning pooling layers altogether. Region of Interest (ROI) pooling is a variant of max pooling where the output size is fixed and the input rectangle is a parameter. Pooling is a crucial component of object detection in convolutional neural networks based on the Fast R-CNN architecture.

[0062] The ReLU above stands for Corrected Linear Unit, which applies a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to 0. It increases the non-linearity of the decision function and the overall network without affecting the receptive field of the convolutional layers. Other functions are also used to increase non-linearity, such as the saturating hyperbolic tangent and the sigmoid function. ReLU is generally superior to other functions because it trains neural networks several times faster without significantly reducing generalization accuracy.

[0063] Leaky ReLU, or Leaky Modified Linear Unit, is a ReLU-based activation function, but it has a smaller slope for negative values ​​instead of a flat slope. The slope coefficients are determined before training; that is, they are not learned during training. This type of activation function is common in tasks with sparse gradients, such as training generative adversarial networks. Leaky ReLU applies an element-wise function: LeakyReLU(x)=max(0,x)+negative_slope min(0,x), or

[0064] Among them, the parameters are: negative_slope: Controls the angle of the negative slope. Default value: 1e–2 inplace: Optionally allows in-place operations to be performed. Default: False.

[0065] After several convolutional and max-pooling layers, high-level inference in a neural network is achieved through fully connected layers. Neurons in a fully connected layer are connected to all activations in the previous layer, as shown in a conventional (non-convolutional) artificial neural network. Therefore, these activations can be computed as affine transformations, with matrix multiplication followed by bias shifts (learned or fixed-bias vector additions).

[0066] The "loss layer" (including the calculation of the loss function) specifies how training penalizes the deviation between the predicted (output) label and the true label, and is usually the last layer of the neural network. Various loss functions suitable for different tasks can be used. Softmax loss is used to predict a single class from K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0, 1]. Euclidean loss is used to regress to real-valued labels.

[0067] In short, Figure 1The diagram illustrates the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map comprising several channels, corresponding to multiple filters in a set of learnable filters for that layer. Then, the feature map is subsampled using, for example, a pooling layer, thereby reducing the size of each channel in the feature map. Next, the data enters another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layers. To establish network connectivity, these parameters need to be synchronized between two connected layers such that the number of input channels in the current layer should equal the number of output channels in the previous layer. For the first layer processing input data (e.g., an image), the number of input channels is typically equal to the number of channels in the data representation; for example, 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. Channels obtained by one or more convolutional layers (and possibly one or more resampling layers) can be passed to an output layer. This output layer can be a convolutional layer or a resampling layer in some implementations. In exemplary and non-limiting implementations, the output layer is a fully connected layer.

[0068] Autoencoders and Unsupervised Learning An autoencoder is an artificial neural network used to learn efficient data decoding in an unsupervised manner. Its schematic diagram is shown below. Figure 2 As shown. The autoencoder includes an encoding end 210 where the input x is fed into the input layer of the encoder subnet 220, and a decoding end 250 where the output x' is output from the decoder subnet 260. The autoencoder aims to learn a representation (encoding) 230 of a set of data x by training networks 220 and 260 to ignore signal "noise," typically used for dimensionality reduction. Together with the reduction-side (encoding end) subnet 220, a reconstruction-side (decoding end) subnet 260 is learned, where the autoencoder attempts to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, hence the name. In the simplest case, given a hidden layer, the encoder level of the autoencoder takes the input... and map it to : .

[0069] This image This is commonly referred to as code 230, latent variable, or latent representation. Here, It is an element-wise activation function, such as the sigmoid function or the modified linear unit. It is a weight matrix. This is the bias vector. Weights and biases are typically initialized randomly and then iteratively updated during training via backpropagation. Afterward, the decoder level of the autoencoder maps h to... Reconstruction of the same shape :

[0070] Among them, the decoder , and Corresponding to the encoder , and Irrelevant.

[0071] Variational autoencoder models make strong assumptions about the distribution of latent variables. These models use variational methods for latent representation learning, which introduces additional loss components and a specific estimator for training the algorithm, called a stochastic gradient variational Bayes (SGVB) estimator. It is assumed that the data is generated by a directed graphical model. The generator is producing the posterior distribution, and the encoder is learning the posterior distribution. approximation ,in, and These represent the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors of a VAE is typically closer to the probability distribution of the training data than that of a standard autoencoder. The objective of a VAE has the following form:

[0072] Here, This represents the Kullback-Leibler divergence. The priors of latent variables are typically set as centered isotropic multivariate Gaussian. Typically, the shapes of the variational and likelihood distributions are chosen such that they are factorized Gaussians:

[0073]

[0074] in, It is the encoder output, and and ) is the decoder output.

[0075] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have sparked researchers' interest in applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression has been proposed using networks based on variational autoencoders.

[0076] Therefore, data compression is considered a fundamental and well-studied problem in engineering, typically involving the design of codes for a given discrete dataset with minimum entropy. This approach relies heavily on an understanding of the probabilistic structure of the data, thus the problem is closely related to probabilistic source modeling. However, since all practical codes must have finite entropy, continuous-valued data (e.g., vectors of image pixel intensities) must be quantized into a finite set of discrete values, which introduces errors.

[0077] In this context, known as the lossy compression problem, a trade-off must be struck between two conflicting costs: the entropy (rate) of the discretized representation and the error (distortion) caused by quantization. Different compression applications, such as data storage or transmission over finite-capacity channels, require different rate-distortion trade-offs.

[0078] Joint optimization of rate and distortion is challenging. Without further constraints, the general problem of optimal quantization in high-dimensional spaces remains difficult to solve. Therefore, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using lossless entropy coding. Due to the central role of the transform, this scheme is called transform decoding.

[0079] For example, JPEG uses Discrete Cosine Transform (DCT) for pixel blocks, while JPEG 2000 uses Multiscale Orthogonal Wavelet Decomposition (MAD). Typically, the three components of the transform decoding method (transform, quantization, and entropy coding) are optimized separately (usually by manually adjusting parameters). Modern video compression standards, such as HEVC, VVC, and EVC, also use transform representations to decode the predicted residual signal. Several transforms are used for this purpose, such as the Discrete Cosine Transform (DCT) and the Discrete Sine Transform (DST), as well as the Low-Frequency Non-Separable Manually Optimized Transform (LFNST).

[0080] Variational image compression The variational autoencoder (VAE) framework can be considered a nonlinear transform decoding model. The transform process can be mainly divided into four parts. This is illustrated in the diagram of the VAE framework. Figure 3A An example was provided in the text.

[0081] The transformation process can be mainly divided into four parts: Figure 3A An example of a VAE framework is shown. In Figure 3AIn this code, encoder 101 maps the input image x to a latent representation (denoted by y) using the function y = f(x). This latent representation can also be referred to as a portion or a point in the "latest space" below. The function f() is a transformation function that converts the input signal x into a more compressible representation y. Quantizer 102... Transform the latent representation y into a quantized latent representation with (discrete) values. Where Q represents the quantizer function. The entropy model, or superencoder / decoder (also known as super-prior), estimates the quantized latent representation. The distribution is determined to obtain the minimum rate achievable by lossless entropy source decoding.

[0082] The latent space can be understood as a compressed representation of data, where similar data points are closer together. The latent space can be used to learn data features and find simpler data representations for analysis. Arithmetic coding (AE) is used to quantize the super-prior 3 latent representation T, and edge information Included in bitstream 2 (by binarization). Furthermore, a decoder 104 is provided that transforms the quantized latent representation into the reconstructed image. , .Signal This is an estimate of the input image x. We expect x to be as close as possible to... In other words, the desired reconstruction quality is as high as possible. However, The higher the similarity to x, the greater the amount of side information that needs to be transmitted. Side information includes, for example, Figure 3A The bitstreams 1 and 2 shown are generated by the encoder and sent to the decoder. Generally, the greater the side information, the higher the reconstruction quality. However, high side information also means a low compression ratio. Therefore, Figure 3A One objective of the system described in the text is to balance reconstruction quality with the amount of side information transmitted in the bitstream.

[0083] exist Figure 3A In this context, component AE 105 is the arithmetic coding module, used to encode the quantized latent representation. and edge information The samples are converted into a binary representation of the bitstream 1. For example, and The samples may include integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​into binary number strings (through a binarization process) (which are then included in the bitstream, which may include other parts of the encoded image or other side information).

[0084] Arithmetic decoding (AD) 106 is the process of restoring the binarization process, in which binary numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0085] It should be noted that this invention is not limited to this specific framework. Furthermore, this invention is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0086] exist Figure 3A In this case, there are two conjoined subnets. In this scenario, the subnets represent a logical division between different parts of the entire network. For example, in... Figure 3A In this code, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" subnets. The "encoder / decoder" subnets are responsible for encoding (generating) and decoding (parsing) the first bitstream "bitstream 1". Figure 3A The second network, comprising modules 103, 108, 109, 110, and 107, is called the "super encoder / decoder" subnet. This second subnet is responsible for generating the second bitstream, "bitstream 2." The two subnets serve different purposes.

[0087] The first subnet is responsible for: Transform the input image x (101) into its latent representation y (which is easier to compress x). Quantize the latent representation y (102) into the quantized latent representation. , Arithmetic coding module 105 uses AE to process the quantized latent representation. Compress the stream to obtain stream "stream 1". The arithmetic decoding module 106 is used to parse the bitstream 1 via AD. Reconstruct the image using the parsed data (104). ).

[0088] The purpose of the second subnet is to obtain the statistical properties of the samples in "stream 1" (e.g., the mean, variance, and correlation among the samples in stream 1) so that the compression of stream 1 by the first subnet is more efficient. The second subnet generates a second stream "stream 2" which includes the aforementioned information (e.g., the mean, variance, and correlation among the samples in stream 1).

[0089] The second network includes an encoding section that incorporates the quantized latent representation. Transform (103) into edge information z, and quantize the edge information z into quantized edge information. and the quantized edge information Encoding (e.g., binarization) (109) results in bitstream 2. In this example, binarization is performed using arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD) 110, which transforms the input bitstream 2 into decoded quantized side information. . Possibly with The same applies because the decoding operation at the arithmetic encoding end is a lossless compression method. Then, the decoded quantized side information... Transform (107) into the decoded edge information . express Statistical properties (e.g., The mean of the samples, or the variance of the sample values, etc.). Then, the decoded latent representation Provided to the aforementioned arithmetic encoder 105 and arithmetic decoder 106 for control The probability model.

[0090] Figure 3A An example of a variational autoencoder (VAE) is described, the details of which may vary in different implementations. For example, in a particular implementation, additional components may exist to more efficiently acquire the statistical properties of samples from bitstream 1. In one such implementation, a context modeler may exist, the goal of which is to extract the cross-correlation information of bitstream 1. The statistical information provided by the second subnet can be used by the arithmetic encoder (AE) 105 and the arithmetic decoder (AD) 106 components.

[0091] Figure 3A The encoder and decoder are depicted in a single figure. As will be apparent to those skilled in the art, the encoder and decoder can and often are embedded in different devices.

[0092] Figure 3B and Figure 3C The separate encoder and decoder components of a VAE framework are described. According to some embodiments, the encoder receives an image as input. The input image may include one or more channels, such as color channels or other types of channels, such as depth channels or motion information channels. The encoder (e.g.) Figure 3B The outputs (shown) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the first subnet of the encoder, and bitstream 2 is the output of the second subnet of the encoder.

[0093] Similarly, in Figure 3C In this process, two bitstreams (bitstream 1 and bitstream 2) are received as inputs and generated at the output. That is, reconstructing (decoding) the image. As mentioned above, a VAE can be divided into different logical units that perform different actions. This is in Figure 3B and Figure 3C The example in the text makes Figure 3B Describe the components involved in encoding the signal (such as video and provided encoding information). This encoding information is then... Figure 3C The decoder and other components in the encoder receive the data for encoding. It should be noted that the encoder and decoder components, represented by the numbers 12x and 14x, functionally correspond to those described above. Figure 3A The component mentioned in the text is represented by the number 10x.

[0094] Specifically, such as Figure 3B As shown, the encoder includes encoder 121, which transforms the input x into a signal y, and then provides the signal y to quantizer 322. Quantizer 122 provides information to arithmetic encoding module 125 and super encoder 123. Super encoder 123 provides the bitstream 2 discussed above to super decoder 147, which in turn provides information to arithmetic encoding module 105 (125).

[0095] The output of the arithmetic encoding module is bitstream 1. Bitstream 1 and bitstream 2 are the outputs of signal encoding, and these two bitstreams are then provided (sent) to the decoding process. Although unit 101 (121) is called an "encoder", it can also be used to... Figure 3B The complete subnet described in the code is called an "encoder". The encoding process typically represents a unit (module) converting the input into an encoded (e.g., compressed) output. Figure 3B As can be seen, unit 121 can actually be considered the core of the entire subnet because it performs the transformation of the input x to y, where y is a compressed version of x. Compression in encoder 121 can be achieved, for example, by applying a neural network or any processing network typically having one or more layers. In such a network, compression can be performed through a concatenation process, which includes downsampling to reduce the size of the input and / or the number of channels. Therefore, the encoder can be referred to, for example, as a neural network (NN) based encoder, etc.

[0096] The remaining parts in the diagram (quantization unit, super encoder, super decoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bit stream). Quantization can be provided to further compress the output of the NN encoder 121 through lossy compression. The AE 125, combined with the super encoder 123 and super decoder 127 used to configure the AE 125, can perform binarization, which can further compress the quantized signal through lossless compression. Therefore, it is also possible to... Figure 3B The entire subnet in the code is called the "encoder".

[0097] Most deep learning (DL) based image / video compression systems reduce the dimensionality of the signal before converting it into binary bits. For example, in a VAE framework, an encoder performing a non-linear transformation maps the input image x to y, where the width and height of y are smaller than the width and height of x. Because y has a smaller width and height, it is smaller in size, and the signal's (size) is reduced, making it easier to compress the signal y. It's important to note that, in general, an encoder doesn't necessarily need to reduce size in two (or often all) dimensions. Instead, some exemplary implementations can provide an encoder that reduces size in only one (or often a subset) dimension.

[0098] In J. Balle, L. Valero Laparra, and EP Simoncelli (2015), “Density Modeling of Images Using a Generalized Normalization Transformation,” published on arXiv, presented at the 4th Int. Conf. for Learning Representations in 2016 (hereinafter referred to as “Balle”), the authors proposed an end-to-end optimization framework for image compression models based on nonlinear transformations. They optimized for mean squared error (MSE) but used a more flexible transformation constructed from linear convolutions and nonlinear concatenation. Specifically, they used a joint nonlinearity of generalized divisive normalization (GDN), inspired by neuron models in biological visual systems, which has proven effective in Gaussianizing image density. This concatenation transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which efficiently implements the parametric form of vector quantization in the original image space. The compressed image is then reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0099] Such an example of the VAE framework is as follows: Figure 4 As shown, it uses six downsampling layers, labeled 401 to 406. The network architecture includes a super-prior model. (Left side (g)) a g s The image autoencoder architecture is shown on the right (h). a h s This corresponds to the implementation of a super-prior autoencoder. The factorized prior model uses the same architecture for analysis and synthetic transformation g. a and g s Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder performs g-quantization on the input image x. a This produces a response y (latent representation) with spatially varying standard deviation. The encoding g... a It includes multiple convolutional layers, with subsampling and generalized divisive normalization (GDN) as the activation function.

[0100] Feed the response to h aIn the first step, the standard deviation distribution of z is summarized. Then, it is quantized, compressed, and transmitted as side information of z. The encoder then uses the quantized vector... To estimate The spatial distribution of standard deviation is used to obtain the probability (or frequency) values ​​of arithmetic coding (AE) and is used to compress and transmit the quantized image representation. (or potential representation). The decoder first recovers from the compressed signal. Then, the decoder uses h s To obtain This provides it with the correct probability estimate so that it can also be successfully recovered. Then, the decoder will Feed g s To obtain the reconstructed image.

[0101] Layers that include downsampling are indicated by a down arrow in the layer description. The layer description "Conv N,k1,2↓" indicates that the layer is a convolutional layer with N channels and a kernel size of k1×k1. For example, k1 can be equal to 5, and k2 can be equal to 3. As mentioned above, 2↓ indicates that downsampling is performed by a factor of 2 in this layer. A factor of 2 downsampling results in one dimension of the input signal being reduced by half on the output side. Figure 4 In this context, 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. Due to the presence of six downsampling layers, if the width and height of the input image 414 (also denoted by x) are given by w and h, then the output signal z 413 has a width and height equal to w / 64 and h / 64, respectively. The modules represented by AE and AD are the arithmetic encoder and arithmetic decoder, whose references... Figures 3A to 3C This will be explained. Arithmetic encoders and decoders are concrete implementations of entropy decoding. AE and AD can be replaced by other entropy coding methods. In information theory, entropy coding is a lossless data compression scheme used to convert the value of a symbol into a binary representation; this is a reversible process. Furthermore, the "Q" in the diagram corresponds to the quantization operation, which is also discussed above... Figure 4 This is mentioned and further explained in the "Quantization" section above. Furthermore, the quantization operation and corresponding quantization unit that are part of component 413 or 415 may not exist and / or may be replaced by another unit.

[0102] exist Figure 4The diagram also shows a decoder including upsampling layers 407 to 412. Another layer 420 is provided between the upsampling layers 411 and 410, in the order of input processing. This additional layer is implemented as a convolutional layer but does not upsample the received input. A corresponding convolutional layer 430 for the decoder is also shown. Such a layer can be provided in a neural network to perform operations on the input that modify specific features without changing the input size. However, providing such a layer is not necessary.

[0103] When viewed in the order in which bitstream 2 is processed by the decoder, the upsampling layers operate in reverse order, from upsampling layer 412 to upsampling layer 407. Here, each upsampling layer is shown providing an upsampling ratio of 2, indicated by ↑. Of course, not all upsampling layers necessarily have the same upsampling ratio; other upsampling ratios, such as 3, 4, 8, etc., can also be used. Layers 407 to 412 are implemented as convolutional layers (conv). Specifically, since these layers may be designed to provide the opposite operation on the input to that of the encoder, the upsampling layers can apply a deconvolution operation to the received input, increasing its size by a factor corresponding to the upsampling ratio. However, the invention is generally not limited to deconvolution, and upsampling can be performed in any other way, such as through bilinear interpolation between two adjacent samples, or through nearest-neighbor sample duplication, etc.

[0104] In the first subnet, some convolutional layers (401 to 403) undergo generalized divisive normalization (GDN) after the encoder and inverse GDN (IGDN) after the decoder. In the second subnet, the activation function applied is ReLU. It should be noted that this invention is not limited to this implementation, and other activation functions can generally be used instead of GDN or ReLU.

[0105] Cloud solutions for machine tasks Video coding for machines (VCM) is another currently popular area of ​​computer science. The main idea behind this approach is to transmit a decoded representation of image or video information for further processing via computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Compared to traditional image and video decoding aimed at human perception, quality features are performance metrics for computer vision tasks, such as object detection accuracy, rather than the quality of the reconstruction. This is in... Figure 5 As shown in the image.

[0106] Machine video coding, also known as collaborative intelligence, is a relatively new paradigm for efficiently deploying deep neural networks in mobile cloud infrastructure. By partitioning the network between the mobile device 510 and the cloud 590 (e.g., a cloud server), computational workload can be distributed, minimizing the system's total energy and / or latency. Typically, collaborative intelligence is a paradigm where neural network processing is distributed across two or more distinct computing nodes, such as devices, but generally any functionally defined node. Here, the term "node" does not refer to the neural network node mentioned above. Rather, a (computing) node here refers to (physically or at least logically) independent devices / modules that implement parts of the neural network. These devices can be different servers, different end-user devices, combinations of servers and / or user devices and / or the cloud and / or processors, etc. In other words, computing nodes can be thought of as nodes belonging to the same neural network and communicating with each other to transmit decoded data within / for the neural network. For example, to perform complex computations, one or more layers can be executed on a first device (e.g., a device on the mobile device 510), and one or more layers can be executed on another device (e.g., a cloud server on the cloud 590). However, the distribution can also be more granular, with a single layer executed on multiple devices. In this invention, the term "multiple" refers to two or more. In some existing schemes, a portion of the neural network function is executed on a device (user device or edge device, etc.) or multiple such devices, and then the output (feature map) is transmitted to the cloud. The cloud is a collection of processing or computing systems located outside the device, where the part of the neural network is being operated. The concept of collaborative intelligence is also extended to model training. In this case, data flows bidirectionally: from the cloud to the mobile device during backpropagation in training, and from the mobile device to the cloud (e.g., during training and forward propagation in inference). Figure 5 (As shown).

[0107] Several research works have proposed semantic image compression by encoding deep features and then reconstructing the input image from these features. First, uniform quantization-based compression is shown, followed by context-based adaptive arithmetic coding (CABAC) based on H.264. In some scenarios, it may be more efficient to transfer the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590 than to send compressed natural image data to the cloud and perform object detection using the reconstructed image. Therefore, compressing the data (features) generated by the mobile end 510, which may include a quantization layer 520 for this purpose, may be advantageous. Correspondingly, the cloud 590 may include an inverse quantization layer 560. Efficient compression of feature maps is beneficial for the compression and reconstruction of images and videos, for both human perception and machine vision. Entropy coding methods (such as arithmetic coding) are popular methods for compressing deep features (i.e., feature maps).

[0108] Today, video content accounts for over 80% of internet traffic, and this proportion is expected to increase further. Therefore, building efficient video compression systems and generating higher-quality frames within a given bandwidth budget is crucial. Furthermore, most video-related computer vision tasks (such as video object detection or video object tracking) are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Simultaneously, techniques in video compression also contribute to action recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as mentioned above. While each module is well-designed, the entire compression system is not end-to-end optimized. It is hoped that by jointly optimizing the entire compression system, video compression performance can be further improved.

[0109] End-to-end image or video compression DNN-based image compression methods can leverage large-scale end-to-end training and highly nonlinear transformations, which is unavailable in traditional methods. However, directly applying these techniques to build an end-to-end learning system for video compression is not easy. First, learning how to generate and compress motion information for video compression remains an unsolved problem. Video compression methods heavily rely on motion information to reduce temporal redundancy in video sequences.

[0110] A simple approach is to use learning-based optical flow to represent motion information. However, current learning-based optical flow methods aim to generate the flow field as accurately as possible. For specific video tasks, accurate optical flow is often not optimal. Furthermore, optical flow significantly increases the amount of data compared to motion information in traditional compression systems, and directly applying existing compression methods to compress optical flow values ​​would significantly increase the number of bits required to store motion information. Secondly, it is unclear how to construct a DNN-based video compression system by minimizing both residual and motion information based on rate-distortion objectives. Rate-distortion optimization (RDO) aims to achieve higher reconstructed frame quality (i.e., less distortion) given a certain number of bits (or bitrate) used for compression. RDO is crucial for video compression performance. To leverage the end-to-end training capabilities of learning-based compression systems, RDO strategies are needed to optimize the entire system.

[0111] exist Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, Zhiyong Gao ; DVC: An End-to-end Deep Video Compression Framework Framework) . IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR) In 2019, Pages 11006-11015 In this paper, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual coding.

[0112] This type of encoder is like Figure 6A shown. Specifically, Figure 6A The overall structure of an end-to-end trainable video compression framework is shown. To compress motion information, the CNN is specified to incorporate optical flow. Transform into a corresponding representation suitable for better compression Specifically, an autoencoder-style network is used to compress optical flow. Motion vector (MV) compression networks, such as... Figure 6B As shown. Network architecture and Figure 4 The terms ga / gs are somewhat similar. Specifically, optical flow... The input consists of a series of convolutional operations and nonlinear transformations, including GDN and IGDN. The number of output channels *c* for the convolution (deconvolution) is 128 in this example, except for the last deconvolutional layer, which is equal to 2 in this example. The kernel size is *k*, for example, *k=3*. Given an optical flow of size *M* × *N* × 2, the MV encoder will generate a motion representation of size *M / 16* × *N* × 16* × 128. Then, the motion representation is quantized (Q), entropy encoded, and used as... The data is sent to the bitstream. The MV decoder receives the quantized representation. Reconstruct motion information using MV encoder Typically, the values ​​of k and c can differ from the examples known in the art above.

[0113] Figure 6C The structure of the motion compensation section is shown. Here, the previous reconstructed frame x is used. t–1 Based on the reconstructed motion information, the warping unit generates a warped frame (typically using an interpolation filter, such as a bilinear interpolation filter). Then, a separate CNN with three inputs generates the predicted image. The architecture of the motion-compensated CNN is also as follows... Figure 6C As shown.

[0114] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to transform the residual into the corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this method can better utilize the power of nonlinear transformations to achieve higher compression efficiency.

[0115] As can be seen from the overview above, CNN-based architectures can be applied to both image and video compression, taking into account the different parts of the video framework, including motion estimation, motion compensation, and residual coding. Entropy coding is a common method for data compression, widely adopted in the industry, and also suitable for feature map compression for human perception or computer vision tasks.

[0116] Machine video encoding Video coding for machines (VCM) is another popular area of ​​computer science. The main idea behind this approach is to transmit coded representations of image or video information for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. Unlike traditional image and video decoding aimed at human perception, VCM focuses on performance characteristics for computer vision tasks, such as object detection accuracy, rather than the quality of the reconstruction.

[0117] A recent study proposes a new deployment paradigm called collaborative intelligence, which uses a deep model partitioning between mobile and cloud. Extensive experiments across various hardware configurations and wireless connectivity modes revealed that the optimal operating point involves partitioning the model, typically located deep within the network, in terms of energy consumption and / or computational latency. The current common approach, where the model resides entirely in the cloud or entirely on the mobile device, has rarely been found to be optimal (if any). The concept of collaborative intelligence is also extended to model training. In this context, data flows bidirectionally: from the cloud to the mobile device during backpropagation in training, and from the mobile device to the cloud during both training and forward propagation in inference.

[0118] In the context of recent deep models used for object detection, lossy compression of deep feature data based on HEVC intra-frame coding has been investigated. It has been pointed out that detection performance degrades with increasing compression levels, and compression-enhanced training has been proposed to minimize this loss by generating models more robust to quantization noise in feature values. However, this is still not optimal because the codec used is highly complex and optimized for compression of natural scenes, rather than for compression of deep features.

[0119] One approach uses the popular YOLOv2 network for object detection tasks to investigate the balance between compression efficiency and recognition accuracy, thus addressing the deep feature compression problem in collaborative intelligence. Here, the term "deep feature" has the same meaning as "feature map." The term "deep" comes from the concept of collaborative intelligence, which involves capturing the output feature map of a hidden (deep) layer and transmitting it to the cloud for inference. This is more efficient than sending compressed natural image data to the cloud and performing object detection using the reconstructed image.

[0120] Efficient feature map compression is beneficial for image and video compression and reconstruction, applicable to both human perception and machine vision. The limitations of state-of-the-art autoencoder-based compression methods also apply to machine vision tasks.

[0121] Functional modules Variable bit rate module Encoders can output bitstreams at different bitrates. Therefore, in some methods, the output of the encoding network is scaled (e.g., each channel is multiplied by a corresponding scaling factor, also known as the target gain value), and the input of the decoding network is inversely scaled (e.g., each channel is multiplied by the reciprocal of the corresponding scaling factor, also known as the target inverse gain value), such as... Figure 7 As shown. The scaling factor can be preset. Different quality levels or quantization parameters correspond to different target gain values. Scaling the output of the encoding network to a smaller value can reduce the bitstream size. Otherwise, the bitstream size may increase.

[0122] Color format conversion RGB and YUV are common color spaces. Conversion between RGB and YUV can be performed according to equations specified in standards such as CCIR 601 and BT.709.

[0123] Independent structures of luminance and chromaticity Some VAE-based codecs use the YUV color space as the input of the encoder and the output of the decoder, such as... Figure 8As shown, the Y component represents luminance, and the UV components represent chrominance. The resolution of the UV components can be equal to or lower than that of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted into a feature map F_Y via a network, and the entropy coding module generates the Y component bitstream based on the feature map F_Y. The UV component is converted into a feature map F_UV via another network, and the entropy coding module generates the UV component bitstream based on the feature map F_UV. In this structure, the feature maps of the Y and UV components can be quantized independently, allowing for flexible bit allocation for luminance and chrominance. For example, for color-sensitive images, the quantization of the UV component feature maps can be reduced, and the number of bits in the UV component bitstream can be increased to improve the reconstruction quality of the UV components and obtain better visual effects.

[0124] In some other methods, the encoder concatenates the Y and UV components and then sends them to the UV component processing module (for converting image information into feature maps). Furthermore, the decoder concatenates the reconstructed feature maps of the Y and UV components and then sends them to the UV component processing module 2 (for converting the feature maps into image information). In this method, the correlation between the Y and UV components can be used to reduce the bitstream of the UV components.

[0125] In this specification, "parameters" are values ​​used in the operations of each layer of a neural network, and may include, for example, weights used when input values ​​are applied to a particular operational expression. Here, parameters can be represented in matrix form. Parameters are values ​​set during training and can be updated as needed using individual training data.

[0126] Figure 9 A specific example of an encoder provided by the currently disclosed embodiments is described. Specifically, the figure illustrates the encoding process for a single component (e.g., the Y or UV components of an image, representing luminance and chrominance, respectively). The per-component encoder is a multi-step process. The first step is the analytical transform. The analytical transform receives two inputs. (The signal to be encoded) and (Auxiliary information to aid encoding). Analyze the transformation output latent tensor. The size of the tensor depends on the height of the input image. and width and principal components ( ) and auxiliary components ( The scaling factor is determined. For the principal component, the parameter... =0, which means that the analysis transformation of the principal component does not receive any auxiliary information (independent encoding). For the auxiliary component, the parameter... =1. For the analytical transformation of the auxiliary components, the auxiliary information is a tensor. It is factored Principal component tensor Resampling.

[0127] The output of the analysis transform is fed into the superencoder. This superencoder generates a hyperparameter tensor. Round it to the nearest hundredth. Reshaped into a 1D array and passed through a lossless encoder ( Encoding, thus forming . The probability distribution of lossless decoding is defined by pre-trained parameters (a portion of the trained model) in the lossless entropy encoder ( me-tANS encoder The cumulative distribution function (denoted as) is calculated based on these pre-trained parameters in the ) ).

[0128] Then, the same steps as the decoder operation are performed at the encoder end (such as...). Figure 10 (as shown), to generate for The entropy parameter of the encoding. The 3D tensor is encoded in the encoder SKIP module. Convert to the set of symbols to be encoded { s} can be skipped The value is encoded.

[0129] Figure 10 A specific example of a decoder provided by the currently disclosed embodiments is described. Specifically, the figure illustrates the decoding process for a single component (e.g., the Y or UV components of an image, representing luminance and chrominance, respectively). The codestream consists of a main component and a secondary component. and Composition. For the primary and secondary color components, the bitstream can be parsed independently and reconstructed using modules composed of the same sequence of the same neural network layers, the only difference being the size of the input tensor and the number of tensor channels. Figure 10 This is the kind of separable decoding described in the text.

[0130] First, it will be passed through a lossless entropy decoder ( Analysis . The probability distribution of lossless decoding is defined by pre-trained parameters (a portion of the trained model), and the cumulative distribution function (calculated based on these pre-trained parameters) is used in the lossless entropy decoder. Figure 10 The Chinese character is represented as ).

[0131] Decoded super-prior tensor Used as input for two different processes: superdecoder (part 11.2) and superscale decoder (part 10.3).

[0132] Then by using the lossless decoder ( Analysis Analysis Assuming the probability distribution is Gaussian, if the following steps are performed, the output will be zero mean and standard deviation: The superscale decoder in the logarithmic domain The output standard deviation tensor is then used to calculate the rate control parameters within the σ scale. Scale it to produce Then, based on the RVS parameter portion within the adaptive σ scale, it is masked and scaled to produce... Finally, regarding tensors... The values ​​are quantized (converted to indices in the probability distribution table). In the decoder SKIP module, some elements of the residual tensor may be skipped (not encoded / decoded) and replaced with zeros. This decoder SKIP module receives the parsed set of syntax elements from the tANS decoder (Part 9.5). s} Receives mask_sigma from the SKIP mask generation module and outputs the residual tensor of the reshaped 3D shape reconstruction. .

[0133] At the decoding end, residual The inverse gain unit is based on the parameters Scaling is performed to produce The residual tensor is then scaled in the inverse residual and variance scale (invRVS) module (part 13.2.3) to form the residual tensor. This residual tensor is used for the reconstructed latent tensor. Super decoder generation The input is fed into a multi-stage context model (MCM), which is an eight-stage neural network process that also reconstructs the residuals. As input, and output the latent space tensor After latent scaling before synthesis (LSBS), the reconstructed latent space tensor Prepare for signal reconstruction. The latent tensor reconstructions of the principal and secondary components are independent of each other.

[0134] exist Figure 9 and Figure 10In this context, it should be noted that the rate control parameter β controls the compression ratio and defines the operation in each of the gain units, σ-scales, and inverse gain units shown in these figures. The positive gain tensor m is used for the encoding end in the gain unit. The logarithmic scale... m log The positive gain tensor is used for the σ-scale. The inverse gain tensor... 24 Used for the decoding end in the inverse gain unit. All three positive gain tensors m, and the inverse gain tensor... 24 Sum of logarithmic domain gain tensors m log The size [C, h4, w4] is equal to the size of the residual tensor.

[0135] In cases where the block size is determined as discussed herein, generating the bitstream further includes including in the bitstream an indication of the size of the blocks in a first plurality of blocks and / or an indication of the size of the blocks in a second plurality of blocks.

[0136] The encoding processing of an input tensor has its corresponding decoding processing, sharing functional correspondences in the processing. In this exemplary and non-limiting embodiment, a method for decoding a tensor representing image data is provided. The tensor may be in matrix form, having a width = w and a height = h in two spatial dimensions, and a third dimension (e.g., depth or number of channels) of size D. The method includes processing the input tensor representing the image data using a neural network comprising at least a first subnet and a second subnet. It should be noted that the width and height of the input tensor of the decoder may differ from the width and height of the input tensor processed by the encoder. Furthermore, as will be apparent to those skilled in the art, the first and second subnets of the decoder may fully or partially (e.g., functionally) perform inverse functions as the first and second subnets of the encoder. However, the inverse function cannot be interpreted strictly mathematically. In practice, the term "inverse" refers to the processing performed to decode the tensor in order to reconstruct the original image data. Those skilled in the art will understand that encoding compression and decoding decompression may also include processing that may not be necessary for decoding and / or encoding. For example, Figure 11 The RDOQ shown is encoder-only processing. It should be noted that the terms "first" and "second" subnets are merely labels to distinguish the subnets of the decoder (and for this purpose, the encoder described above).

[0137] Figure 11An example of a first subnet and / or a second subnet for an encoding branch is shown, including a decoder 1604 and a post-filter 1611. In the method, the processing includes: applying a first subnet to a first tensor, including dividing the first tensor into a first plurality of blocks in a spatial dimension, and processing the first plurality of blocks by the first subnet; after applying the first subnet, applying a second subnet to a second tensor, including dividing the second tensor into a second plurality of blocks in a spatial dimension, and processing the second plurality of blocks by the second subnet. Figure 11 In the example, the first subnet is decoder 1604, whose output is provided as input to the second subnet, which is post-filter 1611. Here, the first tensor is the quantized feature tensor in the latent space. The arithmetic decoder 1606 pre-decodes the data from bitstream 1. Then, it inputs the second tensor into the second subnet 1611 for post-filtering. These are feature tensors, such as features in a feature map or a feature map in the latent space. Therefore, similar to the encoder's processing, the type of input tensor (e.g., first tensor and second tensor) may depend on the processing performed on the previous subnet. Figure 11 In the example, the preceding subnet is encoder 1604. The term "after" does not limit the above-described decoding process to immediately following the direct input of the output of the first subnet to the second subnet. Rather, "after" means, for example, processing the first or second plurality of blocks in a certain temporal order within the same pipeline, which may not be immediate in time.

[0138] Furthermore, the type of input can also depend on the layers on which the processed input data can branch out to serve as input to another (e.g., a subsequent) subnet (e.g., a layer in a neural network (NN) or a layer in an untrained network). For example, such a layer could be the output of a hidden layer. Similar to the encoding process, the first and second tensors are divided into blocks in the decoding process, which have been defined above.

[0139] exist Figure 11 In this context, decoder 1604 can be the first subnet with processing 1 to processing N (processing pipelines 1 to N). For example... Figure 11 As shown, the decoder 1604 will input the feature tensor Divide into N blocks to (First, multiple blocks). Then, the corresponding tensor blocks are processed by the corresponding blocks, i.e., processing 1 to processing N. These blocks do not need to interact with each other (e.g., wait for each other during processing). The result of each processing provides N blocks. to (Tensor). After processing the first plurality of blocks, the decoder subnet 1604 can merge the processed blocks into the first output tensor. .exist Figure 11 In this context, the first output tensor can be a second tensor used as input by the post-filter 1611. The post-filter 1611 can be a second subnet with processes 1 to N (process pipelines 1 to N). Similar to the decoder 1604, the post-filter 1611 will input the tensor... Divided into a second or more blocks to These are processed by processes 1 through N of the post-filter 1611, respectively. Processes 1 through N provide corresponding blocks. to As output, it can be merged into blocks. .exist Figure 11 In the example, merged This refers to the decoded tensor representing the reconstructed image data. Merging may (but is not required to) involve, for example... Figures 12 to 16 The clipping shown is correct. It should be noted that merging / combining into a tensor is not required. or It's conceivable that the second subnet reuses the blocks of the first subnet, modifying them only (either by further subdividing the blocks to refine them, or by combining multiple blocks into a single block to coarsen them). Figure 11 In the example, processes 1 through N can be performed on a block-by-block basis, i.e., process i processes block i. Alternatively, process i can process component i from multiple components of the input tensor. In this case, process i divides component i into multiple blocks and processes these blocks individually or in parallel.

[0140] In some exemplary implementations, the first subnet may be a superdecoder or a combination of a superdecoder and a context modeling network (e.g., multistage context modeling, MCM), and the second subnet may be a combination of a context modeling network (e.g., multistage context modeling, MCM) and a synthesis transform, or only a synthesis transform or only a context modeling network.

[0141] Furthermore, at least two of the corresponding juxtaposed blocks in the first and second plurality of blocks are of different sizes. In other words, the way the subdivision into blocks may differ for each subnet, so each subnet can use different block sizes. However, the input of each subnet (i.e., the first and second tensors) is divided into a grid of blocks with the same size, except for the blocks at the bottom and right image boundaries, which may have smaller sizes.

[0142] Furthermore, the attributes and / or characteristics of the first and second plurality of blocks used in the decoding process are similar to one of the encoding processes discussed earlier. Specifically, in the exemplary implementation described above, adjacent blocks in the first plurality of blocks partially overlap in at least one spatial dimension; and / or adjacent blocks in the second plurality of blocks partially overlap in at least one spatial dimension. Examples of adjacent blocks and partial overlap are as follows: Figure 12 , Figure 13 , Figure 14 , Figure 15 and Figure 16 As shown.

[0143] Furthermore, in some exemplary implementations, blocks within the first plurality of blocks are processed independently by a first subnet; and / or blocks within the second plurality of blocks are processed independently by a second subnet. In other words, the processing of blocks is independent of each other and therefore parallelizable. In one example, at least two blocks within the first plurality of blocks are processed in parallel by the first subnet; and / or at least two blocks within the second plurality of blocks are processed in parallel by the second subnet. Figure 11 The parallel processing at the decoding end is illustrated, where the components of the first input tensor (e.g., block and / or spatial components) are processed. to Processed by decoder 1604, with no interaction between the processing of components 1 through N. The result of each processing is an output tensor component. to They can be further combined into an output tensor. The second subnet can be... Figure 11 The post-filter 1611 in the middle takes the tensors from decoder 1604 (first subnet) As input. In this example, the input of the post-filter 1611 is directly connected to the output of the decoder 1604. Or, Figure 11 Further processing can occur between the post-filter and the decoder. Input tensor The post-filter is divided into N blocks. to Then, the post-filters process the corresponding blocks independently or in parallel. Parallel processing is reflected in... Figure 11 In this process, the processing of components 1 to N can be independent of each other. The result of parallel processing of the post-filter is the reconstructed blocks. to These reconstructed blocks can be merged into a tensor representing the reconstructed image data. .

[0144] Furthermore, the partitioning of the first tensor includes determining the size of a block within a first plurality of blocks based on a first predefined condition; and / or the partitioning of the second tensor includes determining the size of a block within a second plurality of blocks based on a second predefined condition. For example, the first and / or second predefined conditions are based on available decoder hardware resources and / or motion present in the image data, or other features as described above with reference to the encoding.

[0145] The first and second subnets used for decoding processing can have a similar configuration to one or more subnets used for encoding processing. Specifically, the first subnet performs processing through one or more layers including at least one convolutional layer or at least one pooling layer; and / or the second subnet performs processing through one or more layers including at least one convolutional layer or at least one pooling layer. Figure 6A and Figure 6B The decoder within a neural network-based VAE framework is shown, with corresponding convolutional layers, where the input tensor (feature map) is... The corresponding size is then magnified by a factor of 2 (upsampling). Furthermore, the first and second subnets perform corresponding processing as part of the image or moving image decompression. This processing can be performed by… Figure 6A and Figure 6B The VAE encoder shown provides the feature tensor. As input, the image is decompressed and reconstructed; as output, the reconstructed image data (i.e., the decoded tensor) is represented. For example, the first subnet and / or the second subnet perform one of image decoding and image filtering in the convolutional subnet.

[0146] In one implementation, the input tensor is an image or image sequence comprising one or more components, at least one of which is a color component. Alternatively, the input tensor may be a latent spatial representation of the image, or it may be a preprocessed output (e.g., an output tensor). One or more components are color components and / or depth and / or motion maps and / or one or more other feature maps associated with image samples. The input tensor has at least two components, namely a first component and a second component; a first subnet divides the first component into multiple blocks and the second component into additional multiple blocks, wherein at least two corresponding juxtaposed blocks among the multiple blocks and the additional multiple blocks have different sizes; and / or a second subnet divides the first component into multiple blocks and the second component into additional multiple blocks, wherein at least two corresponding juxtaposed blocks among the multiple blocks and the additional multiple blocks have different sizes.

[0147] Figure 12 An example is shown of dividing a first tensor and / or a second tensor into overlapping regions Li (i.e., the first block and the second block), subsequently cropping samples within the overlapping regions, and then stitching the cropped regions together. Each Li includes the total receptive field.

[0148] Figure 13 It shows the relationship with Figure 12 Another similar example divides the first and / or second tensors into overlapping regions Li (i.e., the first block and the second block), except that clipping is removed.

[0149] Figure 14 An example is shown of dividing a first tensor and / or a second tensor into overlapping regions Li (i.e., the first block and the second block), subsequently cropping samples within the overlapping regions, and then stitching the cropped regions together. Each Li includes a subset of the total receptive field.

[0150] Figure 15 An example is shown of dividing the first and / or second tensors into non-overlapping regions Li (i.e., the first block and the second block), followed by cropping the samples and stitching the cropped regions together. Each Li includes a subset of the total receptive field.

[0151] Figure 16 Various parameters are shown, such as the sizes of regions Li, Ri of the first and / or second blocks, and overlapping regions, which can be included in (and parsed from) the bitstream. Any of these parameters can be included in (and parsed from) the bitstream as an indication. According to some embodiments, the block size or the number of blocks (segments) in the horizontal / vertical direction can be explicitly used. According to other embodiments, the size of the overlapping region can be explicitly indicated. Figure 16 The image shown is divided into only four equal parts. It should be understood that an image can be divided into many tiles. Therefore, when using more than four tiles, the size can be mapped to W1+x, which is the indicated tile_size, and the indicated overlap can be x+x. Then, half of the overlap (one x) is discarded on each side of the tile to remove or avoid artifacts.

[0152] Figure 17 This diagram illustrates a portion of the JPEG AI decoding scheme. The entire scheme comprises analysis at the encoding end and synthesis at the decoding end. In the JPEG standard, prediction and residuals exist. Prediction modules exist at both the encoding and decoding ends. A synthesis module also exists at the decoding end. In the JPEG standard, the block-based processing method exists only in the synthesis module at the decoder end. This process processes each block separately, where, after synthesis, the blocks are combined to recreate the final image (or the input to the post-filter).

[0153] Currently, block division is only used in the synthesis module. In the synthesis module, the tensor y is divided into several sub-tensors, forming blocks, and then the synthesis network is applied block by block. This is primarily to save memory. The synthesis network is quite complex, and this inference requires a considerable amount of memory. However, on some devices, available memory is limited. Therefore, the tensor y can be divided into several smaller tensors, called blocks. Inference can then be performed block by block. After synthesis, the inference results of each sub-tensor can be combined to form the output.

[0154] Currently in the standard, composition is performed only in one module, namely the composition module. Chunking is typically not used in other modules. Therefore, only one size of the chunk needs to be specified, and only one chunk can overlap.

[0155] Specifically, in the current context of still image decoding, blocks are implemented in the latent space y. The latent space can also be called the feature space. In JPEG AI decoding, there are two feature spaces: the latent space y^ and the hyper-latest space z^. The network input can be divided into blocks and combined using overlap. Overlap is used because inference near block boundaries in a neural network often produces artifacts. Each block undergoes inference. By removing the amount of overlap, artifacts can be removed.

[0156] Two blocks are recombined using overlapping regions. When the image is reconstructed, these blocks are cropped so that the region between the midpoint of the overlap and the edge of each block is removed. That is, cropping occurs from the middle of the overlapping region to the nearest block edge. Therefore, a portion of the image can be reconstructed without artifacts at the boundaries caused by inference. The overlapping and cropping of the blocks also has a smoothing effect. Therefore, if the overlap is large enough, there are no artifacts at the boundaries between blocks. Thus, when the overlap is large enough, the boundaries between blocks are invisible.

[0157] The required overlap may vary depending on the image. For some images, a large overlap is unnecessary. For others, even a small overlap may be sufficient to remove all boundary artifacts. It is preferable to use the smallest possible overlap while removing as many boundary artifacts as possible. This is because overlap introduces additional complexity to the process due to processing more samples (processing some samples of the input tensor twice). However, for some images, a large overlap is required, making this additional complexity unavoidable. Therefore, it has been recognized that it is beneficial not to fix the overlap size, but rather to use an indicator to indicate the overlap size.

[0158] The typical syntax for chunking indication is shown below. A flag indicates whether chunking is enabled or disabled. If chunking is enabled, the chunk size and the size of the overlapping region are indicated. The overlap is always the same size in both the vertical and horizontal directions. The size of the boundary can be selected based on a metric for the overlap, such as peak signal-to-noise ratio (PSNR), multi-scale structural similarity index measure (MS-SSIM), video multimethod assessment fusion (VMAF), etc., which can be measured against a predetermined threshold. If the threshold is exceeded, the size of the overlap may increase, and vice versa.

[0159]

[0160] Figure 17 Two variations of the block arrangement scheme that can be used in image decoding are also shown. In this example and other examples described herein, block arrangement is used at the decoding end of image processing. However, block arrangement can also be used at the encoding end and in other modules besides those discussed herein. The two types of block arrangement schemes shown are juxtaposed block arrangement and hierarchical block arrangement. As can be seen, in these schemes, blocks are required in different modules. Specifically, in this example, block arrangement is used in the latent prediction module and the synthesis module. The first example variant scheme is juxtaposed block arrangement, starting with a tensor z^ of a specific size. Then, latent prediction is performed using z^, producing y^ of a specific size. The image data is recombined through subsequent synthesis steps of y^. That is, each z^ block is processed one by one by the synthesis module. The latent space y^ information is not used for the next z^ block.

[0161] The second example variant is a hierarchical block-based scheme. In this scheme, z^ consists of larger blocks, allowing a single prediction across the entire block at the prediction module, resulting in a larger region. The larger region is then divided into smaller y^ regions and synthesized. The y^ blocks are smaller because the synthesis network is much more complex, requiring more memory for inference, thus necessitating smaller blocks. However, latent prediction is a relatively shallow metric, so the required memory is not large, and the device can process larger blocks during processing.

[0162] Figure 18Two variations of the chunking scheme are shown in more detail. In this example, the two modules handling chunking are the prediction module and the synthesis module. Here, it can be seen that the chunks have corresponding sizes. However, the resolution of z^ is smaller. For example, a 16x16 region in z^ corresponds to a 64x64 region in the synthesis (y^). There is no hierarchical chunking; for example, a large chunk in y^ is divided into multiple smaller chunks. They are called juxtaposed chunks because the sizes of these chunks correspond to each other. Although the different modules have different resolutions, the chunk size in z^ can be derived from the chunk size in y^ simply by division. The chunk size in y^ is a multiple of this constant, representing the relationship of resolution differences between modules, not the number of chunks.

[0163] The second partitioning scheme illustrates hierarchical partitioning. In this example, the prediction module uses larger blocks. Each larger block can then be divided into smaller blocks for use in the next module (e.g., the synthesis module). The advantage of using hierarchical partitioning is that larger blocks can be used by subnets with lower memory requirements. That is, some modules that run less complex processes can apply the process to a larger amount of data at once. For example, the prediction module can handle more data and therefore can use larger blocks. Other modules that apply different subnets, such as the synthesis module, require more memory to process the same regions. Therefore, smaller blocks are needed for this operation. In the case of juxtaposed partitioning, the size of the blocks used for prediction needs to be the same as the size of the blocks used for synthesis. The synthesis module provides a limiting factor for the block size. However, this means that the prediction module needs to process more blocks. This introduces higher complexity to the prediction module and requires more processing for an increased number of overlapping regions. If the resolution in z^ is four times in each dimension, and for normal reconstruction, block overlap is required to be at least, for example, 3 or 4 samples, then using juxtaposed blocks with 4 samples of overlap in z^ will result in 16 samples of overlap in the synthesis domain. This results in more overlap than is needed, leading to a significant increase in complexity within the synthesis module.

[0164] Therefore, by specifying the block overlap of one module separately from the overlap of the second module, it is possible to achieve the same size of sample overlap for the two modules even if they have different resolutions and block sizes.

[0165] The only drawback of using hierarchical chunking is the introduction of additional latency. This latency occurs when a second module (e.g., the synthesis module) must wait for the entire larger chunk processed by the first module (e.g., the prediction module) to be output by that first module. This small time delay is a trade-off for reducing the processing memory required to implement chunking in two consecutive modules.

[0166] In existing block-segmentation schemes, only one module in a neural network uses blocks. This is typically the synthesis module. Therefore, it is not currently necessary to specify whether the indicated block size or block overlap applies to any particular module. However, blocks can be used in multiple different modules or subnets of a neural network, with different block sizes used in each module. For z^ in the hyperlatent space (the input to the prediction module), the resolution is four times smaller than y^ in the latent space (the input to the synthesis module).

[0167] Therefore, proposals for implementing different chunk sizes and chunk overlaps in different modules need to indicate which size and overlap to use in which module. This can be done explicitly. However, it can also be done by specifying a factor difference between the values ​​of one module and the values ​​of another module. This will depend on whether the chunks in the corresponding module are juxtaposed or hierarchical, as explained above.

[0168] Below are examples of chunk headers used to implement the various proposed modular chunking schemes. This indicative data is referred to herein as a chunk header. However, it should be understood that any suitable name can be given to this data structure to distinguish it from a single chunk, a group of chunks, an image tensor, or the parametric data of an image tensor. According to some embodiments, syntax elements can explicitly indicate the chunk size. According to other embodiments, syntax elements can indicate the number of chunks (segments) in the horizontal / vertical direction. In this way, the chunk size can be derived from the spatial resolution and the number of chunks (segments). According to other embodiments, the size of the overlapping region can be explicitly indicated.

[0169] The first difference between the existing tile header and the proposed instruction header is the specification of the tile type or module. For example, in the tile header syntax below, tile_enable, tile_size, and tile_overlap now contain 'y' to indicate that these are the relevant values ​​for the y^ tile to be used as input to the composition module. This information is handled in no different way than in the previous tile header; it simply allows the assumption that these values ​​are used as input to the composition module.

[0170] To implement chunking across multiple modules, a second set of parameters needs to be specified, specifying which module these parameters belong to. Therefore, the proposed chunking header now includes a second set of fields containing the chunking parameters (e.g., z^) for another module. In this example, it should be noted that the first set of parameters corresponds to the second module in the order of the neural network image data processing chain. This is not a requirement, as described below.

[0171] The proposed tile header syntax now also includes the `tile_z_enable`, `tile_z_derived`, `tile_z_size`, and `tile_z_overlap` fields. The `tile_z_enable` field functions the same as the typical `tile_enable` field in a tile header and indicates whether the attached subnet uses tiles. In this example, the `tile_z_enable` field includes a 1-bit flag that is set to "1" if the module is used to use tiles, and set to "0" if the module is not used to use tiles. Similar to composite tiles, in some embodiments, the explicit name of the module can be added to the syntax element name instead of adding a letter ('y' or 'z'). Specifically, specific names such as `hyper_decode_tile_enable`, `hyper_decode_tile_size`, and `hyper_decode_tile_overlap` or `mcm_tile_enable`, `mcm_tile_size`, and `mcm_tile_overlap` can be used instead of `tile_z_enable`, `tile_z_size`, and `tile_z_overlap`.

[0172] The next field after the tile header is a new field. This field can be called `tile_z_derived`. This field can be used to indicate whether the tile size and tile overlap can be derived from the values ​​of another module. As mentioned above, when tiles in adjacent modules are juxtaposed, the only difference is the corresponding resolution, which associates the tiles only through a scale factor. Therefore, if tiles are juxtaposed, the tile size and tile overlap of one module can be represented by a simple multiplication or division of the values ​​used by the other module. In this example, the prediction module can be provided with z^ tiles of a specific size and overlap for the y^ synthesis module based on the values ​​in this field. However, if these tiles are hierarchical tiles, then a tile in one module also contains image data at different scales compared to tiles in another module, so their relationship cannot be represented by a simple multiple difference. In this case, the tile size and tile overlap must be explicitly specified. Therefore, a field is proposed that indicates whether the tile size and tile overlap are derived from other specified values. For example, taking the z^ and y^ blocks of the prediction and synthesis modules as examples, if `tile_z_is_derived` is set to "1", the z^ block value can be derived from the values ​​of `tile_y_size` and `tile_y_overlap`. This is because the y^ and z^ blocks are co-located. If `tile_z_is_derived` is set to "0", the z^ block value cannot be derived from the values ​​of `tile_y_size` and `tile_y_overlap`. This is because the y^ and z^ blocks are hierarchical blocks.

[0173] When the tile is a layered tile, the tile size and tile overlap value of the second module must be explicitly indicated. This is achieved by providing two additional fields (tile_z_size and tile_z_overlap) in the tile header.

[0174] In some cases, the block parameters in y^ might be derived from the block parameters in z^. While this might not be particularly practical, since block decomposition is almost always used in synthesis operations and less so in prediction operations, it's rare to use block decomposition in prediction but not in synthesis. However, there's no reason why this shouldn't be the case in principle. Therefore, the derivation shown in the example described in this paper can be derived in reverse, where the z^ block parameters in the hyperlatent space are derived from the y^ parameters in the latent space. Since this is merely an example of one implementation of the proposed indication method, it is equally applicable to the indication of block parameters for completely different subnets using block-level processing.

[0175] Similarly, currently only cases exist where hierarchical chunking consists of larger chunks in the z^ domain, while smaller chunks are used in the composition operation. Larger chunks indicate that each chunk represents a larger proportion of the total tensor data than the chunks used in different operations, and this is resolution-independent.

[0176] The proposed indication also requires a different number of bits than existing indications. This is because, currently, tile size and overlap size are indicated in the image domain. The proposed indication can specify tiles in the latent space y^, which has a smaller resolution than the image space, thus only needing to cover a reduced numerical range. Furthermore, tiles with fractional samples or fractional overlapping regions are not possible. Therefore, indicating them in the image space is meaningless and can be replaced by the latent space or hyper-latent space. Thus, as the resolution decreases, fewer bits can be used to indicate tile size and tile overlap. The existing number of bits used to represent tile_size is 13 bits. In the proposed indication syntax, tile_y_size requires at most 9 bits. When indicating tile_overlap, the current syntax requires 8 bits. By representing tile_y_overlap in the latent space, the explicit value of tile overlap can be represented using only 4 bits. This will significantly reduce the number of bits used for indication across the entire image. Furthermore, in the derivational case, another set of tile parameters for another module can be specified using only one additional bit. In cases where derivation is not possible, the second set of block parameters can be explicitly indicated using only 7 bits for block size and 3 bits for block overlap, since the z^ block is represented in a hyperlatent space with a resolution smaller than that of the latent space and the y^ block. Even though the resolution-dependent block overlap of the z^ block is four times smaller than that of the y^ block, the difference in the number of bits used to specify the block overlap is not 2 (4–2). This is because the block overlap has a minimum size, and its usability is limited by this size. Therefore, it must be possible to specify different block overlaps for different subnet modules, which are not necessarily proportional to the difference in resolution.

[0177] To explain the proposed directive method, the default y^ block is placed at the top of the header, as it is assumed that the block will be used most frequently in the composition module, as in the current implementation. However, it should be noted that the proposed directive method is not restrictive; it requires that the y^ block parameters be present in the header before the z^ block parameters. The proposed directive method is also not restrictive; it requires that the parameters of the first module can be derived from the block parameters of the second module. The provided examples and the order of parameters in the block header and their derivation are merely for illustrative purposes and for a simple comparison with existing implementations of block composition.

[0178]

[0179] Below is an example of semantics, which may accompany the syntax presented above.

[0180] Multiple components are indicated. In the example scenario, Y and UV are used to represent the luminance and chrominance components, respectively. However, these components can be any appropriate components, and therefore can be simply referred to as primary and secondary components. Thus, in this particular example, the primary component could be luminance, and the secondary component could be chrominance.

[0181] In this description, the image tensor may not be provided at image resolution. For example, it may be provided in the latent space. That is, the latent space size for width and height is equal to W / 16 and H / 16, respectively, compared to the image resolution scale, where W and H are the image sizes in units of image resolution.

[0182] `tile_y_enable[comp]` is an enable flag used for the reconstructed potential tensor (y). _Y, y The main component (comp==0) and auxiliary component (comp==1) of _UV are divided into blocks.

[0183] tile_y_size[comp] is the reconstructed latent tensor (y _Y, y The block size of the principal component (comp==0) and secondary component (comp==1) of the latent space (UV). The indicated number is interpreted as the number of latent space samples along a block edge in the latent space block.

[0184] tile_y_overlap[comp] is the reconstructed latent tensor (y _Y, y The size of the block overlap between the principal component (comp==0) and secondary component (comp==1) of the _UV. The indicated quantity is interpreted as the number of potential spatial samples spanning the overlapping region from the block edge inward toward the center of the block.

[0185] tile_z_enable[comp] is the enable flag for the main component (comp==0) and auxiliary component (comp==1) of the superdecoder and multi-stage context modeling.

[0186] `tile_z_is_derived[comp]` is a flag indicating that `tile_z_size[comp]` and `tile_z_overlap[comp]` are derived from `tile_y_size[comp]` and `tile_y_overlap[comp]` respectively. The derivation is based on the following equation: tile_z_size[comp] = tile_y_size[comp] / 4 tile_z_overlap[comp] = tile_y_overlap[comp] / 4 In this case, tile_y_size[comp] and tile_y_overlap[comp] should be multiples of 4 (tile_z_is_derived[comp]='1').

[0187] `tile_z_size[comp]` is the tile size for the principal component (comp == 0) and secondary component (comp == 1) in the superdecoder and multi-stage context modeling. The indicated number can be interpreted as the number of hyperlatent space samples along a tile edge within the hyperlatent space tile. The tile size in multi-stage context modeling is set to equal to `tile_z_size[comp]`. 4.

[0188] `tile_z_overlap[comp]` is the size of the tile overlap region for the principal component (comp == 0) and secondary component (comp == 1) in the superdecoder and multi-stage context modeling. The indicated number can be interpreted as the number of hyperlatent space samples spanning the overlap region from the tile edge inwards towards the center of the tile. In multi-stage context modeling, the size of the tile overlap region is set equal to `tile_z_overlap[comp]`. 4.

[0189] The proposed block size and overlap indicators are interpreted as values ​​corresponding to the number of latent space samples. This has the advantage of correlation, as mentioned above regarding the required resolution and number of bits for indicating block size. According to some embodiments, the names of the corresponding syntax elements may include the suffix "in_latent_samples". For example, names such as hyper_decode_tile_size_in_latent_samples, hyper_decode_overlap_in_latent_samples, mcm_tile_size_in_latent_samples, and mcm_overlap_in_latent_samples can be used. In this way, it can be indicated that the indicated numbers correspond to the number of latent space samples.

[0190] In these examples, the modules used for the superdecoder and multistage context modeling (MCM) consistently use juxtaposed chunking because they do not require different memory configurations. However, the two modules have different spatial resolutions; the MCM module's resolution is four times that of the superdecoder. Therefore, to specifically calculate the chunk size and overlap of the MCM module based on a specified z^chunk size, the value must be multiplied by 4. Thus, for different modules with different resolutions, the equivalent relationships to be calculated can be set based on predefined numerical relationships.

[0191] The derivation can be done by dividing the block size, block overlap, or both by four. This division is based on the ratio of the resolution of a block in one module's space to the resolution of a block in another module's space. For example, the resolution of the z^ block in the hyperlatent space is four times smaller than the resolution of the y^ block in the latent space. Therefore, to derive the block size of the z^ block from the block size of the y^ block, the value in tile_y_size needs to be divided by 4. Here, the block size is a length of one unit in terms of the number of samples.

[0192] The proposed instruction scheme supports instruction juxtaposition and hierarchical block partitioning, which can be used in different application scenarios.

[0193] The proposed specification scheme supports specifying different block sizes and different overlap region sizes for different modules. This can be very helpful. For example, in the case of a juxtaposed block with a block overlap of 4 samples in the z^ domain, the block overlap becomes 16 samples in the y^ domain, and 256 samples in the image domain. 256 samples in the image domain is a large overlap, which leads to increased processing complexity. Being able to specify block overlaps separately for lower resolution and higher resolution domains means that smaller overlaps can be specified.

[0194] The proposed indication scheme supports the use of different scales for block size and overlap size in different modules, which saves the number of indication bits for hierarchical block partitioning. This can be seen in the syntax table proposed above, where z^blocks require fewer bits for indicating block size and overlap size than y^blocks.

[0195] Figure 19This is a flowchart illustrating an exemplary method for decoding images based on a neural network architecture. The bitstream includes information indicating the block size and / or block overlap to be used for one or more specific subnetworks of the neural network. The method described herein processes image tensors representing image data using a neural network. The neural network includes a first subnetwork and a second subnetwork. The processing includes indicating the size of the block portions of the image tensor processed by each subnetwork of the neural network in block header data (which may also be referred to as image header, tool header, image parameter set, sequence parameter set, etc.).

[0196] The first step S1910 of the method includes: processing the image data unit to identify a first field of a first predetermined type and a second field of a second predetermined type therein.

[0197] The second step S1920 of the method includes: determining the block size for the second block-level operation based on the value of the first field, and determining the block size for the first block-level operation based on the value of the second field.

[0198] The third step S1930 of the method includes: performing a first block-level operation on the data derived from the image data unit to generate intermediate data.

[0199] The fourth step S1940 of the method includes: performing a second block-level operation on the intermediate data.

[0200] In an exemplary embodiment, the first subnet may include the prediction module as described above. Similarly, the second subnet may include a synthesis module. In another exemplary embodiment, the first subnet may include a superdecoder (only a portion of the prediction module), and the second subnet may include a multistage context module (MCM) or a combination of an MCM and a synthesis module. Intermediate tensor data may include the y tensor described above. The first field may be the first field written in the exemplary syntax shown above. Therefore, the first predetermined type may be a chunk size value. The second field is a field that includes data of a second predetermined type. This second field includes data that can be used to determine the chunk size of the first subnet. In the exemplary implementation described above, the first subnet may be a prediction operation. Therefore, the second field will include information for determining the chunk size of z^. There are two ways to achieve this, one is by declaring an explicit value, as shown in the line using the syntax tile_z_size[comp]. Another way is to provide an indication that the value can be derived. As described above with reference to the exemplary implementation, in this case, the second field will be a flag provided by the syntax tile_z_is_derived[comp]. The above calculations can be performed by providing this instruction in the second field, thereby allowing the necessary block size to be derived from the reference value in the first field according to predetermined settings.

[0201] In some embodiments, the method may include the following steps. For example, the method may include: processing chunk header data to identify a third field of a third predetermined type therein. Then, based on the value of the third field, the method may continue by determining chunk overlap for applying a second subnet and, based on the value of the second field or a fourth field of a fourth predetermined type, determining chunk overlap for applying a first subnet.

[0202] The same approach described above for tile size can be applied to tile overlap. Here, the third field can therefore include an explicit value for tile overlap, for example, that would be used for the second module (e.g., the composition module). The second field can then be the same flag indicating the possibility of tile overlap being derived for the first subnet. The fourth field can be a field that specifies an explicit value for tile overlap in cases where it is impossible to derive it. For example, layered tile overlap is used in the first subnet. That is, the fourth field could be tile_z_overlap[comp].

[0203] As described above, the value of the second field includes an explicit value for the chunk size used to apply the first subnet. The method may include: determining the chunk overlap used to apply the first subnet based on the value of a fourth field, wherein the fourth field includes an explicit value for the chunk overlap. The second field may include a flag that, when set to '1', indicates that the first and second chunk-level operations include juxtaposed chunking, and when set to '0', indicates that the first and second chunk-level operations include hierarchical chunking. The value of the second field may indicate that the chunk size used to apply the first subnet can be determined based on the value of the first field. The value of the second field may indicate that the chunk overlap used to apply the first subnet can be determined based on the value of a third field.

[0204] Therefore, specifically, the first subnet can be used to perform prediction operations on blocks in the hyperlatent space. Intermediate tensor data can include the output of the prediction operations in the latent space. The second subnet can be used to perform synthesis operations on blocks in the latent space.

[0205] The first and second subnets are applied to square blocks, where the block size is represented by the number of samples along any single edge of the block. Block overlap is represented by the number of samples spanned from the edge inwards by its overlap with adjacent blocks. That is, block overlap forms a boundary around the block, defining the intersection area between that block and the next adjacent block in any direction. After processing, these blocks are recombined such that the outer edge of one block overlaps with the inner boundary of the adjacent block. To create the final image, the overlapping area can be cropped at a centerline around the block, placing it in the middle of the overlapping boundary area. The outer portion of the boundary, cropped outwards from the center of the block, is then discarded.

[0206] The block size of a block processed by the first subnet can be represented in a superlatent space requiring up to 7 bits. The syntax table shown above indicates the number of bits required to indicate each value associated with the field on the left in the right column. Block overlap of blocks processed by the first subnet can be represented in a superlatent space requiring up to 3 bits.

[0207] As described above, the chunk size for the first subnet can be determined as a fraction of the value of the first field. Similarly, the chunk overlap of chunks processed by the first subnet can be determined as a fraction of the value of the third field. Explicit values ​​for applying the second subnet can be represented with up to 9 bits for chunk size and up to 4 bits for chunk overlap. The chunk size and chunk overlap for applying the first subnet can be represented in the hyperlatent space and can be converted for multi-stage context modeling by multiplying the explicit values ​​of the second and fourth fields by 4.

[0208] It should be understood that the numerical values ​​of the fields do not transmit the required order. Necessary information, including chunk size and chunk overlap values, as well as derivation flags, can be provided in any combination of fields in an order that can be read and implemented by a neural network. The numerical order transmitted in this document is only used to help explain the concept of content order based on the exemplary syntax provided above. This syntax provides values ​​and indicators for the first and second subnets, and their order is only restricted by their write order.

[0209] Although the accompanying drawings depict operations in a specific order, this should not be construed as requiring such operations to be performed in the specific order shown or sequentially, or requiring the execution of all the operations shown to achieve the desired result. In some cases, multitasking and parallel processing can be advantageously performed. Furthermore, the separation of various system modules and components in the above embodiments should not be construed as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0210] Specific embodiments of this subject matter have been described. Other embodiments are within the scope of the appended claims. For example, the operations described in the claims can be performed in different orders and still achieve the desired results. For example, the processes described in the drawings do not necessarily require execution in the specific order shown to achieve the desired results. In some implementations, multitasking and parallel processing can be advantageously performed.

[0211] In this embodiment, a computer program stored in a non-transient medium is provided, the computer program including code that, when executed in one or more processors, performs the steps of any of the methods described above.

[0212] The corresponding flowchart for processing is as follows: Figure 19 As shown. Furthermore, as described above, the present invention also provides an apparatus (one or more devices) for performing the steps of the above-described method.

[0213] In this exemplary and non-limiting embodiment, a processing apparatus for decoding tensors representing image data is provided. Figure 20A processing apparatus 2000 with corresponding modules for performing the discussed decoding steps is shown, the processing apparatus including processing circuitry 2010. The processing circuitry is configured to process an input tensor via a neural network, the neural network comprising at least a first subnet and a second subnet implemented by an NN processing module - first subnet 2011 and an NN processing module - second subnet 2012. The NN processing module - first subnet 2011 may have a separate partitioning module 1 2013 that partitions the first tensor into a first plurality of blocks in a spatial dimension and processes the first plurality of blocks via the first subnet. Alternatively, the corresponding modules for processing the first input tensor and / or the first plurality of blocks may be implemented in a single module, which may be included in a single circuit or a separate circuit.

[0214] Similarly, the NN processing module - second subnet 2012 and partitioning module 2014 apply the second subnet to the second tensor, including partitioning the second tensor into a second plurality of blocks in the spatial dimension and processing the second plurality of blocks through the second subnet. Modules 2013 and 2014 may also provide functionality to determine the size of blocks in the first plurality of blocks and / or the second plurality of blocks.

[0215] It should be noted that the second tensor processing following the first tensor processing can be implemented, for example, by routing corresponding modules such that signals (in terms of their input and output signals) are input sequentially at corresponding times (not necessarily immediate times). Optionally or additionally, the corresponding indication sequence can be implemented by a software configuration module. The processing apparatus may also have a parsing module 2015, which provides the function of parsing the size indication of blocks in the first plurality of blocks and / or the second plurality of blocks from the bitstream. Module 2015 may also provide the function of extracting the input tensor from the bitstream.

[0216] It should be noted that, although Figure 20 Individual modules 2011 and 2012, as well as modules 2013 and 2014, are shown, but these modules can be combined into a single module, configured to process the first component and the second component separately, including individual processing of the first plurality of blocks and the second plurality of blocks, to achieve pipelined processing of the respective components.

[0217] In this exemplary and non-limiting embodiment, a processing apparatus for decoding tensors representing image data includes: one or more processors; and a non-transitory computer-readable storage medium coupled to the one or more processors and storing a program executable by the one or more processors, wherein the program, when executed by the one or more processors, configures an encoder to perform the method according to the above-described decoding method.

[0218] The device includes a first subnet 2011, specifically configured to perform a first block-level operation on data derived from image data units to generate intermediate data. The device includes a second subnet 2012, configured to perform a second block-level operation on the intermediate data. Processing circuitry 2010 may be configured to: process image data units to identify a first field of a first predetermined type and a second field of a second predetermined type therein, and determine, based on the value of the first field, a block size for the second block-level operation, and based on the value of the second field, a block size for the first block-level operation.

[0219] In block-level operations, multiple chunks or blocks of adjacent samples can be processed. Samples in each chunk or block can be processed together, but separately from samples in other chunks or blocks. Chunks or blocks can be processed serially or in parallel.

[0220] Similar to the method described above, the device is configured such that one or more processors can be used to process image data units to identify a third field of a third predetermined type therein, and determine block overlap for a second block-level operation based on the value of the third field, and determine block overlap for a first block-level operation based on the value of the second field or the value of a fourth field of a fourth predetermined type.

[0221] The value of the second field includes an explicit value for the block size used in the first block-level operation. The value of the fourth field includes an explicit value for the block overlap used in the first block-level operation.

[0222] The second field includes a flag, which, when set to '1', indicates that the first block-level operation and the second block-level operation include juxtaposition of blocks, and when set to '0', indicates that the first block-level operation and the second block-level operation include hierarchical block formation.

[0223] A value of 1 in the second field indicates that the block size used for the first block-level operation can be determined based on the value of the first field. A value of 1 in the second field also indicates that the block overlap used for the first block-level operation can be determined based on the value of the third field.

[0224] The first block-level operation can be a prediction operation performed on blocks in the superlatent space, with intermediate data including the output of the prediction operation in the latent space. The second block-level operation is a synthesis operation performed on blocks in the latent space.

[0225] The first and second block-level operations perform operations on square blocks, where the block size is represented by the number of samples along any single edge of the block. A single edge means that the edge belongs to a single edge of the block. Block overlap is represented by the number of samples spanned from the edge inwards and by its overlap with adjacent blocks.

[0226] The block size processed by the first block-level operation can be represented in a hyperlatent space requiring up to 7 bits. Block overlap of blocks processed by the first block-level operation can be represented in a hyperlatent space of up to 3 bits. The block size used for the first block-level operation can be determined as a fraction of the value of the first field.

[0227] Block overlap processed by the first block-level operation can be determined as a fraction of the value of the third field. Explicit values ​​from the second block-level operation can be represented in up to 9 bits for block size and up to 4 bits for block overlap.

[0228] The block size and block overlap used for prediction operations can be represented in the hyperpotential space and transformed for multi-stage context modeling by multiplying the explicit values ​​of the second or fourth field by 4.

[0229] It should be understood that the chunk size used for the second chunk-level operation can be determined based on the chunk size and chunk overlap values ​​specified for the first chunk-level operation. That is, explicit values ​​in the fields can be provided for the first chunk-level operation, and the values ​​required for the second chunk-level operation can be derived from them. For example, if the `tile_z_is_derived` flag is equal to "1", the chunk size and chunk overlap are derived. Therefore, no explicit values ​​are provided for the second chunk-level operation. Thus, as explained above, the MCM chunk size and overlap will be derived from the corresponding first and third field values.

[0230] This document also provides a data carrier for storing image data units, the image data units comprising data, the data being reconstructed to form an image by performing a first block-level operation on the data derived from the image data units to generate intermediate data and performing a second block-level composite operation on the intermediate data; the image data units comprising a first field of a first predetermined type and a second field of a second predetermined type, the first field of the first predetermined type specifying the block size for the first block-level operation, and the second field of the second predetermined type specifying the block size for the second block-level operation.

[0231] Figure 2121 shows a corresponding system where the above encoder-decoder processing chain can be deployed. Figure 21This is a schematic block diagram of an exemplary decoding system, such as a video, image, audio, and / or other decoding system (or simply a decoding system) that can utilize the techniques described in this application. The video encoder 20 (or simply encoder 20) and video decoder 30 (or simply decoder 30) of the video decoding system 10 represent examples of devices that can be used to perform the various techniques according to the various examples described in this application. For example, video encoding and decoding can employ neural networks, which can be distributed, and the aforementioned bitstream parsing and / or bitstream generation can be applied to transmit feature maps between distributed computing nodes (two or more).

[0232] like Figure 21 As shown, the decoding system 10 includes a source device 12, which is used to provide encoded image data 21, for example, to a destination device 14, for decoding encoded image data 13.

[0233] The source device 12 includes an encoder 20 and may additionally (optionally) include an image source 16, a preprocessor (or preprocessing unit) 18 (e.g., an image preprocessor 18), and a communication interface or communication unit 22.

[0234] Image source 16 may include or may be any type of image capture device, such as a camera for capturing real-world images; and / or any type of image generation device, such as a computer graphics processor for generating computer-animated images; or any other device of any type for acquiring and / or providing real-world images, computer-generated images (e.g., screen content, virtual reality (VR) images) and / or any combination thereof (e.g., augmented reality (AR) images). The image source may be any type of memory or storage device for storing any of the aforementioned images.

[0235] To distinguish between the processing performed by the preprocessor 18 and the preprocessing unit 18, the image or image data 17 may also be referred to as the raw image or raw image data 17.

[0236] The preprocessor 18 receives (raw) image data 17 and performs preprocessing on the image data 17 to obtain a preprocessed image 19 or preprocessed image data 19. The preprocessing performed by the preprocessor 18 may include cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may be an optional component. It should be noted that preprocessing may also utilize neural networks (e.g., Figures 1 to 7 (any one of them), the neural network uses an presence indicator to indicate.

[0237] The video encoder 20 is used to receive preprocessed image data 19 and provide encoded image data 21.

[0238] The communication interface 22 of the source device 12 can be used to receive the encoded image data 21 and send the encoded image data 21 (or any other processed version thereof) to another device (e.g., the destination device 14) or any other device via the communication channel 13 for storage or direct reconstruction.

[0239] Destination device 14 includes decoder 30 (e.g., video decoder 30) and may additionally (optionally) include communication interface or communication unit 28, post-processor 32 (or post-processing unit 32) and display device 34.

[0240] The communication interface 28 of the destination device 14 is used to receive, for example, encoded image data 21 (or any other processed version thereof) directly from the source device 12 or from any other source such as a storage device (e.g., an encoded image data storage device) and to provide the encoded image data 21 to the decoder 30.

[0241] Communication interfaces 22 and 28 can be used to send or receive encoded image data 21 or encoded data 13 via a direct communication link (e.g., a direct wired or wireless connection) between source device 12 and destination device 14, or via any type of network (e.g., wired or wireless networks or any combination thereof, or any type of private and public network), or any combination thereof.

[0242] For example, the communication interface 22 can be used to encapsulate the encoded image data 21 into a suitable format such as a message, and / or process the encoded image data using any type of transmission encoding or processing, so as to transmit it via a communication link or communication network.

[0243] For example, the communication interface 28 corresponding to the communication interface 22 can be used to receive transmitted data and process the transmitted data using any type of corresponding transmission decoding or processing and / or decapsulation to obtain encoded image data 21.

[0244] Both communication interface 22 and communication interface 28 can be configured as Figure 21 The communication channel 13, indicated by the arrow pointing from source device 12 to destination device 14, is a one-way communication interface, or configured as a two-way communication interface. It can be used to send and receive messages, establish connections, acknowledge and exchange any other information related to the communication link and / or data transmission (e.g., encoded image data transmission). The decoder 30 receives encoded image data 21 and provides decoded image data 31 or a decoded image 31.

[0245] The post-processor 32 of the destination device 14 is used to post-process the decoded image data 31 (also known as reconstructed image data) (e.g., decoded image 31) to obtain post-processed image data 33 (e.g., post-processed image 33). For example, the post-processing performed by the post-processing unit 32 may include color format conversion (e.g., from YCbCr to RGB), color correction, trimming or resampling, or any other processing, such as preparing the decoded image data 31 for display by the display device 34, etc.

[0246] The display device 34 of the destination device 14 is used to receive the post-processed image data 33 to display the image to a user or viewer. The display device 34 can be or includes any type of display for representing the reconstructed image, such as an integrated or external display or monitor. For example, the display can include a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0247] although Figure 21 Source device 12 and destination device 14 are described as separate devices; however, device embodiments may also include two devices or two functions, namely source device 12 or its corresponding function and destination device 14 or its corresponding function. In these embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software or by separate hardware and / or software or any combination thereof.

[0248] Based on the description, it is obvious to the technicians that... Figure 21 The presence and (precise) division of different units or functions in the source device 12 and / or destination device 14 shown may vary depending on the actual device and application.

[0249] Encoder 20 (e.g., video encoder 20) or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30, may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video decoding processors, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to embody various modules including neural networks or portions thereof. Decoder 30 may be implemented via processing circuitry 46 to embody any decoding system or subsystem described herein. The processing circuitry may be used to perform various operations discussed below. If the foregoing technical portions are implemented in software, the device may store instructions for that software in a suitable non-transitory computer-readable storage medium, and these instructions may be executed in hardware using one or more processors to perform the techniques of the present invention. Video encoder 20 or video decoder 30 may be integrated into a single device as part of a combined encoder / decoder (CODEC), such as... Figure 22 As shown.

[0250] Source device 12 and destination device 14 can include any of a variety of devices, including any type of handheld or fixed device, such as a laptop or tablet computer, mobile phone, smartphone, tablet / tabletcomputer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, video streaming device (e.g., content service server or content distribution server), broadcast receiver device, broadcast transmitter device, etc., and may or may not use any type of operating system. In some cases, source device 12 and destination device 14 can be configured for wireless communication. Therefore, source device 12 and destination device 14 can be wireless communication devices.

[0251] In some cases, Figure 21The video decoding system 10 shown is merely an example, and the technology of this application can be applied to video decoding setups (e.g., video encoding or video decoding) that do not necessarily involve any data communication between the encoding and decoding devices. In other examples, data is retrieved from local memory, streamed over a network, etc. The video encoding device may encode data and store it in memory, and / or the video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data into memory and / or retrieve data from memory and decode it.

[0252] Figure 23 This is a schematic diagram of a video decoding device 8000 provided for embodiments of the present invention. The video decoding device 8000 is suitable for implementing the disclosed embodiments described herein. In one embodiment, the video decoding device 8000 may be a decoder (such as...) Figure 21 Video decoder 30) or encoder (such as Figure 21 (video encoder 20).

[0253] The video decoding device 8000 includes: an input port 8010 (or input port 8010) and a receiving unit (Rx) 8020 for receiving data; a processor, logic unit, or central processing unit (CPU) 8030 for processing data; a transmitting unit (Tx) 8040 and an output port 8050 (or output port 8050) for transmitting data; and a memory 8060 for storing data. The video decoding device 8000 may also include optical-to-electrical (OE) components and electrical-to-optical (EO) components coupled to the input port 8010, the receiving unit 8020, the transmitting unit 8040, and the output port 8050, serving as the output or input of optical or electrical signals.

[0254] Processor 8030 is implemented through hardware and software. Processor 8030 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 8030 communicates with ingress port 8010, receiver unit 8020, transmitter unit 8040, egress port 8050, and memory 8060. Processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the embodiments disclosed above. For example, the neural network-based codec 8070 is used to implement, process, prepare, or provide various decoding operations. Therefore, including the neural network-based codec 8070 significantly improves the functionality of the video decoding device 8000, enabling transitions between different states of the video decoding device 8000. Alternatively, the neural network-based codec 8070 can be implemented with instructions stored in memory 8060 and executed by processor 8030.

[0255] The memory 8060 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store programs as selected for execution, as well as instructions and data read during program execution. For example, the memory 8060 may be volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0256] Figure 24 A simplified block diagram of an apparatus provided for an exemplary embodiment, which can be used as Figure 21 The source device 12 and the destination device 14 are either or both of them.

[0257] The processor 9002 in device 9000 can be a central processing unit. Alternatively, the processor 9002 can be any other type of device or multiple devices, existing or to be developed in the future, capable of manipulating or processing information. While the disclosed implementation can be implemented using a single processor, such as the processor 9002 shown in the figure, using more than one processor can improve speed and efficiency.

[0258] In one implementation, the memory 9004 in the device 9000 may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as memory 9004. Memory 9004 may include code and data 9006 accessed by processor 9002 via bus 9012. Memory 9004 may also include an operating system 9008 and an application program 9010, which includes at least one program that allows processor 9002 to execute the methods described herein. For example, application program 9010 may include applications 1 to N, and may also include a video decoding application that performs the methods described herein.

[0259] The device 9000 may also include one or more output devices, such as a display 9018. In one example, the display 9018 may be a touch-sensitive display that combines a display with a touch-sensitive element capable of sensing touch input. The display 9018 may be coupled to the processor 9002 via a bus 9012.

[0260] Although bus 9012 of device 9000 is described herein as a single bus, bus 9012 may include multiple buses. Furthermore, auxiliary memory may be directly coupled to other components of device 9000 or accessible via a network, and may include a single integrated unit (e.g., a memory card) or multiple units (e.g., multiple memory cards). Therefore, device 9000 can be implemented in a variety of configurations.

[0261] Figure 25 A schematic diagram of a video decoding device 10000 provided for an embodiment of the present invention.

[0262] Platform 10002 in system 10000 can be a cloud server or a local server. Alternatively, platform 10002 can be any other type of device, or multiple devices, capable of computing, storing, transcoding, encrypting, rendering, decoding, or encoding. While the disclosed implementation can be implemented using a single platform such as platform 10002 as shown in the figure, using more than one platform can improve speed and efficiency.

[0263] The content delivery network (CDN) 10004 in system 10000 can be a group of geographically distributed servers. Alternatively, CDN 10004 can be any other type of device, or multiple devices, capable of caching data, scheduling, distributing, or accelerating the delivery of web page content by bringing it closer to the user's location. While the disclosed implementation can be achieved using a single CDN such as CDN 10004 as shown in the figure, using more than one CDN can improve speed and efficiency.

[0264] The terminal 10006 in device 10000 can be a mobile phone, computer, television, laptop, or camera. Alternatively, the terminal 10006 can be any other type of device capable of displaying video or images, or multiple devices.

Claims

1. An image processing apparatus for reconstructing an image from image data units, characterized in that, The device includes one or more processors, the one or more processors being used for: The image data unit is processed to identify a first field of a first predetermined type and a second field of a second predetermined type therein, and a block size for a second block-level operation is determined based on the value of the first field, and a block size for a first block-level operation is determined based on the value of the second field; Perform the first block-level operation on the data derived from the image data unit to generate intermediate data; Perform the second block-level operation on the intermediate data.

2. The image processing apparatus according to claim 1, characterized in that, The one or more processors are configured to process the image data unit to identify a third field of a third predetermined type therein, and determine block overlap for the second block-level operation based on the value of the third field, and determine block overlap for the first block-level operation based on the value of the second field or the value of a fourth field of a fourth predetermined type.

3. The image processing apparatus according to claim 1 or 2, characterized in that, The value of the second field includes an explicit value for the block size used in the first block-level operation.

4. The image processing apparatus according to claim 2 or 3, characterized in that, The value of the fourth field includes an explicit value for the block overlap used in the first block-level operation.

5. The image processing apparatus according to claim 1 or 2, characterized in that, The second field includes a flag, which, when set to '1', indicates that the first block-level operation and the second block-level operation include juxtaposition of blocks, and when set to '0', indicates that the first block-level operation and the second block-level operation include hierarchical block formation.

6. The image processing apparatus according to claim 5, characterized in that, When the value of the second field is 1, it indicates that the block size used for the first block-level operation can be determined based on the value of the first field.

7. The image processing apparatus according to claim 5 or 6, characterized in that, When the value of the second field is 1, it indicates that the block overlap used for the first block-level operation can be determined based on the value of the third field.

8. The image processing apparatus according to any one of the preceding claims, characterized in that, The first block-level operation is a prediction operation performed on blocks in the super-latent space, and the intermediate data includes the output of the prediction operation in the latent space. The second block-level operation is a synthesis operation performed on blocks in the latent space.

9. The image processing apparatus according to any one of the preceding claims, characterized in that, The first block-level operation and the second block-level operation are performed on square blocks, the block size of which is represented by the number of samples along any single edge of the block.

10. The image processing apparatus according to any one of the preceding claims, characterized in that, The block overlap is represented as the number of samples spanned from the edge inwards by its overlap with adjacent blocks.

11. The image processing apparatus according to any one of the preceding claims, characterized in that, The block size of the block processed by the first block-level operation is represented in a hyperlatent space that requires up to 7 bits.

12. The image processing apparatus according to any one of the preceding claims, characterized in that, The blocks processed by the first block-level operation are represented in a super-latent space of up to 3 bits.

13. The image processing apparatus according to any one of the preceding claims, characterized in that, The block size used for the first block-level operation is determined as a fraction of the value of the first field.

14. The image processing apparatus according to any one of the preceding claims, characterized in that, The overlap of the blocks processed by the first block-level operation is determined as the score of the value of the third field.

15. The image processing apparatus according to claim 3 or 4, characterized in that, The explicit value of the second block-level operation is represented by up to 9 bits for the block size and up to 4 bits for the block overlap.

16. The image processing apparatus according to claim 3 or 4, characterized in that, The block size and block overlap used for the prediction operation are represented in the hyperlatent space and transformed for multi-stage context modeling by multiplying the explicit value of the second field or the fourth field by 4.

17. A data carrier, characterized in that, An image data unit is stored, the image data unit comprising data, the data being reconstructed to form an image by performing a first block-level operation on the data derived from the image data unit to generate intermediate data and performing a second block-level composite operation on the intermediate data; the image data unit includes a first field of a first predetermined type and a second field of a second predetermined type, the first field of the first predetermined type specifying the block size for the first block-level operation, and the second field of the second predetermined type specifying the block size for the second block-level operation.

18. A method for processing image tensors representing image data via a neural network including a first subnet and a second subnet, characterized in that, The processing includes indicating the size of the block portion of the image tensor processed by each subnet of the neural network in the block header data, the method comprising: The chunk header data is processed to identify a first field of the first predetermined type and a second field of the second predetermined type therein, and the chunk size for applying the second subnet is determined based on the value of the first field, and the chunk size for applying the first subnet is determined based on the value of the second field; The first subnet is applied to the block portion derived from the image tensor to generate intermediate tensor data; The second subnet is applied to the block portion derived from the intermediate tensor data.

19. The method according to claim 18, characterized in that, The method includes: processing the chunk header data to identify a third field of a third predetermined type therein, determining chunk overlap for applying the second subnet based on the value of the third field, and determining chunk overlap for applying the first subnet based on the value of the second field or the value of a fourth field of a fourth predetermined type.

20. The processing apparatus according to claim 18 or 19, characterized in that, The value of the second field includes an explicit value for applying the block size of the first subnet.

21. The method according to claim 19 or 20, characterized in that, The method includes: determining a block overlap for applying a first subnet based on the value of the fourth field, wherein the fourth field includes an explicit value for the block overlap.

22. The method according to claim 18 or 19, characterized in that, The second field includes a flag, which, when set to '1', indicates that the first block-level operation and the second block-level operation include juxtaposition of blocks, and when set to '0', indicates that the first block-level operation and the second block-level operation include hierarchical block formation.

23. The method according to claim 22, characterized in that, The value of the second field indicates that the block size used to apply the first subnet can be determined based on the value of the first field.

24. The method according to claim 22 or 23, characterized in that, The value of the second field indicates that the block overlap used to apply the first subnet can be determined based on the value of the third field.

25. The method according to any one of claims 18 to 24, characterized in that, The first subnet is used to perform a prediction operation on blocks in the hyperlatent space, the intermediate tensor data including the output of the prediction operation in the latent space, and the second subnet is used to perform a synthesis operation on blocks in the latent space.

26. The method according to any one of claims 18 to 25, characterized in that, The first and second subnets are applied to square blocks, the block size of which is represented by the number of samples along any single edge of the block.

27. The method according to any one of claims 18 to 26, characterized in that, The block overlap is represented as the number of samples spanned from the edge inwards by its overlap with adjacent blocks.

28. The method according to any one of claims 18 to 27, characterized in that, The block size of the block processed by the first subnet is represented in a hyperlatent space that requires up to 7 bits.

29. The method according to any one of claims 18 to 28, characterized in that, The blocks processed by the first subnet are represented in a super-latent space of up to 3 bits.

30. The method according to any one of claims 18 to 29, characterized in that, The block size of the first subnet is determined as a portion of the value of the first field.

31. The method according to any one of claims 18 to 30, characterized in that, The block overlap of the blocks processed by the first subnet is determined to be part of the value of the third field.

32. The method according to claim 20 or 21, characterized in that, The explicit values ​​used to apply the second subnet are represented in up to 9 bits for the block size and in up to 4 bits for the block overlap.

33. The method according to claim 25, characterized in that, The patch size and patch overlap of the first subnet are represented in the hyperlatent space and transformed for multi-stage context modeling by multiplying the explicit values ​​of the second and fourth fields by 4.

34. An image processing apparatus for processing image tensors representing image data via a neural network, characterized in that, The processing includes: indicating the size of the block portions of the image tensor, the device being used to indicate the size in the block header data of multiple subnetworks of the neural network, and formatting the indication according to the following syntax: tile_header (comp) { tile_y_enable[comp] if (tile_y_enable[comp]) { tile_y_size[comp] tile_y_overlap[comp] } tile_z_enable[comp] if (tile_z_enable[comp]) { tile_z_is_derived[comp] if (!tile_z_is_derived) { tile_z_size[comp] tile_z_overlap[comp] } } }。

Images