Neural network with a variable number of channels and its operation method

Abstract

A neural network is provided that includes a first neural network layer and a second neural network layer. The first neural network has a first number of channels, C in as input, and the second channel number C out where the first number of channels is different from the second number of channels and C out =p*C in / q and C in is a multiple of q, and C in , C out , p, and q are integers. The second neural network layer has a second number of channels, C out is configured to take as input.

Description

[Technical Field]

[0001] Embodiments of this disclosure generally relate to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for encoding and decoding images and / or videos from a bitstream using multiple processing layers.

[0002] Cross-reference of related applications This patent application claims priority to international patent application PCT / EP2022 / 079255, filed on 20 October 2022. The disclosure of the aforementioned patent application is incorporated herein by reference in its entirety. [Background technology]

[0003] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded block-wise by predicting blocks and then further coding only the differences between the original blocks and their predictions. In particular, such encoding may include transformation, quantization, and bitstreaming, often entropy encoding. Typically, the three components of a hybrid encoding method—transformation, quantization, and entropy encoding—are optimized separately. Modern video compression standards such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use transformed representations to code the residual signal after prediction.

[0004] Recently, neural network architectures have been applied to image and / or video coding. Generally, these neural network (NN)-based approaches can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize some parts of end-to-end encoding frameworks, such as the selection or compression of prediction parameters. Moreover, some neural network-based approaches have also been discussed for use in hybrid image and video encoding frameworks, for example, as trained deep learning models for intra or inter prediction in image or video encoding.

[0005] The end-to-end optimized image or video encoding applications described above all have in common that they generate some feature map data that should be transmitted between the encoder and the decoder.

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units that can predict an output for a given input. Some neural networks include one or more hidden layers in addition to an output layer. Corresponding feature maps may be provided as the output of each hidden layer. Such corresponding feature maps from each hidden layer may be used as input to subsequent layers in the network, i.e., subsequent hidden layers or output layers. Each layer of the network produces an output from a given input according to the current values ​​of each set of parameters. In a neural network that is split between devices, for example, between an encoder and a decoder, between a device and the cloud, or between different devices, the feature maps at the output of the split location (e.g., device 1) are compressed and sent to the remaining layers of the neural network (e.g., device 2).

[0007] Further improvements to encoding and decoding using trained network architectures may be desirable. [Overview of the project]

[0008] The present invention is described in the appended claims. The above and other objects are achieved by the subject matter of the independent claims. Further embodiments are evident from the dependent claims, specification and drawings.

[0009] Specific embodiments are outlined in the attached independent claims, and other embodiments are outlined in the dependent claims.

[0010] According to the first embodiment, a neural network is provided which includes a first neural network layer and a second neural network. The first neural network has a first number of channels C in It is acquired (received) as input, and the number of second channels C outconfigured to output. Here, the number of first channels is different from the number of second channels, C out =p*C in / q, where C in is a multiple of q, and C in , C out , p, and q are (positive) integers. The second neural network layer is configured to obtain (receive) the number of second channels C out as an input. For example, the C in input channels may be color components, such as RGB or YGB color components, or may include them. For example, the C out output channels exist in the latent space (see the following detailed description). For example, the integer q is equal to 2. For example, p / q = 5 / 2 or p / q = 2.

[0011] In contrast to the prior art, in a neural network according to the first aspect, the number of channels in a neural network where the number of channels changes from the input side to the output side of the first neural network layer is guaranteed to be an integer on the output side of the first neural network layer. For example, for each neural network layer of a neural network, for its respective input channel number C in and its respective output channel number C out , the condition C out =p*C in / q can be satisfied. Here, C in is a multiple of q, and C in , C out , p, and q are (positive) integers. Alternatively, for a neural network layer of a neural network where the number of channels changes from one neural network layer to another, this condition is satisfied. In contrast to the prior art, all channel numbers in a neural network where the number of channels changes from one neural network layer to another can be restricted to integer values.

[0012] In many applications, particularly in the context of video encoding, even small arithmetic errors are unacceptable. For example, the quality of composite video frames obtained based on autoencoders and / or hyperscale decoders (see below) can be significantly affected by arithmetic / numerical errors caused by floating-point operations, especially those performed on the decoder side on a different computer hardware architecture than the one used on the encoder side. Limiting the number of channels in a neural network to integers, where the number of channels changes from one neural network layer to another, improves the overall encoding accuracy and avoids the traditional need to perform rounding of fractional channels, which previously resulted in data / information loss.

[0013] The features described with reference to the following embodiments may be combined in any way that is deemed appropriate.

[0014] In one implementation, the neural network further includes a third neural network layer, the second neural network layer is configured to output the third channel number C', and the third neural network layer is configured to acquire (receive) the third channel number C'. Here, C' = p' * C out / q' and C out c' is a multiple of q', and C', p', and q' are (positive) integers. A similar condition may also hold for the output channels of the third neural network layer. The number of third channels C' is equal to the number of first channels C'. in and / or the number of second channels C out This may differ from the condition C'=p'*C out / q' also conveniently guarantees that the number of input and output channels are integers for both the first and second neural network layers.

[0015] When a neural network includes first, second, and third neural network layers, different additional conditions may also be met. According to one implementation, the number of third channels is less than the number of second channels, and the number of second channels is less than the number of first channels. This allows for a continuous reduction in tensor size while maintaining an integer number of channels. According to another implementation, the number of third channels is less than the number of second channels, and the number of second channels is greater than the number of first channels. This allows for an increase in tensor size (e.g., for temporal hypothesis storage) and a subsequent decrease in tensor size (e.g., after the input data has been used for hypothesis selection) while maintaining an integer number of channels. According to yet another implementation, the number of third channels is greater than the number of second channels, and the number of second channels is greater than the number of first channels. This allows for a continuous increase in tensor size while maintaining an integer number of channels.

[0016] In a different implementation, the neural network does not allow the number of third channels to be greater than the number of second channels, and the number of second channels to be less than the number of first channels. In other words, in the neural network according to the first embodiment and its implementation, the number of third channels cannot be greater than the number of second channels, and the number of second channels cannot be less than the number of first channels. This prevents data loss and subsequent (inaccurate) data recreation.

[0017] According to another implementation, the number of first channels C in and the number of second channels C out C' is a multiple of the chunk size. The third number of channels C' can be similar. Furthermore, the total number of channels in the neural network can be a multiple of the chunk size. The chunk size is, for example, 16 or 32. The first number of channels C in , and / or the number of second channels C outThe number of channels, and / or the third channel number C', and / or the total number of channels in the neural network can generally be multiples of 16 or 32. The chunk size defines the number of output channels that can be processed in parallel by the associated computational means. The total number of output channels can be divided into chunks to process each chunk having a chunk size in parallel. Providing the number of channels as a multiple of the chunk size can further increase the speed of the computations involved, for example, the computations involved in the video coding process. Providing the number of channels as a multiple of 16 or 32 can generally increase the speed of the computations involved, for example, the computations involved in the video coding process.

[0018] As already stated, the neural network according to the first embodiment and its implementation can, advantageously, be applied to the purpose of video encoding. According to one implementation, the first neural network layer or subnet of the neural network is a hyperscale decoder subnet. According to another implementation, the first neural network layer or subnet of the neural network is a prediction fusion subnet. However, this application is not limited to the first neural network layer or subnet of the neural network being a hyperscale decoder subnet or a prediction fusion subnet.

[0019] Generally, note that the following hierarchy of terms is used herein: A neural network includes (neural network) subnets, also called nets, and a (neural network) subnet includes one or more neural network layers. A subnet may include one or more other subnets. In some cases, the term neural network layer may be used for a neural network subnet.

[0020] In further implementations, the first neural network layer has a data path for at least one of the primary and secondary components (first channel number C). in It includes (corresponding to). The primary component can be one of the RGB or YUV color components. The secondary component can be another of the RGB or YUV color components. The coding of the secondary component can be based on the primary component. The tensor size involved in processing the secondary component can be smaller than the tensor size involved in processing the primary component.

[0021] According to another implementation of the neural network in the first embodiment, the neural network includes at least one neural network subnet consisting of consecutive neural network layers, and for at least one neural network subnet, the number of channels changes from one neural network layer to another, and for consecutive neural network layers, at least one of the following conditions is satisfied: a) Each of the consecutive neural network layers is configured to output, in processing order, only a number of channels that is less than or greater than the number of channels received from the preceding neural network layer. b) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset is configured to output, in processing order, only a number of channels greater than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset, and ii) Each of the consecutive neural network layers of the second subset is configured to output, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the second subset. c) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset is configured to output, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset. ii) None of the consecutive neural network layers of the second subset are configured to output a number of channels greater than the number of channels received from the preceding neural network layer of the second subset, in the order of processing, and iii) The first neural network layer in the second subset of consecutive neural network layers is configured to output only a number of channels less than the number of channels received from the last neural network layer in the first subset of consecutive neural network layers.

[0022] This further develops the additional conditions for implantation mentioned above.

[0023] In further implementations, each of the consecutive neural network layers is configured to output a number of channels that is a multiple of 16 or 32.

[0024] Depending on further implementations, at least one neural network subnet is one of the hyperscale decoder subnets and a predictive fusion subnet.

[0025] A method for operating a neural network having a variable number of channels in the neural network layer is provided according to a second embodiment. The method for operating a neural network according to the second embodiment and its implementation, as described below, can be implemented in a neural network according to the first embodiment and its implementation. Furthermore, a neural network according to the first embodiment and its implementation can be used by the method according to the second embodiment. The method according to the second embodiment and its implementation provides the same advantages as described above.

[0026] A method for operating a neural network having a variable number of channels in a neural network layer, according to the second embodiment, involves the first neural network layer having a first number of channels C in The first neural network layer takes the input (receives) and the second channel number C out This is the step to output, and the number of first channels is different from the number of second channels, C out =p*C in / q and C in is a multiple of q, and C in , C out Steps where p, q, and q are (positive) integers, and the second neural network layer adds the second number of channels C. out The process includes the step of obtaining (receiving) the input. The integer q may be equal to 2. For example, p / q = 5 / 2 or p / q = 2.

[0027] According to one implementation, the method in the second aspect further includes the steps of outputting the number of third channels C' by a second neural network layer and obtaining (receiving) the number of third channels C' by a third neural network layer, where C' = p' * C out / q' and C outc' is a multiple of q', and c', p', and q' are (positive) integers.

[0028] According to another implementation, one of the following conditions is met: namely, the number of third channels is less than the number of second channels and the number of second channels is less than the number of first channels, or the number of third channels is less than the number of second channels and the number of second channels is greater than the number of first channels, or the number of third channels is greater than the number of second channels and the number of second channels is greater than the number of first channels.

[0029] According to another implementation, it is not possible for the number of third channels to be greater than the number of second channels, nor is it possible for the number of second channels to be less than the number of first channels.

[0030] According to another implementation, the number of first channels C in and the number of second channels C out This is a multiple of the chunk size. The chunk size can be 16 or 32.

[0031] In another implementation, the first neural network layer, or subnet of the neural network, is a hyperscale decoder network.

[0032] In another implementation, the first neural network layer, or subnet of the neural network, is a predictive fusion network.

[0033] According to another implementation, the first neural network layer includes data paths for primary and / or secondary components.

[0034] According to another implementation, the neural network includes at least one neural network subnet consisting of consecutive neural network layers, and for at least one neural network subnet, the number of channels changes from one neural network layer to another, and with respect to consecutive neural network layers, at least one of the following conditions is met: a) Each of the consecutive neural network layers outputs, in processing order, only a number of channels that is less than or greater than the number of channels received from the preceding neural network layer. b) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset outputs, in processing order, only a number of channels greater than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset, and ii) Each of the consecutive neural network layers of the second subset outputs, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the second subset. c) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset outputs, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset. ii) None of the consecutive neural network layers of the second subset output a number of channels greater than the number of channels received from the preceding neural network layer of the second subset, in the processing order, iii) The first neural network layer in the processing order of the second subset outputs only a number of channels less than the number of channels received from the last neural network layer in the processing order of the first subset.

[0035] In a different implementation, the method includes the step of outputting a number of channels that is a multiple of 16 or 32 for each of the successive neural network layers.

[0036] According to another implementation, at least one neural network subnet is one of the hyperscale decoder subnets and one of the predictive fusion subnets.

[0037] Furthermore, a method for encoding data is provided, which includes steps relating to the operation of a neural network, according to the second embodiment and any implementation thereof.

[0038] Furthermore, a method for decoding encoded data is provided, which includes steps relating to the operation of a neural network, according to the second embodiment and any implementation thereof.

[0039] Furthermore, a computer program product is provided which includes program code stored on a non-temporary medium. Here, when the program is executed on one or more processors, it performs a method of operating a neural network according to the second embodiment and any implementation thereof.

[0040] Furthermore, a device for encoding data is provided, which includes a processing circuit configured to perform steps relating to a method of operating a neural network, according to the second embodiment and any implementation thereof.

[0041] Furthermore, a device for decoding data is provided, which includes a processing circuit configured to perform steps relating to a method of operating a neural network, according to the second embodiment and any implementation thereof.

[0042] Furthermore, a device is provided for decoding at least a portion of an encoded image. The device comprises a processing circuit configured to provide an entropy model, and includes performing steps relating to a method of operating a neural network according to a second embodiment and any implementation thereof: processing a bitstream by a neural network based on the provided entropy model to obtain latent tensors representing the components of an image; and processing the latent tensors to obtain tensors representing the components of an image.

[0043] Furthermore, a device for encoding at least a portion of an image is provided, comprising a neural network according to the first embodiment or any implementation thereof.

[0044] Furthermore, a device is provided for decoding at least a portion of an encoded image, comprising a neural network, according to the first embodiment or any implementation thereof.

[0045] Details of one or more embodiments are revealed in the accompanying drawings and the following description. Other features, purposes, and advantages will be apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0046] Embodiments of the present invention will be described in more detail below with reference to the attached drawings. [Figure 1] Figure 1 is a schematic diagram showing the channels processed by the layers of a neural network. [Figure 2] Figure 2 is a schematic diagram illustrating the autoencoder type of neural network. [Figure 3A] Figure 3A is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperprior model. [Figure 3B] Figure 3B is a schematic diagram showing a typical network architecture for the encoder side, including a hyperplier model. [Figure 3C] Figure 3C is a schematic diagram showing a typical network architecture on the decoder side, including a hyperplier model. [Figure 4] Figure 4 is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperplier model. [Figure 5] Figure 5 is a block diagram showing the structure of a cloud-based solution for machine-based tasks such as machine vision tasks. [Figure 6A] Figure 6A is a block diagram showing a neural network-based end-to-end video compression framework. [Figure 6B] Figure 6B is a block diagram showing some illustrative details related to the application of neural networks for motion field compression. [Figure 6C] Figure 6C is a block diagram showing some illustrative details related to the application of neural networks for motion compensation. [Figure 7A] Figure 7A shows an exemplary AI-trainable image codec architecture. [Figure 7B]FIG. 7B is a diagram related to convolution and deconvolution with a variable number of channels (some examples of Cin > Cout and Cin < Cout). [Figure 8] FIG. 8 is a block diagram showing an exemplary encoder - side sub - network. [Figure 9A] FIG. 9a is a block diagram showing an exemplary decoder - side sub - network comprising a hyper - scale decoder sub - network and a prediction fusion sub - network. [Figure 9B] FIG. 9b is a block diagram showing an alternative architecture of the hyper - scale decoder sub - network as compared to that shown in FIG. 9a. [Figure 9C] FIG. 9c is a block diagram showing an alternative architecture of the prediction fusion sub - network as compared to that shown in FIG. 9a. [Figure 10] FIG. 10 is a diagram showing some exemplary commonly used NN elements. [Figure 11] FIG. 11 is an example for explaining the implementation of the present embodiment as compared to the conventional method. [Figure 12] FIG. 12 is another example for explaining the implementation of the present embodiment as compared to the conventional method. [Figure 13] FIG. 13 is a block diagram showing an exemplary encoding device or decoding device. [Figure 14] FIG. 14 is a block diagram showing another exemplary encoding device or decoding device. [Figure 15] FIG. 15 is a block diagram showing an exemplary video coding system configured to implement an embodiment of the present disclosure. [Figure 16] FIG. 16 is a block diagram showing another exemplary video coding system configured to implement an embodiment of the present disclosure. [Figure 17] FIG. 17 is a block diagram showing an exemplary encoding device or decoding device. [Figure 18]Figure 18 is a block diagram showing another exemplary encoding or decoding device. [Figure 19] Figure 19 is a block diagram showing another exemplary encoding or decoding device. [Figure 20] Figure 20 is a block diagram showing a neural network including a first neural network layer and a second neural network layer according to one embodiment. [Figure 21] Figure 21 is a flowchart illustrating a method for operating a neural network in which the number of channels in the neural network layer is variable, according to one embodiment.

[0047] Similar reference numbers and names in different drawings may refer to the same elements. [Modes for carrying out the invention]

[0048] The following description refers to the accompanying drawings, which form part of the present disclosure and, as illustrative examples, illustrate specific embodiments of the embodiments of the present disclosure or specific embodiments of the present disclosure in which such embodiments may be used. It is understood that embodiments of the present disclosure may be used in other embodiments and may include structural or logical modifications not shown in the drawings. The following detailed description should therefore not be constrained to be restrictive, and the scope of the present disclosure is defined by the accompanying claims.

[0049] For example, disclosures relating to a described method also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding apparatus may include one or more units for performing the described method steps (e.g., one unit performing one or more steps, or multiple units each performing one or more of the steps), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, if a particular apparatus is described based on one or more units, e.g., functional units, the corresponding method may include one step performing the function of one or more units (e.g., one step performing the function of one or more units, or multiple steps each performing one or more of the functions of multiple units), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that the various exemplary embodiments and / or features of the aspects described herein may be combined with each other unless otherwise specified.

[0050] The following provides an overview of some of the technical terms and frameworks used in embodiments of this disclosure.

[0051] Artificial neural networks Artificial neural networks (ANNs), or connectionist systems, are computing systems vaguely conceived from the biological neural networks that make up animal brains. Such systems typically "learn" to perform tasks by considering examples, rather than being programmed with task-specific rules. For example, in image recognition, they can learn to identify cats in other images and identify images containing cats by analyzing exemplary images manually labeled as "cat" or "no cat," and using the results. They do this without any prior knowledge of cats, such as having fur, a tail, whiskers, or a cat-like face. Instead, they automatically generate discriminative characteristics from the examples they process.

[0052] An ANN (Anthropogenic Network) is based on a collection of connected units or nodes called artificial neurons, which loosely model neurons in the biological brain. Each connection can transmit signals to other neurons, just like synapses in the biological brain. The artificial neuron that receives the signal can then process it and transmit signals to the neurons it is connected to.

[0053] In an ANN implementation, the "signal" at a connection is a real number, and the output of each neuron is calculated by some nonlinear function relating to the sum of its inputs. This connection is called an edge. Neurons and edges typically have weights that are adjusted as learning progresses. The weights increase or decrease the strength of the signal at the connection. Neurons may have thresholds such that a signal is transmitted only when the aggregate signal exceeds that threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. Signals travel from the first layer (input layer) to the last layer (output layer), and possibly traversing multiple layers.

[0054] The original goal of the ANN approach was to solve problems in the same way the human brain does. Over time, attention has shifted to performing specific tasks, leading to deviations from biology. ANNs are used for a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally thought to be left to humans, such as painting.

[0055] The name “convolutional neural network” (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0056] Figure 1 schematically illustrates the general concept of processing using neural networks such as CNNs. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is the layer to which the input (such as portion 11 relating to the input image shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve using multiplication or other dot products. The result of the layers is one or more feature maps (indicated by solid lines representing empty rectangles), sometimes also called channels. Resampling (such as subsampling) may occur, involving some or all of the layers. As a result, the feature maps can become smaller, as shown in Figure 1. Note that convolution using stride can also reduce the size of the input feature maps (resampling). In CNNs, the activation function is a ReLU (Rectified Linear Unit) layer, or leaky ReLU, followed by additional convolutions such as pooling layers, fully connected layers, and normalization layers. These layers are called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. These layers are colloquially called convolutions, but this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for indices in a matrix in that it affects how weights are determined at a particular index point.

[0057] As shown in Figure 1, when programming a CNN to process images, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth). It is known that image depth can be composed of the channels of the image. After passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in a neural network has the following attributes: a convolutional kernel defined by width and height (hyperparameters); the number of input channels and output channels (hyperparameters); and the depth of the convolutional filter (input channels) is equal to the number of channels (depth) of the input feature map.

[0058] In the past, conventional multilayer perceptron (MLP) models have been used for image recognition. However, due to their perfect connectivity between nodes, they suffered from high dimensionality and did not scale well for higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to process efficiently and efficiently at a scale with perfect connectivity. Furthermore, such network architectures do not take into account the spatial structure of the data, treating distant input pixels as if they were adjacent to each other. This ignores the locality of reference in image data, both computationally and semantically. Therefore, perfect connectivity of neurons is wasteful for purposes such as image recognition, which are governed by spatially localized input patterns.

[0059] Convolutional neural networks (CNNs) are a biologically inspired variation of multilayer perceptrons, specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by leveraging the strong spatially local correlations present in natural images. The convolutional layer is the core building block of a CNN. The layer's parameters consist of a set of learnable filters (kernels, as described above), which have a small receptive field but extend throughout the full depth of the input volume. During the forward pass, each filter is convolved across the width and height of the input volume, and the dot product between the filter's entry and the input is computed to generate a two-dimensional activation map of that filter. As a result, the network learns filters that become active when certain specific types of features are detected at certain spatial locations in the input.

[0060] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, all entries within the output volume can also be interpreted as the outputs of neurons that observe a small region within the input and share parameters with neurons in the same activation map. A feature map, or activation map, is the output activation for a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activations of different parts of an image, and also called a feature map because it is a mapping of where a given type of feature is found within the image. High activation means that a given feature has been found.

[0061] Another important concept in CNNs is pooling, a form of nonlinear downsampling. Several nonlinear functions exist for performing pooling, with max pooling being the most common. It partitions the input image into a set of non-overlapping rectangles and outputs the maximum value for each of these sub-regions.

[0062] Intuitively, the precise location of a feature is less important than its coarse location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers progressively reduce the spatial size of representations, reducing the number of parameters in the network, the memory footprint, and the computational complexity, and thus also work to control overfitting. In CNN architectures, it is common to periodically insert pooling layers between successive convolutional layers. Pooling operations provide another form of translation invariance.

[0063] A pooling layer operates independently for all depth slices of the input and spatially resizes them. The most common form is a pooling layer with a 2x2 filter applied with a stride of 2 along both width and height for each depth slice in the input, discarding 75% of the activations. In this case, each max operation exceeds the number 4. The depth dimension remains invariant. In addition to max pooling, pooling units can use other functions such as average pooling or l2-norm pooling. Average pooling has often been used historically, but recently it has fallen out of favor compared to max pooling, which actually often performs better. There is a recent trend toward using smaller filters or discarding the pooling layer altogether due to the aggressive reduction in the size of the representation. "Region of Interest" pooling (also known as ROI pooling) is a variation of max pooling where the output size is fixed and the input rectangle is parameterized. Pooling is a key component of convolutional neural networks for object detection based on the fast R-CNN architecture.

[0064] ReLU, as mentioned above, is an abbreviation for rectified linear unit, and it applies a non-saturated activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, such as the saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalized accuracy.

[0065] Leaky Modified Linear Unit, or Leaky ReLU, is a type of activation function based on ReLU, but with a small slope rather than a flat slope for negative values. The slope coefficient is determined before training, i.e., it is not learned during training. This type of activation function is common for tasks with sparse gradients, such as training generative adversarial networks. Leaky ReLU applies a function on the elements.

number

[0066] After several convolutional and max-pooling layers, high-level inference in the neural network is performed through fully connected layers. Neurons in fully connected layers have connections to all activations in the previous layer, as seen in typical (non-convolutional) artificial neural networks. Thus, their activations can be computed as affine transformations using matrix multiplication followed by bias offsets (vector addition of learned or fixed bias terms).

[0067] The “loss layer” (including the calculation of the loss function) specifies how training penalizes the deviation between the predicted (output) label and the true label, and is typically the final layer of a neural network. Various loss functions can be used that are suitable for different tasks. Softmax loss is used to predict a single class out of K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. Euclidean loss is used to regress to real-valued labels.

[0068] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map containing several channels, corresponding to the number of filters in the set of learnable filters in this layer. The feature map is then subsampled, for example, using a pooling layer, which reduces the dimensionality of each channel in the feature map. Next, the data comes to another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish network connectivity, these parameters need to be synchronized between two connected layers so that the number of input channels for the current layer is equal to the number of output channels for the previous layer. For the first layer processing input data, e.g., an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly resampling layers) can be passed to the output layer. Such output layers may be convolutional or resampling in some implementations. In exemplary and non-exclusive implementations, the output layer is a fully coupled layer.

[0069] Autoencoders and unsupervised learning An autoencoder is a type of artificial neural network used to learn efficient data encoding in an unsupervised manner. A schematic diagram of it is shown in Figure 2. The autoencoder includes an encoder side 210 with input x which is fed into the input layer of the encoder subnetwork 220, and a decoder side 250 with output x' which is output from the decoder subnetwork 260. The purpose of the autoencoder is to learn representations (encodings) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore the signal "noise". Along with the reduction (encoder) side subnetwork 220, the reconstruction (decoder) side subnetwork 260 is learned, where the autoencoder attempts to generate a representation x', and thus its name, from the reduced encoding 230 that is as close as possible to its original input x. In its simplest form, given one hidden layer, the encoder stage of the autoencoder takes input x and maps it to h.

number

[0070] This image h is typically called code 230, latent variable, or latent representation. Here, σ is an element-wise activation function, such as a sigmoid function or modified linear unit. W is the weight matrix, and b is the bias vector. The weights and biases are usually initialized randomly and then iteratively updated through backpropagation during training. The decoder stage of the autoencoder then maps h to a reconstructed x' that is identical in shape to x.

number

[0071] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach for latent representation learning, resulting in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. It is used when the data is directed to a graphical model p θ It is generated by (x|h), and the encoder is the posterior distribution p θ Approximation q for (x|h) θ It is assumed that (x|h) is being learned, where φ and θ are the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vector of a VAE is typically much closer to the probability distribution of the training data than a standard autoencoder. The objective of a VAE has the following form:

number

[0072] Here, D KL This represents the Kullback-Leibler divergence. The prior probability for the latent variable is usually a central isotropic multivariate Gaussian.

number

number

[0073] Recent advances in the field of artificial neural networks, and in particular in convolutional neural networks, have enabled researchers to consider applying neural network-based techniques to the tasks of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0074] Therefore, data compression is considered a fundamental and well-studied problem in engineering, and is generally formulated with the goal of designing code for a given discrete data ensemble with minimum entropy. This solution relies heavily on knowledge of the probabilistic structure of the data, and therefore the problem is closely related to probabilistic source modeling. However, since all practical code must have a finite entropy, continuous data (such as a vector of image pixel intensity) must be quantized into a finite set of discrete values, which introduces errors.

[0075] In this context, known as the lossy compression problem, two competing costs must be traded off: the entropy (rate) of the discretized representation and the error (distortion) resulting from quantization. Different compression applications, such as data storage or transmission over channels with limited capacity, require different rate-distortion trade-offs.

[0076] Simultaneous optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is difficult to solve. For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-value representation, independently quantizing its elements, and then encoding the resulting discrete representation using lossless entropy codes. This scheme is called transform coding due to the central role of the transformation.

[0077] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG 2000 uses multiscale orthogonal wavelet decomposition. Typically, the three components of the transform encoding method—namely, the transform, the quantizer, and the entropy code—are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use the transformed representation to encode the residual signal after the prediction. Several transforms are used for this purpose, including discrete cosine and sine transforms (DCT, DST), as well as low-frequency non-separable manually optimized transforms (LFNST).

[0078] Variational image compression The Variable Auto-Encoder (VAE) framework can be considered as a nonlinear transformation encoding model. The transformation process can be mainly divided into four parts, which are illustrated in Figure 3A, which shows the VAE framework.

[0079] The transformation process can be divided into four main parts. Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (indicated by y) via the function y=f(x). This latent representation is also referred to below as a part or point in the “latent space”. The function f() is a transformation function that transforms the input signal x into a more compressible representation y. The quantizer 102 transforms the latent representation y into a quantized latent representation

number

[0080] The latent space can be understood as a compressed representation of data where similar data points are close to each other within the latent space. The latent space is useful for learning data features and finding simpler representations of the data for analysis. The quantized latent representation T, y^, and the side information of the hyperplier 3 are shown.

number

number

[0081] In Figure 3A, component AE 105 is an arithmetic encoding module that converts samples of the quantized latent representation y^ and side information z^ into a binary representation bitstream 1. The samples of y^ and z^ may include, for example, integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​into a string of binary digits (through the process of binariization) (which are then included in a bitstream that may have further parts corresponding to the encoded image or further side information).

[0082] Arithmetic decoding (AD) 106 is the process of reversing the binary conversion process, where the binary digits are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0083] It should be noted that this disclosure is not limited to this particular framework. Furthermore, this disclosure is not limited to image or video compression and may also apply to object detection, image generation, and recognition systems.

[0084] In Figure 3A, there are two interconnected subnetworks. In this context, a subnetwork is a logical division between parts of the entire network. For example, in Figure 3A, modules 101, 102, 104, 105, and 106 are called the “Encoder / Decoder” subnetwork. The “Encoder / Decoder” subnetwork is responsible for encoding (generating) and decoding (parsing) the first bitstream, “bitstream1”. The second network in Figure 3A includes modules 103, 108, 109, 110, and 107 and is called the “HyperEncoder / Decoder” subnetwork. The second subnetwork is responsible for generating the second bitstream, “bitstream2”. The two subnetworks have different purposes.

[0085] The first subnetwork is responsible for the following: ● Convert the input image x into its latent representation y (which is easier to compress than x) 101 ● Quantizing the latent representation y into a quantized latent representation y^102 ● Using AE via the arithmetic encoding module 105, compress the quantized latent representation y^ to obtain the bitstream "bitstream 1". ● Use the arithmetic decoding module 106 to analyze bitstream 1 via AD. ● Reconstruct the reconstructed image (x^) using the analyzed data.

[0086] The purpose of the second subnetwork is to obtain the statistical properties of the samples in "bitstream 1" (e.g., mean, variance, and correlation between samples in bitstream 1), thereby making the compression of bitstream 1 by the first subnetwork more efficient. The second subnetwork generates a second bitstream, "bitstream 2," which contains the aforementioned information (e.g., mean, variance, and correlation between samples in bitstream 1).

[0087] The second network includes converting the quantized latent representation y^ into side information z103, quantizing the side information z into quantized side information z^, and encoding (e.g., binarizing) the quantized side information z^ into bitstream 2109. In this example, binarization is performed by arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD)110, which converts the input bitstream 2 into decoded quantized side information z^'. Since arithmetic encoding and decoding are lossless compression methods, z^' can be identical to z^. The decoded quantized side information z^' is then converted into decoded side information y^'107, where y^' represents a statistical property of y^ (e.g., the sample mean of y^, or the variance of the sample values, etc.). The decoded latent representation y^' is then provided to the arithmetic encoder 105 and arithmetic decoder 106 described above to control the probabilistic model of y^.

[0088] Figure 3A illustrates an example of a VAE (Variable Automatic Encoder), the details of which may differ in different implementations. For example, in a particular implementation, additional components may exist to more efficiently obtain the statistical characteristics of the samples in bitstream 1. In one such implementation, there may be a context modeler that aims to extract cross-correlation information of bitstream 1. The statistical information provided by the second subnetwork may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0089] Figure 3A shows an encoder and decoder in a single figure. As will be apparent to those skilled in the art, encoders and decoders may and very often be embedded in different devices.

[0090] Figure 3B shows the encoder component of the VAE framework, and Figure 3C shows the decoder component of the VAE framework in isolation. As input, according to several embodiments, the encoder receives a picture. The input picture may include one or more channels, such as a color channel or other types of channels, e.g., a depth channel or a motion information channel, etc. The outputs of the encoder are bitstream 1 and bitstream 2 (as shown in Figure 3B). Bitstream 1 is the output of the encoder's first subnetwork, and bitstream 2 is the output of the encoder's second subnetwork.

[0091] Similarly, in Figure 3C, two bitstreams, bitstream 1 and bitstream 2, are received as input, and the reconstructed (decoded) image, z^, is produced as output. As mentioned above, a VAE can be divided into different logical units that perform different actions. This is illustrated in Figures 3B and 3C, in which case Figure 3B shows the components involved in encoding the signal, such as the video and the provided encoding information. This encoded information is then received by the decoder component shown in Figure 3C, for example, encoding. Note that the encoder and decoder components indicated by numerals 12x and 14x may, in their function, correspond to the component indicated by numeral 10x in Figure 3A.

[0092] Specifically, as shown in Figure 3B, the encoder includes encoder 121, which converts input x into signal y, which is then provided to quantizer 322. Quantizer 122 provides information to arithmetic encoding module 125 and hyperencoder 123. Hyperencoder 123 provides the bitstream 2 already described above to hyperdecoder 147, which in turn provides its information to arithmetic encoding module 105 (125).

[0093] The output of the arithmetic encoding module is bitstream 1. Bitstreams 1 and 2 are the outputs of the signal encoding, which are then provided (transmitted) to the decoding process. Unit 101 (121) is called the “encoder”, but it is also possible to refer to the entire subnetwork described in Figure 3B as the “encoder”. The encoding process generally refers to a unit (module) that converts an input to an encoded (e.g., compressed) output. From Figure 3B, it can be seen that unit 121 converts input x to y, which is a compressed version of x, so the unit can actually be considered the core of the entire subnetwork. Compression in encoder 121 can be achieved, for example, by applying a neural network, or generally any processing network having one or more layers. In such a network, compression can be performed by cascading, including downsampling that reduces the size and / or number of channels in the input. Thus, the encoder can be called, for example, a neural network (NN) based encoder.

[0094] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or converting the compressed output y into a sequence of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE 125, which combines the hyperencoder 123 and hyperdecoder 127 used to constitute the AE 125, can perform binarization, which can further compress the quantized signal by lossless compression. Therefore, it is also possible to refer to the entire subnetwork in Figure 3B as the "encoder".

[0095] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary digits (bits). For example, in the VAE framework, the encoder, which is a nonlinear transformation, maps the input image x to y, where y has smaller width and height than x. Since y has smaller width and height, and therefore smaller size, the dimensionality (size) of the signal is reduced, and thus it is easier to compress the signal y. It should be noted that, generally, an encoder does not necessarily need to reduce the size of both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one (or generally a subset thereof) dimension.

[0096] In "Density Modeling of Images Using a Generalized Normalization Transformation" (2015), co-authored by J. Bale, L. Valero Laparra, and EP Simoncelli, presented at the 4th International Conference on Learning Representation 2016 as an arXiv e-print (hereinafter referred to as "Bale"), the authors proposed a framework for end-to-end optimization of image compression models based on nonlinear transformations. The authors optimize for mean squared error (MSE) but use a more flexible transformation constructed from a cascade of linear convolution and nonlinearity. Specifically, the authors use a generalized divisive normalization (GDN) coupled nonlinearity, conceived from a model of neurons in biological visual systems and proven effective for Gaussianizing image density. This cascaded transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), effectively performing parametric vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0097] An example of such a VAE framework is shown in Figure 4, which utilizes six downsampling layers marked with 401 to 406. The network architecture includes a hyperprior model. (g) a ,g s ) shows the image autoencoder architecture, and on the right (h a ,h s ) corresponds to an autoencoder that implements Hyperplier. The factorization prior model is the analytic transformation g a and composite transformation g sThe same architecture is used for both. Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder processes the input image x into g a It targets (subject to) and generates a response y (latent representation) with a spatially varying standard deviation. Encoding g a It includes multiple convolutional layers with subsampling, and generalized divisive normalization (GDN) as the activation function.

[0098] The response is h a It is supplied to and the distribution of the standard deviation at z is summarized. z is then quantized, compressed, and transmitted as side information. The encoder,

number

[0099] Layers that include downsampling are indicated by a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer with N channels, and the convolutional kernel is of size k1 x k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned above, 2↓ means that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 results in one dimension of the input signal being reduced by half in the output. In Figure 4, 2↓ indicates that both the width and height of the input image are reduced by half. Since there are six downsampling layers, if the width and height of the input image 414 (also shown using x) are given by w and h, the output signal z^413 has a width and height equal to w / 64 and h / 64, respectively. The modules indicated by AE and AD are the arithmetic encoder and arithmetic decoder, which are described with reference to Figure 3. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD can be replaced by other means of entropy coding. In information theory, entropy coding is a reversible data compression method used to convert the values ​​of symbols into binary representations, which is a revertible process. Also, "Q" in the figure corresponds to the quantization operation described above in relation to Figure 4, and is further described in the "Quantization" section. Furthermore, the quantization operation and the corresponding quantization unit as part of component 413 or 415 are not necessarily present and / or can be replaced by the unit.

[0100] Figure 4 also shows a decoder that includes upsampling layers 407-412. A further layer 420 is provided between the upsampling layers 411 and 410 in the input processing order, implemented as a convolutional layer but not providing upsampling to the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such layers may be provided in the NN to perform operations on the input that do not change the size of the input but change certain characteristics. However, such layers are not always present.

[0101] In terms of the processing order of bitstream 2 through the decoder, the upsampling layers are executed in the reverse order, i.e., from upsampling layer 412 to upsampling layer 407. Here, each upsampling layer is shown to provide upsampling at an upsampling ratio of 2, indicated by ↑. Of course, it is not necessarily required that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8 may also be used. Layers 407-412 are implemented as convolutional layers (conv). Specifically, since an upsampling layer may be intended to provide an operation on the input that is the inverse of the encoder operation, an upsampling layer can apply a deconvolution operation to the received input, resulting in its size being increased by a coefficient corresponding to the upsampling ratio. However, this disclosure is not limited to deconvolution in general, and upsampling may be performed in any other manner. This could involve bilinear interpolation between two adjacent samples, or nearest neighbor sample copying, etc.

[0102] In the first subnetwork, several convolutional layers (401-403) are followed by generalized divisive normalization (GDN) on the encoder side, and then inverse GDN (IGDN) on the decoder side. In the second subnetwork, the activation function applied is ReLU. Note that this disclosure is not limited to this implementation, and generally, other activation functions may be used instead of GDN or ReLU.

[0103] Cloud solutions for machine tasks Video coding for machines (VCM) is another popular direction in computer science these days. The main idea behind this approach is to transmit coded representations of image or video information intended for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video encoding intended for human perception, the quality characteristic is not the reconstructed quality, but rather the performance on computer vision tasks, such as object detection accuracy. This is illustrated in Figure 5.

[0104] Video coding for machines, also known as collaborative intelligence, is a relatively new paradigm for the efficient deployment of deep neural networks across mobile cloud infrastructure. By splitting the network between the mobile side (510) and the cloud side (590) (e.g., cloud servers), it is possible to distribute the computational workload, resulting in minimized overall system energy and / or latency. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed among two or more different computing nodes, e.g., devices, but generally any functionally defined nodes. Here, the term “node” does not mean the nodes of the neural network described above. Rather, a (computational) node here refers to a separate device / module (physically or at least logically) that implements parts of the neural network. Such devices can be a mix of different servers, different end-user devices, servers and / or user devices, and / or clouds, and / or processors. In other words, computing nodes can be considered as nodes that belong to the same neural network and communicate with each other within / for the purpose of transmitting encoded data within the neural network. For example, to enable the execution of complex computations, one or more layers may run on a first device (such as a device in mobile-side 510), and one or more layers may run on another device (such as a cloud server in cloud-side 590). However, the distribution may also be finer, and a single layer may run on multiple devices. In this disclosure, the term “plurality” refers to two or more. In some existing solutions, some of the neural network functionality runs on a device (such as a user device or edge device), or on multiple such devices, and then the output (feature map) is passed to the cloud.The cloud is a collection of processing or computing systems located outside of the device that powers a portion of the neural network. The concept of collaborative intelligence is similarly extended to model training. In this case, data flows both from the cloud to the mobile device during the backward propagation in training, and from the mobile device to the cloud during the forward path in training (as shown in Figure 5), and so does inference.

[0105] Several studies have proposed semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization has been demonstrated, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to send the output of the hidden layer (deep feature map) 550 from the mobile part 510 to the cloud 590 rather than sending compressed natural image data to the cloud and then performing object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is useful for image and video compression and reconstruction for both human perception and machine vision. Entropy encoding methods, such as arithmetic encoding, are a common approach to compressing deep features (i.e., feature maps).

[0106] Currently, video content contributes to over 80% of internet traffic, and this percentage is expected to increase further. Therefore, it is crucial to build efficient video compression systems and generate higher-quality frames within a given bandwidth budget. In addition, a large portion of video-related computer vision tasks, such as video object detection or tracking, are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Furthermore, techniques in video compression are also useful for motion recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancies in video sequences, as mentioned above. While each module is well-designed, the overall compression system is not end-to-end optimized. It is desirable to further improve video compression performance by collaboratively optimizing the entire compression system.

[0107] End-to-end image or video compression DNN-based image compression methods can utilize large-scale end-to-end training and highly nonlinear transformations not used in conventional approaches. However, directly applying these techniques to build an end-to-end learned system for video compression is not trivial. Firstly, learning how to generate and compress motion information tailored for video compression remains an unresolved problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0108] A straightforward solution is to use learning-based optical flow to represent motion information. However, current learning-based optical flow approaches aim to generate the most accurate flow field possible. Accurate optical flow is often not optimal for a particular video task. In addition, the amount of data in optical flow is significantly larger compared to motion information in conventional compression systems, and directly applying existing compression approaches to compress optical flow values ​​significantly increases the number of bits required to store motion information. Secondly, it is unclear how to build a DNN-based video compression system by minimizing rate-distortion-based objectives for both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality reconstructed frames (i.e., less distortion) given the number of bits (or bitrate) for compression. RDO is crucial for video compression performance. To leverage the end-to-end training capabilities for learning-based compression systems, an RDO strategy that optimizes the entire stem is needed.

[0109] In "DVC: An End-to-end Deep Video Compression Framework," in the proceedings of the IEEE / CVF conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006-11015, Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao, the authors propose an end-to-end deep video compression (DVC) model that collaboratively learns motion estimation, motion compression, and residual encoding.

[0110] Such encoders are shown in Figure 6A. In particular, Figure 6A shows the overall structure of an end-to-end trainable video compression framework. To compress motion information, optical flow v t For better compression, use an appropriate corresponding expression m t A CNN was specified to convert to [a specific format]. Specifically, an autoencoder network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 6B. The network architecture is somewhat similar to that of Figure 4g. a / g s It is similar to, in particular, optical flow v t This is fed into a series of convolution operations and nonlinear transformations, including GDN and IGDN. The number of output channels c for convolution (deconvolution) is equal to 2 in this example, except for the last deconvolution layer, which is exemplarily 128 here. The kernel size is k, for example k=3. Given an optical flow of size M×N×2, the MV encoder generates a motion representation m of size M / 16×N / 16×128. t This generates the motion representation, which is then quantized (Q) and entropy coded.

number

number

[0111] Figure 6C shows the structure of the motion compensation unit. Here, the previous reconstructed frame x t-1Then, using the reconstructed motion information, a warping unit generates warped frames (usually with the help of interpolation filters such as bilinear interpolation filters). A separate CNN with three inputs then generates the predicted picture. The architecture of the motion-compensated CNN is also shown in Figure 6C.

[0112] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to convert the residuals into their corresponding latent representations. Compared to the discrete cosine transform in conventional video compression systems, this approach better utilizes the power of nonlinear transformations and achieves higher compression efficiency.

[0113] From the above overview, it can be seen that CNN-based architectures can be applied to both image and video compression, considering the different parts of the video framework, including motion estimation, motion compensation, and residual encoding. Entropy coding is a common method used for data compression, and it is widely adopted by the industry and is also applicable to feature map compression for either human perception or computer vision tasks.

[0114] Video encoding for machines Video coding for machines (VCM) is another direction in computer science that has become popular recently. The main idea behind this approach is to transmit encoded representations of image or video information that are intended for further processing by computer vision (CV) algorithms such as object segmentation, detection, and recognition. In contrast to traditional image and video encoding intended for human perception, the quality characteristic is not the reconstructed quality, but rather the performance on computer vision tasks, such as object detection accuracy.

[0115] Recent research proposes a new deployment paradigm called collaborative intelligence, in which deep models are partitioned between mobile and cloud. Extensive experimentation under various hardware configurations and wireless connectivity modes has revealed that the optimal operating point in terms of energy consumption and / or computational latency is often a point deep in the network, involving partitioning the model. It has been found that today's common solutions, where the model resides entirely in the cloud or entirely on the mobile, are rarely optimal (if any). The concept of collaborative intelligence has also been extended to model training. In this case, data flows in both directions: from cloud to mobile during the backward propagation in training, and from mobile to cloud during the forward path in training, and similarly to inference.

[0116] In the context of recent deep models for object detection, irreversible compression of deep feature data has been studied based on HEVC intracoding. Degradation of detection performance with increasing compression levels has been noted, and compression-augmented training has been proposed to minimize this loss by generating models more robust to quantization noise in feature values. However, this remains a sub-optimal solution, as the codecs employed are highly complex and optimized for natural scene compression rather than deep feature compression.

[0117] The problem of deep feature compression for collaborative intelligence has been addressed by approaches for object detection tasks using the common YOLOv2 network, for the purpose of studying the trade-off between compression efficiency and recognition accuracy. Here, the term "deep features" has the same meaning as "feature map." The word "deep" comes from the idea of ​​collaborative intelligence where output feature maps from several hidden (deep) layers are captured and then transferred to the cloud for inference. This appears to be more efficient than sending compressed natural image data to the cloud and then performing object detection using the reconstructed images.

[0118] Efficient feature map compression is useful for compressing and reconstructing images and videos for both human perception and machine vision. The aforementioned drawbacks of modern autoencoder-based approaches to compression are also valid for machine vision tasks.

[0119] The quality metrics in the JPEG AI common training and test conditions (CTTC) are selected based on their correlation with human perception. Seven different metrics, though not a single one, are used by JPEG AI, each sensitive to different types of artifacts. It has been noted that most quality metrics are computed in the YUV color space, and some of them use only the luminance component. The JPEG AI trainable image codec is built on the assumption that one color component contains the majority of the information and has a greater impact on human perception than others. To extract this strongest color component, the RGB input first proceeds to a color conversion. The default color conversion is RGB->YUV BT.709 (full range). Adaptive color conversion using a signaling color conversion matrix is ​​supported.

[0120] Figure 7A shows an exemplary AI-learnable image codec architecture. The primary and secondary color components are encoded separately using networks with the same architecture but different numbers of channels. In Figure 7A, neural subnetworks / dataflows used only on the encoder side are marked by dashed boxes / lines. Solid boxes indicate subnetworks used on the decoder (or both the decoder and encoder sides). All boxes with the same name are subnetworks with the same architecture, differing only in the size of the input-output tensor and the number of channels. Subnetworks are also referred to as networks in this disclosure.

[0121] The number of channels in a latent tensor can vary from layer to layer in a neural network. Figure 7B shows several typical cases of convolution and inverse convolution where the number of channels in the input and output tensors differs. Typically, the neural network algorithm is C in (Number of channels in the input tensor) and C out This includes an equation relating to the number of channels in the output tensor, which is often the ratio C. out =p*C in / q

[0122] In a trainable image encoding, the input signal to be encoded is denoted as x, the latent space tensor at the bottleneck of the variational autoencoder is y, and the hyperparameter (tensor at the bottleneck of the hyperencoder) is z. The prediction fusion net generates a prediction μ for the latent space tensor y. The residual signal r = y - μ is assumed to be a Gaussian distribution with zero mean and variance σ, which is the output of the hyperscale decoder net.

number

[0123] The quadratic components are coded together. The quadratic component joint and associated tensor are marked with "UV" in Figure 7A. The quadratic components are coded at a reduced resolution. Thus, the quadratic tensor color component and the first-order tensor color component before the analytic transformation network pass through the downsampling module. The input to the analytic transformation network for the quadratic components is a concatenated tensor of three color components at a reduced resolution, with a size of W / 2 × W / 2 × 3. The output of the analytic transformation network for the quadratic components is C s ×h YUV ×w UV The number of channels allocated for secondary component encoding is C. s =64. For the quadratic components, the output of the hyperencoder network is C s ×h hpUV ×w hpUV That is the case.

[0124] The prediction subnetworks, for example, the prediction fusion network in Figure 7A, operate independently and have the same architecture for the first and second-order components. The prediction subnetwork outputs a tensor of the same size w×h×C as the latent space tensor y. The prediction subnetwork receives a tensor of the same space size but has four times more channels, w×h×4C. Half of the channels arrive as the output of the context model network, and the other half are the output of the hyperdecoder network.

[0125] Convolution with downsampling stride is part of JPEG AI, a learnable codec architecture. A padding layer is inserted before the convolution with downsampling stride to ensure that the tensor size is always a multiple of the downsampling stride.

[0126] Figure 8 is a block diagram illustrating an exemplary encoder-side subnetwork. For example, the analytic transform network may include a sequence of four convolutions with a downsampling stride of 2. The tensor sizes in different parts of the analytic transform network are shown in Figure 8 as one example. As can be seen from the figure, peak memory usage occurs after the first convolutional layer.

[0127] As shown in Figure 8, each of these convolutions is preceded by a padding layer. The padding layer modifies the tensor size as follows: h t =ceil(h t-1 ,s), w t =ceil(w t-1 ,s) where s is the downsampling convolution stride (s=2). The nonlinear activation layer in the analytic-transform network is a residual nonlinear unit (ResAU) with an attention mechanism (e.g., the ResAU block in Figure 10). A ResAU consists of element-wise nonlinear operations (ReLU), a convolutional layer, element-wise multiplication operations (tahn), and residual connections. A detailed structure of the ResAU used in the analytic-transform network (also called a transform module) is shown in Figure 10.

[0128] A residual non-local attention block (RNAB) is used in the analytic transformation network. The RNAB is placed between the first downsampling convolution and the second downsampling convolution and serves as an attention mechanism. Since the RNAB (e.g., the RNAB in Figure 10) contains one downsampling and one upsampling convolution, a series of residual blocks (e.g., RBs shown in Figure 10), a padding layer is inserted before the RNAB. The RNAB also contains a series of residual blocks (e.g., RBs shown in Figure 10).

[0129] The encoder side may also include a hyperencoder network. For example, as shown in Figure 8, the hyperencoder network may include a sequence of two convolutions with a downsampling stride of 2, three convolutions without tensor resizing, and a ReLU as activation (e.g., the ReLU in Figure 10). The tensor sizes at different parts of the analytic transform are shown in Figure 8. Similar to the analytic transform network, each downsampling step is preceded by a padding layer.

[0130] Bitstream #1 and Bitstream #1 (as shown in Figure 7A) are encoded / decoded using a fixed probability density model. The discrete cumulative distribution function is stored in a predetermined fixed table and used to analyze the quantized hyper prior latent z^. The quantized hyper prior latent z^ is then processed by a hyperscale decodernet, which is an NN-based subnetwork used to generate the Gaussian variance σ. Subsequently, the quantized residual latent sample

number

number

[0131] Figure 9a is a block diagram of the decoder-side subnetwork. The hyperscale decoder net 910 includes a sequence of two deconvolutions with an upsampling stride of 2, two convolutions without tensor resizing, and leaky ReLU (e.g., leaky ReLU in Figure 10) as the activation size of the tensor in different parts of the analytic transformation, as shown below the block diagram in Figure 9. In contrast to the hyperencoder, each upsampling convolutional layer is followed by a cropping operation.

[0132] The entire process is shown in Figure 9 as the Prediction Fusion Net 920, which is also called the Prediction Gather Net. At the start of the latent sample prediction process, the hyperdecoder net 910 performs an inverse transform operation on the hyperprior latent z^. The output of this process is concatenated with the output of the context model subnetwork and then processed by the Prediction Fusion Net 920 to generate a predicted value μ. The predicted value is then added to the quantized residual sample r^ to obtain the quantized latent sample z^. The hyperdecoder net (Figure 9) includes a sequence of two inverse convolutions with an upsampling stride of 2, three convolutions without tensor size changes, and a leaky ReLU as an activation. In contrast to the hyperencoder, each upsampling convolutional layer is followed by a cropping operation. Predictive fusion network 920 (Figure 9) consists of a sequence of convolutions in which the number of channels gradually decreases.

[0133] As shown in the example in Figure 9, the Synthesis Transform Net contains a sequence of four convolutions with an upsampling stride of 2. The tensor sizes in different parts of the Synthesis Transform Net are shown in Figure 9. It can be noticed that the peak memory usage (which is also the maximum number of multiplication operations) appears before the last upsampling convolutional layer. For a first-order component Synthesis Transform, the number of channels is as follows: =Cin = 128; C out = 2 Regarding the secondary component synthesis conversion, the number of channels is as follows. C = 64; C in = 128 + 64; C out = 2

[0134] Note that the latent sample prediction process is an autoregressive process. However, the quantized latent samples in different rows

Number

[0135] In JPEG AI VMuC-2.0, for example, in the "Hyper Scale Decoder Net" and "Prediction Fusion (Gather) Net", the tensor size in the channel dimension is a fraction. In actual implementation, the fraction is truncated and data is lost.

[0136] This problem is specific to the CCS architecture adopted by JPEG AI VMuC-2.0. This is because the number of channels in the primary component pipeline (Cp) and the secondary component pipeline (Cs) is not a multiple of 3. In almost all papers, a channel number of C = 192 (a multiple of 3 since 3-color RGB is encoded together) is used.

[0137] The number of channels in a neural network (NN) subnetwork often varies from layer to layer. One typical example is a gather subnetwork, which operates as a fusion of multiple hypotheses generated by a preceding subnetwork. In this disclosure, the terms “gather” and “fusion” have the same meaning.

[0138] According to the embodiment, the NN design principle includes at least one of the following conditions: 1. If the number of channels changes from layer to layer within a subnetwork, that number of channels must be an integer at all layers. C in The subnetwork design is only possible if the number of channels C is a multiple of q. out =p*C in It has / q. This can prevent data loss. For example, q is equal to 2. 2. When the number of channels changes within a subnetwork, it will decrease within that subnetwork and will not subsequently increase. ●When the tensor sizes (in processing order) are C0, C1, ..., Ck-1, Ck, ..., CN, ● 0>C1>…>Ck-1>Ck>…>CN indicates a decrease in tensor size after the input data has been used.

number

[0139] This can provide efficient NPU / GPU resource utilization.

[0140] To implement the above NN design principle, embodiments of the present invention provide a neural network, for example, as shown in FIGS. 1 - 10. The neural network includes a first neural network layer and a second neural network layer. The neural network layer can obtain a first number of channels (C in ) as an input and output a second number of channels (C out ). Here, the first number of channels is different from the second number of channels, and C out = p * C in / q, C in is a multiple of q, and C in , C out , p, and q are integers. The second neural network layer is configured to obtain a second number of channels (C out ) as an input.

[0141] As one example, the first neural network layer is a hyperscale decoder network as shown in Figures 7 and 9. Alternatively, the first neural network layer could be a predictive fusion (gather) network as shown in Figures 7 and 9.

[0142] The first neural network layer contains data paths for the primary and / or secondary components. The second neural network layer has a similar structure to the first neural network layer. The neural network may further include a third neural network layer. The second neural network layer is configured to output the third channel number (C'), and the third neural network layer is configured to obtain the third channel number (C'), where C' = p' * C out / q' is such that C' is a multiple of q', and C', p', and q' are integers. The third neural network layer may have a structure similar to that of the first neural network layer. As described above, the number of third channels is smaller than the number of second channels and the number of second channels is smaller than the number of first channels, or the number of third channels is smaller than the number of second channels and the number of second channels is larger than the number of first channels, or the number of third channels is larger than the number of second channels and the number of second channels is larger than the number of first channels. However, it is not permissible for the number of third channels to be greater than the number of second channels, and for the number of second channels to be less than the number of first channels. In possible implementations, the number of first channels (C in ) and the number of second channels (C out ) is a multiple of chunk size 8. The chunk size can be 16 or 32.

[0143] To realize embodiments of the present invention, two examples are provided in Figures 11 and 12. As shown in Figures 11 and 12, in the conventional method, the output of Conv 5 / 3C'3 in the hyperscale decoder net is 213.3333 when Cp=128 and 106.6667 when Cs=64. Thus, the output of the number of channels is a non-integer. In the embodiment provided in Figure 11, the output of Conv 5 / 2C'3'3 in the hyperscale decoder net is 320 when Cp=128 and 160 when Cs=64. Thus, the output of the number of channels is an integer.

[0144] Similarly, in a predictive fusion (gather) network, using the conventional method, the output of Conv 10 / 3C'1'1 in a hyperscale decoder network would be 426.6667 when Cp=128 and 213.3333 when Cs=64. Therefore, the output number of channels is a non-integer. In the implementation provided in Figure 11, the output of Conv 7 / 2C'1'1 would be 448 when Cp=128 and 224 when Cs=64. Therefore, the output number of channels is an integer.

[0145] Figure 11 shows an arbitrary design for a hyperscale decoder network and a predictive fusion (gather) network. Figure 12 provides another selection for a hyperscale decoder network and a predictive fusion (gather) network. The hyperscale decoder network in Figure 11 can also operate together with the hyperscale decoder network and / or the predictive fusion (gather) network in Figure 12. Similarly, the predictive fusion (gather) network in Figure 11 can also operate together with a hyperscale decoder network and / or a predictive fusion (gather). This invention does not limit the combinations of these networks. A network is also called a neural network layer or a neural network subnetwork.

[0146] The order of implementation in Figures 11 and 12 is from right to left.

[0147] In the example of the hyperscale decoder network in Figure 11, p / q = 2 and p' / q' = 5 / 2. In the example of the hyperscale decoder network in Figure 12, p / q = 5 / 2 and p' / q' = 2.

[0148] As shown in the example in Figure 11, the tensor size for the number of channels (C) in a hyperscale decoder network can include 3 / 2C, 2C, 5 / 2C, and 3 / 2C in that order. As shown in the example in Figure 12, the tensor size for the number of channels (C) in a hyperscale decoder network can include 3 / 2C, 5 / 2C, 2C, and 3 / 2C in that order.

[0149] In the example of a predictive fusion (gather) network in Figure 11, p / q = 7 / 2 and p' / q' = 3. In the example of a predictive fusion (gather) network in Figure 12, p / q = 9 / 2 and p' / q' = 7 / 2.

[0150] As shown in the example in Figure 11, the tensor size of the number of channels (C) in a predictive fusion (gather) network can include 7 / 2C, 3C, and 5 / 2C in that order. As shown in the example in Figure 12, the tensor size of the number of channels (C) in a predictive fusion (gather) network can include 9 / 2C, 7 / 2C, and 5 / 2C in that order.

[0151] In this disclosure, when the number of channels in a subnetwork is variable, data loss can be prevented by maintaining the number of channels as an integer at every layer. When the number of channels in a subnetwork is variable, efficient NPU / GPU resource utilization is achieved by maintaining the number of channels as multiple chunk sizes at every layer. Furthermore, it is not permissible for the number of third channels to be greater than the number of second channels, and for the number of second channels to be less than the number of first channels. This prevents data loss and further data reconstruction.

[0152] Functional modules Variable bitrate module An encoder can output a bitstream at different bitrates. Therefore, in some ways, as shown in Figure 13, the output of the encoding network is scaled (e.g., each channel is multiplied by a corresponding scaling factor, also called the target gain value), and the input of the decoding network is descaled (e.g., each channel is multiplied by a corresponding reciprocal scaling factor, also called the target inverse gain value). The scaling factors can be preset. Different quality levels or quantization parameters correspond to different target gain values. If the output of the encoding network is scaled to a smaller value, the bitstream size can be decreased. Otherwise, the bitstream size can be increased.

[0153] Color format conversion RGB and YUV are common color spaces. Conversion between RGB and YUV can be performed according to formulas specified in standards such as CCIR 601 and BT.709.

[0154] Separation structures for Luma and Chroma As shown in Figure 14, some VAE-based codecs use the YUV color space as the input to the encoder and the output to the decoder. The Y component represents luma, and the UV component represents chroma. The resolution of the UV component may be the same as or lower than that of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted to a feature map F_Y through a network, and an entropy coding module generates a bitstream of the Y component based on the feature map F_Y. The UV component is converted to a feature map F_UV through another network, and an entropy coding module generates a bitstream of the UV component based on the feature map F_UV. Under this structure, the feature maps of the Y component and the UV component may be quantized independently, and as a result, bits can be flexibly allocated to luma and chroma. For example, for color-sensitive images, the feature map of the UV component may be less quantized, and the number of bitstream bits for the UV component may be increased to improve the reconstruction quality of the UV component.

[0155] In some other methods, the encoder concatenates the Y and UV components and then sends them to a UV component processing module (to convert the image information into a feature map). In addition, the decoder concatenates the reconstructed feature map of the Y component and the reconstructed feature map of the UV component and then sends them to a UV component processing module 2 (to convert the feature map into image information). In this method, the correlation between the Y and UV components may be used to reduce the bitstream of the UV component.

[0156] While the operations are shown in the drawings in a specific order, this should not be understood as requiring that such operations be performed in a specific or sequential order shown, or that all described operations be performed, in order to achieve the desired result. In given circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the embodiments described above should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0157] Specific embodiments relating to the subject matter have been described. Other embodiments are within the following claims. For example, the operations described in the claims may be performed in a different order, and still the desired results may be achieved. As one example, the process shown in the accompanying drawings does not necessarily require the specific order or sequence shown to achieve the desired results. In a given embodiment, multitasking and parallel processing may be advantageous.

[0158] A corresponding system capable of deploying the encoder / decoder processing chain described above is shown in Figure 15. Figure 15 is a schematic block diagram showing exemplary coding systems, e.g., video, image, audio, and / or other coding systems (or short coding systems), that may utilize the technique of this application. The video encoder 20 (or short encoder 20) and video decoder 30 (or short decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform the technique according to the various examples described in this application. For example, the video encoding and decoding may employ a neural network, which is distributed and can apply the bitstream parsing and / or bitstream generation described above to transmit feature maps between two or more distributed computing nodes.

[0159] As shown in Figure 15, the encoding system 10 includes a source device 12 configured to provide the encoded picture data 21 to a destination device 14 for decoding, for example, the encoded picture data 13.

[0160] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18, and a communication interface or communication unit 22.

[0161] The picture source 16 may have, or may be, any kind of picture capture device, e.g., a camera for capturing real-world pictures, and / or any kind of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other kind of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may be any kind of memory or storage device for storing any of the aforementioned pictures.

[0162] Unlike the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 is also called the raw picture or raw picture data 17.

[0163] The preprocessor 18 is configured to receive (raw) picture data 17 and perform preprocessing on the picture data 17 to obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., RGB to YCbCr), color correction, or denoising. It can be understood that the preprocessing unit 18 may be an arbitrary component. Note that the preprocessing may also employ a neural network using presence indicator signaling (such as one of the figures 1-7).

[0164] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0165] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and to send the encoded picture data 21 (or any further processed version thereof) via the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0166] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device.

[0167] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) for example directly from the source device 12, or from any other source, such as a storage device, for example, an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0168] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network, or any combination thereof, or any type of private and public network, or any combination thereof.

[0169] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any kind of transmission encoding or processing for transmission over a communication link or communication network.

[0170] A communication interface 28, which forms a counterpart to communication interface 22, may be configured to receive transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging, for example, to obtain encoded picture data 21.

[0171] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows pointing from source device 12 to destination device 14 for communication channel 13 in Figure 15. They may also be configured to send and receive messages, and to acknowledge and exchange any other information relating to the communication link and / or data transmission, such as encoding picture data transmission, for example, to set up a connection. Decoder 30 is configured to receive encoding picture data 21 and to provide decoded picture data 31 or decoded picture 31.

[0172] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, to obtain post-processed picture data 33, for example, the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare the decoded picture data 31 for display by the display device 34, for example.

[0173] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for displaying the picture to a user or viewer, for example. The display device 34 is or may include any type of display for displaying the reconstructed picture, such as an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a microLED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0174] Although Figure 15 shows the source device 12 and the destination device 14 as separate devices, the device embodiment may also include both functions, namely the source device 12 or its corresponding function, and the destination device 14 or its corresponding function. In such embodiments, the source device 12 or its corresponding function and the destination device 14 or its corresponding function may be implemented using the same hardware and / or software, by separate hardware and / or software, or by any combination thereof.

[0175] As will become apparent to those skilled in the art based on this description, the functions of different units, or the existence and (exact) division of functions within the source device 12 and / or destination device 14 as shown in Figure 15, may vary depending on the actual device and application.

[0176] The encoder 20 (e.g., video encoder 20), or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuits, such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuit 46 to embody various modules, including a neural network or a part thereof. The decoder 30 may be implemented via processing circuit 46 to embody any encoding system or subsystem described herein. The processing circuits may be configured to perform various operations, as described below. When the technique is partially implemented in software, the device may store instructions for the software in a suitable, non-temporary computer-readable storage medium. The instructions may then be executed in hardware using one or more processors to perform the technique of the present disclosure. Either the video encoder 20 or the video decoder 30 can be integrated as part of a composite encoder / decoder (codec) in a single device, for example, as shown in Figure 16.

[0177] The source device 12 and destination device 14 include any of a wide range of devices, including any kind of handheld or stationary device, such as a notebook or laptop computer, mobile phone, smartphone, tablet or tablet computer, camera, desktop computer, set-top box, television, display device, digital media player, video game console, (such as a content service server or content distribution server), video streaming device, broadcast receiver device, broadcast transmitter device, etc., and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Thus, the source device 12 and destination device 14 may be wireless communication devices.

[0178] In some cases, the video coding system 10 shown in Figure 15 is merely an example, and the techniques of this application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily involve data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device can encode data and store it in memory, and / or a video decoding device can retrieve data from memory and decode it. In some examples, encoding and decoding do not communicate with each other, but are performed by devices that simply encode data into memory and / or retrieve data from memory and decode it.

[0179] Figure 17 is a schematic diagram of a video coding device 8000 according to an embodiment of the present invention. The video coding device 8000 is suitable for carrying out the disclosed embodiments as described herein. In one embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 in Figure 15, or an encoder, such as the video encoder 20 in Figure 15.

[0180] The video coding device 8000 includes an inlet port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing data, a transmitter unit (Tx) 8040 and an exit port 8050 (or output port 8050) for transmitting data, and memory 8060 for storing data. The video coding device 8000 may also include optical-electrical (OE) components and electrical-optical (EO) components coupled to the inlet port 8010, receiver unit 8020, transmitter unit 8040, and exit port 8050 for optical or electrical signal input or output.

[0181] Processor 8030 is implemented by hardware and software. Processor 8030 can be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. Processor 8030 communicates with an input port 8010, a receiver unit 8020, a transmitter unit 8040, an output port 8050, and a memory 8060. Processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Including the neural network-based codec 8070 thus provides a substantial improvement to the functionality of video coding device 8000 and results in the conversion of video coding device 8000 to different states. Alternatively, the neural network-based codec 8070 is implemented as instructions stored in memory 8060 and executed by processor 8030.

[0182] Memory 8060 includes one or more disks, tape drives, and solid state drives and may be used as an overflow data storage device, storing such programs when a program is selected for execution and storing instructions and data read during program execution. Memory 8060 may be, for example, volatile and / or non-volatile and may be read only memory (ROM), random access memory (RAM), ternary associative memory (TCAM), and / or static random access memory (SRAM).

[0183] FIG. 18 is a schematic block diagram of an apparatus that can be used as either or both of source device 12 and destination device 14 from FIG. 15, according to an exemplary embodiment.

[0184] The processor 9002 in device 9000 can be a central processing unit. Alternatively, the processor 9002 can be any other type of device, or multiple devices, that can operate on, or process, information that currently exists or will be developed in the future. The disclosed implementation can be carried out using a single processor as illustrated, e.g., processor 9002, but advantages in speed and efficiency can be achieved using two or more processors.

[0185] The memory 9004 in device 9000 can be, in one implementation, a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as memory 9004. The memory 9004 can include code and data 9006 that is accessed by processor 9002 using bus 9012. The memory 9004 can further include an operating system 9008 and application programs 9010. The application programs 9010 include at least one program that enables processor 9002 to execute the methods described herein. For example, the application programs 9010 can include applications 1-N. These can further include video coding applications that execute the methods described herein.

[0186] Device 9000 can also include one or more output devices, such as display 9018. In one example, display 9018 can be a touch-sensitive display that combines a display with a touch-sensitive element operable to sense touch input. Display 9018 can be coupled to processor 9002 via bus 9012.

[0187] Although shown here as a single bus, the bus 9012 of device 9000 may consist of multiple buses. Furthermore, the secondary storage device may be directly coupled to other components of device 9000, or accessed via a network, and may comprise a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Device 9000 can therefore be implemented in a wide variety of configurations.

[0188] Figure 19 is a block diagram of a video encoding system 10000 according to an embodiment of the present disclosure.

[0189] A neural network 2000 according to an embodiment of the present disclosure is shown in Figure 20. The neural network 2000 includes a first neural network layer 2010 and a second neural network layer 2020. The first neural network 2010 takes a first number of channels C as input. in It acquires (receives) and then the second channel number C out It is configured to output the following. Here, the number of first channels is different from the number of second channels, and C out =p*C in / q. Here, C in is a multiple of q, and C in , C out p and q are (positive) integers. The second neural network layer 2020 has a second channel count C. out It is configured to take (receive) C as input. For example, C in The input channel is a color component, for example, RGB or YGB color components, or may contain them. For example, C out The output channel exists in latent space. Condition C out =p*C in / q is also the first neural network layer 2010 C out The number of output channels C in The output channels of the second neural network layer 2020, which receive as individual input channels, may also be satisfied. According to one embodiment, this condition is satisfied for each of the C of each neural network layer of the neural network 2000. in Input Channel and C out This condition is met for output channels. Generally, the number of channels or the number of output channels can be a multiple of 16 or 32.

[0190] Limiting the variable number of channels to an integer significantly improves the accuracy and speed of calculations compared to conventional techniques.

[0191] Neural Network 2000 may consist of one of the neural network architectures shown in Figures 1 to 10, or one of the devices shown in Figures 15 to 19.

[0192] A neural network operation method 2100 with a variable number of channels in the neural network layer is shown in Figure 21. For example, method 2100 is for operating a neural network 2000 having a first neural network layer 2010 and a second neural network layer 2020 as shown in Figure 20. Method 2100 uses the first neural network layer to input a first number of channels C in Step 2110 acquires (receives) and the first neural network layer determines the second channel number C out Step 2120 outputs the following, where the number of first channels is different from the number of second channels, C out =p*C in / q and C in is a multiple of q, and C in , C out p, q, and step numbers are (positive) integers, and the second neural network layer takes the second channel number C as input. out Step 2130 includes obtaining (receiving) C' = p'*C out This includes the step of outputting / q'. Here, C out p' is a multiple of q', and C', p', and q' are (positive) integers and are channels by the second neural network layer. The video encoding or decoding method may, advantageously, include method 2100 shown in Figure 21.< / ck>

Claims

1. A neural network used in a video coding system, The processor in the aforementioned system: This is the first neural network layer, which functions to take the first channel number Cin as input and output the second channel number Cout. The number of the first channel is different from the number of the second channel, and Cout = p * Cin / q. Cin is a multiple of q, and Cin, Cout, p, and q are positive integers. The first neural network layer, A second neural network layer is configured to take the second channel number Cout as input and output the third channel number C'. Includes, The first neural network layer, or the subnet of the neural network, is a hyperscale decoder subnet or a predictive fusion subnet. Neural network.

2. q is equal to 2. The neural network according to claim 1.

3. The aforementioned neural network further includes a third neural network layer, The third neural network layer causes the processor to function in order to obtain the third number of channels C'. C' = p' * Cout / q', where Cout is a multiple of q', and C', p', and q' are positive integers. The neural network according to claim 1.

4. The number of third channels is smaller than the number of second channels, and the number of second channels is smaller than the number of first channels, or The number of third channels is smaller than the number of second channels, and the number of second channels is larger than the number of first channels, or The number of third channels is greater than the number of second channels, and the number of second channels is greater than the number of first channels, The neural network according to claim 3.

5. The aforementioned neural network is The case in which the number of third channels is greater than the number of second channels and the number of second channels is less than the number of first channels is not permitted. The neural network according to claim 3.

6. p / q = 5 / 2 and p' / q' = 2, or p / q=2 and p' / q'=5 / 2 The neural network according to claim 3.

7. The number of first channels C in and the second number of channels C out is a multiple of the chunk size, The neural network according to claim 1.

8. The chunk size is 16 or 32. The neural network according to claim 7.

9. The first neural network layer includes a data path for at least one of the primary and secondary components. The neural network according to claim 1.

10. The neural network includes at least one neural network subnet consisting of consecutive neural network layers, For the aforementioned at least one neural network subnet, with respect to consecutive neural network layers where the number of channels changes from one neural network layer to another, at least one of the following conditions is satisfied: a) Each of the consecutive neural network layers is configured to output, in processing order, only a number of channels that is less than or greater than the number of channels received from the preceding neural network layer. b) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset is configured to output, in processing order, only a number of channels greater than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset, and ii) Each of the consecutive neural network layers of the second subset is configured to output, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the second subset. c) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset is configured to output, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset. ii) None of the consecutive neural network layers of the second subset are configured to output a number of channels greater than the number of channels received from the preceding neural network layer of the second subset, in the processing order, and iii) Of the consecutive neural network layers in the second subset, the first neural network layer in the processing order is configured to output only a number of channels less than the number of channels received from the last neural network layer in the processing order of the consecutive neural network layers in the first subset. The neural network according to claim 1.

11. Each of the aforementioned consecutive neural network layers is configured to output a number of channels that is a multiple of 16 or 32. The neural network according to claim 10.

12. The at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet. The neural network according to claim 10.

13. A method for operating a neural network used in a video coding system, having a variable number of channels in the neural network layer, The processor in the aforementioned system: The first neural network layer takes the first channel number Cin as input, The first neural network layer outputs the second channel number Cout, The number of first channels differs from the number of second channels, where Cout = p * Cin / q, Cin is a multiple of q, and Cin, Cout, p, and q are positive integers, and the step is... The second neural network layer takes the second channel number Cout as input and outputs the third channel number C', They will implement this, The first neural network layer, or the subnet of the neural network, is a hyperscale decoder subnet or a predictive fusion subnet. method.

14. q is equal to 2. The method according to claim 13.

15. The above method further provides the processor with: The third neural network layer obtains the aforementioned third channel number C'. C'=p'*C out / q' and C out The step is such that is a multiple of q', and C', p', and q' are positive integers. The method according to claim 13, which enables the implementation of the method.

16. The number of third channels is smaller than the number of second channels, and the number of second channels is smaller than the number of first channels, or The number of third channels is smaller than the number of second channels, and the number of second channels is larger than the number of first channels, or The number of third channels is greater than the number of second channels, and the number of second channels is greater than the number of first channels, The method according to claim 15.

17. The aforementioned neural network is The number of third channels is not permitted to be greater than the number of second channels, nor is the number of second channels less than the number of first channels. The method according to claim 15.

18. p / q = 5 / 2 and p' / q' = 2, or p / q=2 and p' / q'=5 / 2 The method according to claim 15.

19. The number of first channels C in and the second number of channels C out is a multiple of the chunk size, The method according to claim 13.

20. The chunk size is 16 or 32. The method according to claim 19.

21. The first neural network layer includes data paths for primary and / or secondary components. The method according to claim 13.

22. The neural network includes at least one neural network subnet consisting of consecutive neural network layers, For the aforementioned at least one neural network subnet, with respect to consecutive neural network layers where the number of channels changes from one neural network layer to another, at least one of the following conditions is satisfied: a) Each of the consecutive neural network layers outputs, in processing order, only a number of channels that is less than or greater than the number of channels received from the previous neural network layer among the consecutive neural network layers. b) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset outputs, in processing order, only a number of channels greater than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset, and ii) Each of the consecutive neural network layers of the second subset outputs, in processing order, only a number of channels less than the number of channels received from the previous neural network layer of the consecutive neural network layers of the second subset. c) The consecutive neural network layers consist of a first subset of consecutive neural network layers, and in the processing order, a second subset of consecutive neural network layers follows. i) Each of the consecutive neural network layers of the first subset outputs, in processing order, only a number of channels less than the number of channels received from the preceding neural network layer of the consecutive neural network layers of the first subset. ii) None of the consecutive neural network layers of the second subset output a number of channels greater than the number of channels received from the preceding neural network layer of the second subset, in the processing order, iii) Of the consecutive neural network layers in the second subset, the first neural network layer in the processing order outputs only a number of channels less than the number of channels received from the last neural network layer in the first subset in the processing order. The method according to claim 13.

23. Each of the aforementioned consecutive neural network layers is configured to output a number of channels that is a multiple of 16 or 32. The method according to claim 22.

24. The at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet. The method according to claim 22.

25. A method of encoding data, A step relating to the method described in any one of claims 13 to 24, method.

26. A method for decoding encoded data, A step relating to the method described in any one of claims 13 to 24, method.

27. A computer program that includes program code stored on a non-temporary storage medium, When the program is executed on one or more processors, it performs the method according to any one of claims 13 to 24. Computer program.

28. A device for encoding data, A processing circuit configured to carry out the steps according to any one of claims 13 to 24, Device.

29. A device for decoding data, A processing circuit configured to carry out the steps according to any one of claims 13 to 24, Device.

30. A device for decoding at least a portion of an encoded image, Includes a processing circuit configured to provide an entropy model, To carry out the steps according to any one of claims 13 to 24, Based on the provided entropy model, the bitstream is processed by a neural network to obtain a latent tensor representing the components of the image, and Processing the aforementioned latent tensor to obtain a tensor representing the components of the image, A device including a device.

31. A device for encoding at least a portion of an image, A neural network including the one described in any one of claims 1 to 12, Device.

32. A device for decoding at least a portion of an image, A neural network including the one described in any one of claims 1 to 12, Device.

Images