Neural network with variable number of channels and method of operation thereof

Abstract

A neural network is provided that includes a first neural network layer and a second neural network layer. The first neural network has a first number of channels, C in as input, and the second channel number C out where the first number of channels is different from the second number of channels and C out =p*C in / q and C in is a multiple of q, and C in , C out , p, and q are integers. The second neural network layer has a second number of channels, C out is configured to take as input.

Description

[Technical Field]

[0001] FIELD OF THE DISCLOSURE Embodiments of the present disclosure generally relate to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for so encoding and decoding images and / or videos from a bitstream using multiple processing layers.

[0002] CROSS-REFERENCE TO RELATED APPLICATIONS This patent application claims priority to International Patent Application PCT / EP2022 / 079255, filed October 20, 2022, the disclosure of which is incorporated herein by reference in its entirety. [Background technology]

[0003] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded block-wise by predicting the block and then further coding only the difference between the original block and its prediction. In particular, such encoding may include transform, quantization, and bitstream encoding, often including entropy coding. Typically, the three components of a hybrid encoding method—transform, quantization, and entropy encoding—are optimized separately. Modern video compression standards, such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC), also use transformed representations and code the residual signal after prediction.

[0004] Recently, neural network architectures have been applied to image and / or video coding. Generally, these neural network (NN)-based approaches can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning has been used to determine or optimize some parts of end-to-end encoding frameworks, such as prediction parameter selection or compression. Moreover, some neural network-based approaches have also been discussed for use in hybrid image and video encoding frameworks, for example, for implementation as trained deep learning models for intra- or inter-prediction in image or video encoding.

[0005] The end-to-end optimized image or video encoding applications mentioned above have in common that they generate some feature map data to be communicated between the encoder and decoder.

[0006] A neural network is a machine learning model that uses one or more layers of nonlinear units that can predict an output for a received input based on the feature maps. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map can be provided as the output of each hidden layer. Such corresponding feature map of each hidden layer can be used as an input to a subsequent layer in the network, i.e., the subsequent hidden layer or the output layer. Each layer of the network generates an output from the received input according to the current values ​​of a respective set of parameters. In a neural network that is split between devices, for example, between an encoder and a decoder, between a device and a cloud, or between different devices, the feature map at the output of the split location (e.g., a first device) is compressed and then transmitted to the remaining layers of the neural network (e.g., a second device).

[0007] Further improvements in encoding and decoding using trained network architectures may be desirable. Summary of the Invention

[0008] The invention is set out in the accompanying claims. These and other objects are achieved by the subject matter of the independent claims. Further embodiments are evident from the dependent claims, the description and the drawings.

[0009] Particular embodiments are outlined in the accompanying independent claims, and other embodiments are outlined in the dependent claims.

[0010] According to a first aspect, there is provided a neural network including a first neural network layer and a second neural network. The first neural network has a first number of channels C in is received as input, and the second channel number C outwhere the first number of channels is different from the second number of channels and C out =p*C in / q and C in is a multiple of q, and C in , C out , p, and q are (positive) integers. The second neural network layer has a second number of channels, C out It is configured to receive as input. For example, C in The input channels may be or include color components, for example, RGB or YGB color components. out The output channels exist in a latent space (see detailed explanation below), for example, an integer q equal to 2, e.g., p / q=5 / 2 or p / q=2.

[0011] In contrast to the prior art, the neural network according to the first aspect ensures that the number of channels in a neural network that varies from the input side to the output side of the first neural network layer is an integer at the output side of the first neural network layer. For example, for each of the neural network layers of the neural network, its respective number of input channels C in and the number of output channels C out Regarding condition C out =p*C in / q can be satisfied, where C in is a multiple of q, and C in , C out , p, and q are (positive) integers. Alternatively, this condition is met for neural network layers of a neural network in which the number of channels varies from one neural network layer to another. In contrast to the prior art, all channel numbers in a neural network in which the number of channels varies from one neural network layer to another can be restricted to integer values.

[0012] In many applications, especially in the context of video encoding, even small arithmetic errors cannot be tolerated. For example, the quality of synthesized video frames obtained based on an autoencoder and / or a hyperscale decoder (see below) can be significantly affected by arithmetic / numerical errors caused by floating-point operations, especially floating-point operations performed on the decoder side on a computer hardware architecture different from that used on the encoder side. Restricting the number of channels in a neural network, whose number of channels varies from one neural network layer to another, to integers, particularly improves overall encoding accuracy and avoids the conventional need to perform rounding of fractional channels, which traditionally causes data / information loss.

[0013] The features described with reference to the following embodiments may be combined as seen fit.

[0014] According to one implementation, the neural network further includes a third neural network layer, the second neural network layer further configured to output a third number of channels C', and the third neural network layer configured to receive the third number of channels C', where C'=p'*C out / q' and C out is a multiple of q', and C', p', and q' are (positive) integers. A similar condition can be held for the output channels of the third neural network layer. The third number of channels C' is the number of the first number of channels C in and / or the number of second channels C out The condition C'=p'*C out / q' advantageously ensures that the number of input and output channels is an integer for both the first and second neural network layers.

[0015] When the neural network includes first, second, and third neural network layers, different additional conditions may also be satisfied. According to one implementation, the third channel number is less than the second channel number, which is less than the first channel number. This allows for a continuous reduction in tensor size while maintaining an integer number of channels. According to another implementation, the third channel number is less than the second channel number, which is greater than the first channel number. This allows for an increase in tensor size (e.g., for temporal hypothesis storage) followed by a decrease in tensor size (e.g., after input data is used for hypothesis selection) while maintaining an integer number of channels. According to another implementation, the third channel number is greater than the second channel number, which is greater than the first channel number. This allows for a continuous increase in tensor size while maintaining an integer number of channels.

[0016] According to another implementation, the neural network does not allow the third channel number to be greater than the second channel number and the second channel number to be less than the first channel number. In other words, in a neural network according to the first aspect and its implementations, the third channel number cannot be greater than the second channel number and the second channel number cannot be less than the first channel number. This prevents data loss and subsequent (inaccurate) data recreation.

[0017] According to another implementation, the first channel number C in and the number of second channels C out is a multiple of the chunk size. The third number of channels C' can be similar. Furthermore, the total number of channels in the neural network can be a multiple of the chunk size. The chunk size is, for example, 16 or 32. The first number of channels C in , and / or the second number of channels C out, and / or the third number of channels C′, and / or the total number of channels of the neural network may generally be a multiple of 16 or 32. The chunk size defines the number of output channels that can be processed in parallel by the associated computation means. The total number of output channels may be divided into chunks for parallel processing, each chunk having the chunk size. Providing the number of channels as a multiple of the chunk size may further increase the speed of the computations involved, for example, the computations involved in a video coding process. Providing the number of channels as a multiple of 16 or 32 may generally increase the speed of the computations involved, for example, the computations involved in a video coding process.

[0018] As already mentioned, the neural network according to the first aspect and its implementations can be advantageously applied for video encoding purposes. According to one implementation, the first neural network layer or neural network subnet is a hyperscale decoder subnet. According to another implementation, the first neural network layer or neural network subnet is a prediction fusion subnet. However, the present application is not limited to the first neural network layer or neural network subnet being a hyperscale decoder subnet or a prediction fusion subnet.

[0019] Note that in general, the following term hierarchy is used herein: A neural network includes (neural network) subnets, also called nets, and a (neural network) subnet includes one or more neural network layers. A subnet may include one or more other subnets. In some cases, the term neural network layer may be used for a neural network subnet.

[0020] According to a further implementation, the first neural network layer may include a data path (a first number of channels C) for at least one of the first-order and second-order components. in The primary component may be one of the RGB or YUV color components, and the secondary component may be another of the RGB or YUV color components. The coding of the secondary component may be based on the primary component, and the tensor size involved in processing the secondary component may be smaller than the tensor size involved in processing the primary component.

[0021] According to another implementation of the neural network according to the first aspect, the neural network includes at least one neural network subnet consisting of successive neural network layers, and for at least one neural network subnet, at least one of the following conditions is satisfied for the successive neural network layers, where the number of channels varies from one neural network layer to another: a) each successive neural network layer is configured to output, in processing order, only a number of channels that is either less than or greater than the number of channels received from a previous one of the successive neural network layers; b) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; i) each of the successive neural network layers of the first subset is configured to output, in processing order, only a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the first subset; and ii) each of the successive neural network layers of the second subset is configured to output, in processing order, only a number of channels that is less than the number of channels received from a previous one of the successive neural network layers of the second subset; c) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; i) each of the successive neural network layers of the first subset is configured to output, in processing order, only a number of channels that is less than the number of channels received from a previous one of the successive neural network layers of the first subset; ii) none of the successive neural network layers of the second subset is configured to output, in processing order, a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the second subset; and iii) the first neural network layer in the processing order of the successive neural network layers of the second subset is configured to output only a smaller number of channels than the number of channels received from the last neural network layer in the processing order of the successive neural network layers of the first subset.

[0022] This further develops the additional conditions of implantation mentioned above.

[0023] According to a further implementation, each successive neural network layer is configured to output a number of channels that is a multiple of 16 or 32.

[0024] According to a further implementation, the at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet.

[0025] According to a second aspect, a method for operating a neural network having a variable number of channels in a neural network layer is provided. The method for operating a neural network according to the second aspect and its implementations, as described below, can be implemented in a neural network according to the first aspect and its implementations. Furthermore, a neural network according to the first aspect and its implementations can be used by a method according to the second aspect. The method according to the second aspect and its implementations provides similar advantages as described above.

[0026] According to a second aspect, a method for operating a neural network having a variable number of channels in its neural network layers includes: in The first neural network layer receives the second number of channels C out The number of first channels is different from the number of second channels, and C out =p*C in / q and C in is a multiple of q and C in , C out , p, and q are (positive) integers, and the second neural network layer generates the second number of channels C out as input. The integer q may be equal to 2. For example, p / q=5 / 2 or p / q=2.

[0027] According to one implementation, the method according to the second aspect further includes the steps of outputting, by the second neural network layer, a third channel number C', and receiving, by the third neural network layer, the third channel number C', where C'=p'*C out / q' and C outis a multiple of q', and C', p', and q' are (positive) integers.

[0028] According to another implementation, one of the following conditions is met: the third channel number is smaller than the second channel number and the second channel number is smaller than the first channel number, or the third channel number is smaller than the second channel number and the second channel number is greater than the first channel number, or the third channel number is greater than the second channel number and the second channel number is greater than the first channel number.

[0029] According to another implementation, the third channel number is not greater than the second channel number and the second channel number is not less than the first channel number.

[0030] According to another implementation, the first channel number C in and the number of second channels C out is a multiple of the chunk size, which can be 16 or 32.

[0031] According to another implementation, the first neural network layer, or a subnet of the neural network, is a hyperscale decoder net.

[0032] According to another implementation, the first neural network layer, or a subnet of the neural network, is a predictive fusion net.

[0033] According to another implementation, the first neural network layer includes a data path for the first-order component or / and the second-order component.

[0034] According to another implementation, the neural network includes at least one neural network subnet consisting of successive neural network layers, and for at least one neural network subnet, at least one of the following conditions is satisfied for the successive neural network layers, where the number of channels varies from one neural network layer to another: a) each successive neural network layer outputs, in processing order, only a number of channels that is less than or greater than the number of channels it receives from a previous neural network layer in the successive neural network layer; b) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; i) each of the successive neural network layers of the first subset outputs, in processing order, only a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the first subset; and ii) each of the successive neural network layers of the second subset outputs, in processing order, only a number of channels that is less than the number of channels it receives from a previous one of the successive neural network layers of the second subset; c) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; i) each of the successive neural network layers of the first subset outputs, in processing order, only a number of channels that is less than the number of channels it receives from a previous one of the successive neural network layers of the first subset; ii) none of the successive neural network layers of the second subset outputs, in processing order, a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the second subset; and iii) the first neural network layer in the processing order of the successive neural network layers of the second subset outputs only a number of channels that is less than the number of channels received from the last neural network layer in the processing order of the successive neural network layers of the first subset.

[0035] According to another implementation, the method includes outputting, by each successive neural network layer, a number of channels that is a multiple of 16 or 32.

[0036] According to another implementation, the at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet.

[0037] Further, there is provided a method of encoding data, comprising steps according to the method of operating a neural network according to the second aspect and any implementation thereof.

[0038] Further, there is provided a method of decoding encoded data, comprising steps according to the method of operating a neural network according to the second aspect and any implementation thereof.

[0039] Further provided is a computer program product comprising program code stored on a non-transitory medium, which when executed on one or more processors performs the method of operating a neural network according to the second aspect and any implementation thereof.

[0040] Further provided is an apparatus for encoding data, wherein the apparatus includes processing circuitry configured to perform steps of the method for operating a neural network according to the second aspect and any implementation thereof.

[0041] Further provided is an apparatus for decoding data, wherein the apparatus includes processing circuitry configured to perform the steps of the method for operating a neural network according to the second aspect and any implementation thereof.

[0042] Further provided is an apparatus for decoding at least a portion of an encoded image, the apparatus comprising a processing circuit configured to provide an entropy model, and performing steps of a method of operating a neural network according to the second aspect and any implementation thereof, processing a bitstream with the neural network based on the provided entropy model to obtain latent tensors representing components of the image, and processing the latent tensors to obtain tensors representing components of the image.

[0043] Further, according to the first aspect or any implementation thereof, there is provided an apparatus for encoding at least a portion of an image, comprising a neural network.

[0044] Further, there is provided an apparatus for decoding at least a portion of an encoded image, comprising a neural network according to the first aspect or any implementation thereof.

[0045] The details of one or more embodiments are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawings]

[0046] In the following, embodiments of the invention will be explained in more detail with reference to the accompanying drawings. [Figure 1] FIG. 1 is a schematic diagram showing the channels processed by the layers of a neural network. [Figure 2] Figure 2 is a schematic diagram showing an autoencoder type of neural network. [Figure 3A] FIG. 3A is a schematic diagram illustrating an example network architecture for the encoder and decoder side, including a hyperprior model. [Figure 3B] FIG. 3B is a schematic diagram showing a general network architecture for the encoder side including the hyperplier model. [Figure 3C] FIG. 3C is a schematic diagram showing a general network architecture on the decoder side including the hyperplier model. [Figure 4] FIG. 4 is a schematic diagram illustrating an exemplary network architecture for the encoder and decoder side, including the hyperplier model. [Figure 5] FIG. 5 is a block diagram illustrating the architecture of a cloud-based solution for machine-based tasks, such as machine vision tasks. [Figure 6A] FIG. 6A is a block diagram illustrating a neural network-based end-to-end video compression framework. [Figure 6B] FIG. 6B is a block diagram illustrating some exemplary details of applying a neural network for motion field compression. [Figure 6C] FIG. 6C is a block diagram illustrating some exemplary details of applying a neural network for motion compensation. [Figure 7A] FIG. 7A is an exemplary AI trainable image codec architecture. [Figure 7B]FIG. 7B is a diagram related to convolution and deconvolution having a variable number of channels (some examples of Cin > Cout and Cin < Cout). [Figure 8] FIG. 8 is a block diagram showing an exemplary encoder - side sub - network. [Figure 9A] FIG. 9a is a block diagram showing an exemplary decoder - side sub - network comprising a hyper - scale decoder sub - network and a prediction fusion sub - network. [Figure 9B] FIG. 9b is a block diagram showing an alternative architecture of the hyper - scale decoder sub - network as compared to that shown in FIG. 9a. [Figure 9C] FIG. 9c is a block diagram showing an alternative architecture of the prediction fusion sub - network as compared to that shown in FIG. 9a. [Figure 10] FIG. 10 is a diagram showing some exemplary commonly used NN elements. [Figure 11] FIG. 11 is an example for explaining the implementation of the present embodiment as compared to the conventional method. [Figure 12] FIG. 12 is another example for explaining the implementation of the present embodiment as compared to the conventional method. [Figure 13] FIG. 13 is a block diagram showing an exemplary encoding device or decoding device. [Figure 14] FIG. 14 is a block diagram showing another exemplary encoding device or decoding device. [Figure 15] FIG. 15 is a block diagram showing an exemplary video coding system configured to implement an embodiment of the present disclosure. [Figure 16] FIG. 16 is a block diagram showing another exemplary video coding system configured to implement an embodiment of the present disclosure. [Figure 17] FIG. 17 is a block diagram showing an exemplary encoding device or decoding device. [Figure 18]FIG. 18 is a block diagram illustrating another exemplary encoding or decoding device. [Figure 19] FIG. 19 is a block diagram illustrating another exemplary encoding or decoding device. [Figure 20] FIG. 20 is a block diagram illustrating a neural network including a first neural network layer and a second neural network layer according to one embodiment. [Figure 21] FIG. 21 is a flow chart illustrating a method for operating a neural network with a variable number of channels in the neural network layer, according to one embodiment.

[0047] Like reference numbers and designations in different drawings may indicate like elements. DETAILED DESCRIPTION OF THE INVENTION

[0048] In the following description, reference is made to the accompanying drawings, which form a part of this disclosure and which show, by way of illustration, specific aspects of embodiments of the present disclosure or in which embodiments of the present disclosure may be used. It is understood that embodiments of the present disclosure may be used in other ways and may include structural or logical changes not shown in the drawings. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present disclosure is defined by the appended claims.

[0049] For example, it is understood that disclosure related to a described method also applies to a corresponding device or system configured to perform the method, and vice versa. For example, when one or more particular method steps are described, a corresponding apparatus may include one or more units for performing the described one or more method steps (e.g., one unit performing one or more steps, or multiple units performing one or more of the steps, respectively), even if such one or more units are not explicitly described or shown in the drawings. On the other hand, for example, when a particular apparatus is described based on one or more units, e.g., functional units, a corresponding method may include one step of performing the function of the one or more units (e.g., one step of performing the function of one or more units, or multiple steps of performing one or more of the functions, respectively), even if such one or more steps are not explicitly described or shown in the drawings. Furthermore, it is understood that features of various exemplary embodiments and / or aspects described herein may be combined with each other, unless otherwise specified.

[0050] Below is provided an overview of some used technical terms and frameworks in which embodiments of the present disclosure may be employed.

[0051] Artificial Neural Networks Artificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that make up animal brains. Such systems generally "learn" to perform tasks by considering examples, without being programmed with task-specific rules. For example, in image recognition, they can learn to identify cats in other images and to identify images containing cats by analyzing example images manually labeled as "cat" or "no cat" and using the results. They do this without any prior knowledge of cats, such as having fur, tails, whiskers, or cat-like faces. Instead, they automatically generate discriminative characteristics from the examples they process.

[0052] ANNs are based on a collection of connected units or nodes called artificial neurons, loosely modeling neurons in a biological brain. Each connection can send a signal to other neurons, like a synapse in a biological brain. An artificial neuron that receives a signal can then process it and send the signal to the neurons connected to it.

[0053] In an ANN implementation, the "signals" in the connections are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. This connection is called an edge. Neurons and edges typically have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have thresholds such that a signal is sent only if the aggregate signal exceeds that threshold. Neurons are typically aggregated into layers. Different layers may perform different transformations on their inputs. A signal travels from the first layer (input layer) to the last layer (output layer), possibly after traversing the layers multiple times.

[0054] The original goal of the ANN approach was to solve problems in the same way that the human brain does. Over time, attention has shifted to performing specific tasks, resulting in a deviation from biology. ANNs are used for a variety of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, medical diagnosis, and even activities traditionally thought of as the preserve of humans, such as painting.

[0055] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution, which is a special kind of linear operation. A convolutional network is a neural network that uses convolution instead of the common matrix multiplication in at least one of its layers.

[0056] Figure 1 shows a schematic of the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of an input layer, an output layer, and multiple hidden layers. The input layer is the layer to which input (such as portion 11 of an input image as shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve using multiplication or other dot products. The result of a layer is one or more feature maps (denoted by solid, empty rectangles), sometimes also called channels. There may be resampling (e.g., subsampling) involved in some or all of the layers. As a result, the feature maps may be smaller, as shown in Figure 1. Note that convolution using strides may also reduce (resampling) the size of the input feature maps. The activation function in a CNN is a Rectified Linear Unit (ReLU) layer or a Leaky ReLU, followed by additional convolutions: pooling, fully connected, and normalization layers, called hidden layers because their inputs and outputs are masked by the activation function and the final convolution. The layer is colloquially called a convolution, but this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. The index in the matrix is ​​important in that it affects how the weight is determined at a particular index point.

[0057] As shown in Figure 1, when programming a CNN to process images, the input is a tensor with the shape (number of images) × (image width) × (image height) × (image depth). It is known that the image depth can be composed of the image channels. After passing through the convolutional layer, the image is abstracted into a feature map with the shape (number of images) × (feature map width) × (feature map height) × (feature map channels). A convolutional layer in a neural network has the following attributes: A convolution kernel defined by its width and height (hyperparameters); The number of input and output channels (hyperparameters); The depth of the convolutional filter (input channels) is equal to the number of channels of the input feature map (depth).

[0058] In the past, traditional multilayer perceptron (MLP) models have been used for image recognition. However, due to full connectivity between nodes, they suffer from high dimensionality and do not scale well for higher-resolution images. A 1000 x 1000 pixel image with RGB color channels has 3 million weights, which is too high to be efficiently processed at a scale with full connectivity. Also, such network architectures do not take into account the spatial structure of the data and treat input pixels that are far apart the same as pixels that are close to each other. This ignores the locality of reference in the image data, both computationally and semantically. Therefore, full connectivity of neurons is useless for purposes such as image recognition, which are dominated by spatially local input patterns.

[0059] Convolutional neural networks are biologically inspired variants of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by exploiting the strong spatially local correlations present in natural images. The convolutional layer is the core building block of CNNs. The layer's parameters consist of a set of learnable filters (the kernels described above) that have small receptive fields but extend throughout the entire depth of the input volume. During a forward pass, each filter is convolved across the width and height of the input volume, computing dot products between the filter's entries and the input to generate a two-dimensional activation map for that filter. As a result, the network learns filters that become active when it detects certain types of features at certain spatial locations in the input.

[0060] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of a convolutional layer. Therefore, every entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with neurons in the same activation map. A feature map, or activation map, is the output activation for a given filter. Feature map and activation have the same meaning. In some papers, it is called an activation map because it is a mapping corresponding to the activation of different parts of the image, and it is also called a feature map because it is a mapping of where certain types of features are found in the image. High activation means that the certain features are found.

[0061] Another important concept in CNNs is pooling, a form of nonlinear downsampling. There are several nonlinear functions to perform pooling, among which max pooling is the most common. It partitions the input image into a set of non-overlapping rectangles and outputs the maximum value for each of those subregions.

[0062] Intuitively, the precise location of a feature is less important than its coarse location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers progressively reduce the spatial size of the representation, reducing the number of parameters in the network, memory footprint, and computational effort, and therefore also serving to control overfitting. It is common to periodically insert pooling layers between successive convolutional layers in CNN architectures. The pooling operation provides another form of translation invariance.

[0063] A pooling layer operates independently on every depth slice of the input and spatially resizes it. The most common form is a pooling layer with a size 2x2 filter applied at each depth slice in the input by every 2 along both width and height, with a stride of 2, discarding 75% of the activations. In this case, each max operation exceeds the number of 4. The depth dimension remains unchanged. In addition to max pooling, the pooling unit can use other functions, such as average pooling or l2-norm pooling. Average pooling was often used historically, but has recently fallen out of favor compared to max pooling, which often performs better in practice. Due to the aggressive reduction in representation size, there has been a recent trend toward using smaller filters or discarding pooling layers entirely. "Region of Interest" pooling (also known as ROI pooling) is a variant of max pooling in which the output size is fixed and the input rectangle is a parameter. Pooling is a key component of convolutional neural networks for object detection based on the Faster R-CNN architecture.

[0064] ReLU stands for rectified linear unit and is an application of a non-saturating activation function. It effectively removes negative values ​​from the activation map by setting them to zero. This increases the nonlinearity of the decision function and the entire network without affecting the receptive field of the convolutional layer. Other functions are also used to increase nonlinearity, such as saturated hyperbolic tangent and sigmoid functions. ReLU is often preferred over other functions because it trains neural networks several times faster without a significant penalty to generalization accuracy.

[0065] Leaky Modified Linear Unit (ReLU), or leaky ReLU, is an activation function based on ReLU, but with a small slope for negative values ​​instead of a flat slope. The slope coefficient is determined before training; it is not learned during training. This type of activation function is common in tasks with sparse gradients, such as training generative adversarial networks. Leaky ReLU applies an element-wise function.

number

[0066] After several convolutional and max-pooling layers, high-level inference in neural networks occurs via fully connected layers. Neurons in fully connected layers have connections to all activations in previous layers, as in typical (non-convolutional) artificial neural networks. Therefore, their activations can be computed as affine transformations using matrix multiplication followed by bias offsets (vector addition of learned or fixed bias terms).

[0067] The "loss layer" (which contains the computation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is typically the final layer of a neural network. Various loss functions suitable for different tasks can be used: Softmax loss is used to predict a single class out of K mutually exclusive classes; Sigmoid cross entropy loss is used to predict K independent probability values ​​in [0,1]; Euclidean loss is used to regress to real-valued labels.

[0068] In summary, Figure 1 shows the data flow in a typical convolutional neural network. First, an input image passes through a convolutional layer and is abstracted into a feature map containing several channels, corresponding to the number of filters in the layer's set of learnable filters. The feature map is then subsampled, for example, using a pooling layer, which reduces the dimensionality of each channel in the feature map. Next, the data comes to another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels is a layer hyperparameter. To establish network connectivity, these parameters need to be synchronized between two connected layers so that the number of input channels for the current layer is equal to the number of output channels of the previous layer. For the first layer, which processes input data, e.g., an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., three channels for an RGB or YUV representation of an image or video, or one channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly a resampling layer) can be passed to an output layer. Such an output layer may be convolutional or resampled in some implementations. In an exemplary and non-limiting implementation, the output layer is a fully connected layer.

[0069] Autoencoders and Unsupervised Learning An autoencoder is a type of artificial neural network used to learn efficient data encodings in an unsupervised manner. A schematic diagram is shown in Figure 2. An autoencoder includes an encoder side 210 with input x fed into the input layer of an encoder subnetwork 220, and a decoder side 250 with output x' output from a decoder subnetwork 260. The goal of an autoencoder is to learn a representation (encoding) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise." Along with the reduction (encoder) side subnetwork 220, a reconstruction (decoder) side subnetwork 260 is trained, where the autoencoder attempts to generate a representation x' (hence its name) from the reduced encoding 230 that is as close as possible to its original input x. In the simplest case, given a single hidden layer, the encoder stage of an autoencoder takes the input x and maps it to h.

number

[0070] This image h is typically called the code 230, latent variable, or latent representation. Here, σ is an element-wise activation function such as a sigmoid function or a rectified linear unit. W is a weight matrix, and b is a bias vector. The weights and biases are often initialized randomly and then iteratively updated through backpropagation during training. The decoder stage of the autoencoder then maps h to a reconstruction x' that is identical in shape to x.

number

[0071] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use a variational approach for latent representation learning, resulting in an additional loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. It is based on the assumption that the data is represented by a directed graphical model p θ (x|h), and the encoder has a posterior distribution p θ Approximation q to (x|h) θ We assume that we are learning (x|h), where φ and θ denote the parameters of the encoder (recognition model) and decoder (generative model), respectively. The probability distribution of the latent vectors in a VAE typically matches the probability distribution of the training data much more closely than standard autoencoders. The objective of a VAE has the following form:

number

[0072] where D KL represents the Kullback-Leibler divergence. The priors for the latent variables are usually centrally isotropic multivariate Gaussians.

number

number

[0073] Recent advances in the field of artificial neural networks, and in particular convolutional neural networks, have enabled researchers to focus on applying neural network-based techniques to image and video compression tasks. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0074] Data compression is therefore considered a fundamental and well-studied problem in engineering and is commonly formulated with the goal of designing a code for a given discrete data ensemble with minimal entropy. The solution relies heavily on knowledge of the probabilistic structure of the data, and the problem is therefore closely related to probabilistic source modeling. However, because all practical codes must have finite entropy, continuous-valued data (such as a vector of image pixel intensities) must be quantized into a finite set of discrete values, which introduces error.

[0075] In this context, known as a lossy compression problem, two competing costs must be traded off: the entropy (rate) of the discretized representation and the error (distortion) resulting from quantization. Different compression applications, such as data storage or transmission over limited-capacity channels, require different rate-distortion tradeoffs.

[0076] Joint optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is intractable. For this reason, most existing image compression methods operate by linearly transforming a data vector into an appropriate continuous-valued representation, independently quantizing its elements, and then encoding the resulting discrete representation using a lossless entropy code. This scheme is called transform coding, due to the central role of the transform.

[0077] For example, JPEG uses a discrete cosine transform on blocks of pixels, and JPEG 2000 uses multi-scale orthogonal wavelet decomposition. Typically, the three components of a transform encoding method—the transform, the quantizer, and the entropy code—are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. Several transforms are used for that purpose, such as the discrete cosine and sine transform (DCT, DST) and the low frequency non-separable manually optimized transform (LFNST).

[0078] Variational image compression The Variable Auto-Encoder (VAE) framework can be considered as a nonlinear transform encoding model. The transform process can be mainly divided into four parts. This is illustrated in Figure 3A, which shows the VAE framework.

[0079] The transformation process can be divided into four main parts. Figure 3A illustrates the VAE framework. In Figure 3A, an encoder 101 maps an input image x to a latent representation (denoted as y) via a function y = f(x). This latent representation is also referred to below as a portion or point in the "latent space". The function f() is a transformation function that transforms the input signal x into a more compressible representation y. A quantizer 102 converts the latent representation y into a quantized latent representation

number

[0080] Latent space can be understood as a compressed data representation where similar data points are close to each other in the latent space. Latent space is useful for learning data features and finding simpler representations of data for analysis. The quantized latent representation, T, ŷ, and the side information of the hyperpriority 3 are

number

number

[0081] 3A, component AE 105 is an arithmetic encoding module that converts samples of the quantized latent representation ŷ and side information ẑ into a binary representation bitstream 1. The samples of ŷ and ẑ may, for example, comprise integers or floating-point numbers. One purpose of the arithmetic encoding module is to convert the sample values ​​(through a process of binarization) into a string of binary digits (which are then included in a bitstream that may comprise further portions corresponding to the encoded image or further side information).

[0082] Arithmetic decoding (AD) 106 is a process that reverses the binarization process, where the binary digits are converted back to sample values. Arithmetic decoding is provided by arithmetic decoding module 106.

[0083] Note that the present disclosure is not limited to this particular framework. Furthermore, the present disclosure is not limited to image or video compression, and may be applied to object detection, image generation, and recognition systems as well.

[0084] In FIG. 3A, there are two interconnected sub-networks. A sub-network in this context is a logical division between parts of an overall network. For example, in FIG. 3A, modules 101, 102, 104, 105, and 106 are called the "Encoder / Decoder" sub-network. The "Encoder / Decoder" sub-network is responsible for encoding (generating) and decoding (parsing) a first bitstream, "bitstream1." The second network in FIG. 3A includes modules 103, 108, 109, 110, and 107 and is called the "Hyper-Encoder / Decoder" sub-network. The second sub-network is responsible for generating a second bitstream, "bitstream2." The two sub-networks have different purposes.

[0085] The first subnetwork is responsible for: Transforming an input image x into its latent representation y (which is more compressible than x) Quantizing the latent representation y into the quantized latent representation y^ Compressing the quantized latent representation ŷ using AE by an arithmetic encoding module 105 to obtain a bitstream “Bitstream 1” Parsing the bitstream 1 via AD using an arithmetic decoding module 106 ● Using the analyzed data, reconstruct the reconstructed image (x^)

[0086] The purpose of the second sub-network is to obtain statistical properties of the samples of "Bitstream 1" (e.g., mean, variance, and correlation between samples of Bitstream 1), so that the first sub-network can compress Bitstream 1 more efficiently. The second sub-network generates a second bitstream, "Bitstream 2," that includes said information (e.g., mean, variance, and correlation between samples of Bitstream 1).

[0087] The second network includes converting 103 the quantized latent representation ŷ into side information z, quantizing the side information z into quantized side information ẑ, and encoding (e.g., binarizing) 109 the quantized side information ẑ into bitstream 2. In this example, binarization is performed by arithmetic encoding (AE). The decoding portion of the second network includes arithmetic decoding (AD) 110, which converts the input bitstream 2 into decoded quantized side information ẑ'. Because arithmetic encoding and decoding are lossless compression methods, ẑ' may be identical to ẑ. The decoded quantized side information ẑ' is then converted 107 into decoding side information ŷ', which represents a statistical property of ŷ (e.g., the mean value of the samples of ŷ or the variance of the sample values). The decoded latent representation y' is then provided to the arithmetic encoder 105 and arithmetic decoder 106 described above, which control the probability model of y.

[0088] 3A illustrates an example of a VAE (Variable Auto-Encoder), the details of which may vary in different implementations. For example, in a particular implementation, additional components may be present to more efficiently obtain statistical characteristics of samples of bitstream 1. In one such implementation, there may be a context modeler that aims to extract cross-correlation information of bitstream 1. The statistical information provided by the second sub-network may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0089] Figure 3A shows the encoder and decoder in a single diagram. As will be apparent to those skilled in the art, the encoder and decoder may be, and very often are, embedded in different devices.

[0090] FIG. 3B shows the encoder component of the VAE framework, and FIG. 3C shows the decoder component of the VAE framework in isolation. According to some embodiments, the encoder receives a picture as input. The input picture may include one or more channels, such as color channels or other types of channels, e.g., depth channels or motion information channels. The outputs of the encoder (as shown in FIG. 3B) are Bitstream 1 and Bitstream 2. Bitstream 1 is the output of the encoder's first sub-network, and Bitstream 2 is the output of the encoder's second sub-network.

[0091] Similarly, in FIG. 3C, two bitstreams, Bitstream 1 and Bitstream 2, are received as input, and a reconstructed (decoded) image, ẑ, is produced at the output. As mentioned above, the VAE can be divided into different logical units that perform different actions. This is illustrated in FIG. 3B and FIG. 3C, where FIG. 3B shows the components involved in encoding a signal, such as video, and provided encoding information. This encoded information is then received, for example, by the decoder component shown in FIG. 3C for encoding. Note that the encoder and decoder components labeled 12x and 14x may correspond in function to the components described above in FIG. 3A and labeled 10x.

[0092] Specifically, as seen in Figure 3B, the encoder includes an encoder 121 that processes an input x into a signal y, which is then provided to a quantizer 322. The quantizer 122 provides information to an arithmetic encoding module 125 and a hyper-encoder 123. The hyper-encoder 123 provides bitstream 2, already described above, to a hyper-decoder 147, which in turn provides its information to the arithmetic encoding module 105 (125).

[0093] The output of the arithmetic encoding module is Bitstream 1. Bitstream 1 and Bitstream 2 are the output of the signal encoding, which are then provided (sent) to the decoding process. Unit 101 (121) is called an "encoder," but the complete subnetwork illustrated in FIG. 3B can also be called an "encoder." The encoding process generally refers to a unit (module) that converts an input into an encoded (e.g., compressed) output. From FIG. 3B, it can be seen that unit 121 can actually be considered the core of the entire subnetwork, since it converts input x into y, which is a compressed version of x. Compression in encoder 121 can be achieved, for example, by applying a neural network, or in general, any processing network with one or more layers. In such a network, compression can be performed by cascading processes, including downsampling to reduce the size and / or number of input channels. Therefore, the encoder can be called, for example, a neural network (NN)-based encoder.

[0094] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all parts responsible for improving the efficiency of the encoding process or converting the compressed output y into a series of bits (bitstream). Quantization can be provided to further compress the output of the NN encoder 121 by lossy compression. The combination of the hyperencoder 123 and hyperdecoder 127 used to form the AE 125 can perform binarization, which can further compress the quantized signal by lossless compression. Therefore, the entire subnetwork of FIG. 3B can also be referred to as an "encoder."

[0095] Most Deep Learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it into binary digits (bits). For example, in a VAE framework, an encoder, which is a nonlinear transform, maps an input image x to y, where y has a smaller width and height than x. Because y has a smaller width and height and therefore a smaller size, the dimensionality of the signal is reduced, and therefore it is easier to compress signal y. Note that, in general, an encoder does not necessarily need to reduce the size in both (or, in general, all) dimensions. Rather, some example implementations may provide an encoder that reduces the size in only one (or, in general, a subset thereof) dimension.

[0096] In "Density Modeling of Images Using a Generalized Normalization Transformation" (2015), an arXiv e-print published at the 4th International Conference on Learning Representation in 2016 (hereafter referred to as "Bale"), the authors propose a framework for end-to-end optimization of image compression models based on nonlinear transformations. The authors optimize for mean squared error (MSE) but use a more flexible transformation constructed from a cascade of linear convolutions and nonlinearities. Specifically, they use a generalized divisive normalization (GDN) combined nonlinearity inspired by models of neurons in the biological visual system and proven effective for Gaussianizing image densities. This cascade transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), which effectively implements a parametric form of vector quantization on the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0097] Such an example of a VAE framework is shown in Figure 4 and utilizes six downsampling layers marked with 401 to 406. The network architecture includes a hyperprior model. a ,g s ) shows the image autoencoder architecture, and the right side (h a ,h s ) corresponds to an autoencoder that implements a hyperplier. The factorized prior model is a function of the analytic transformation g a and synthetic transformation g sThe same architecture is used for the arithmetic encoder and decoder. Q represents quantization, and AE and AD represent arithmetic encoder and decoder, respectively. The encoder converts the input image x into g a The encoding g is then generated as a response y (latent representation) with a spatially varying standard deviation. a contains multiple convolutional layers with subsampling and generalized divisive normalization (GDN) as the activation function.

[0098] The response is h a and summarizing the distribution of the standard deviation in z. z is then quantized, compressed, and transmitted as side information.

number

[0099] Layers that include downsampling are indicated with a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer with N channels and the convolution kernel is k1 x k1 in size. For example, k1 may be equal to 5 and k2 may be equal to 3. As mentioned above, 2↓ means that downsampling by a factor of 2 is performed in this layer. Downsampling by a factor of 2 results in one of the dimensions of the input signal being reduced by half at the output. In FIG. 4, 2↓ indicates that both the width and height of the input image are reduced by a factor of 2. Because there are six downsampling layers, if the width and height of the input image 414 (also shown using x) are given by w and h, then the output signal ẑ 413 has a width and height equal to w / 64 and h / 64, respectively. The modules denoted by AE and AD are an arithmetic encoder and an arithmetic decoder, which are described with reference to FIG. 3. The arithmetic encoder and decoder are specific implementations of entropy coding. AE and AD may be replaced by other means of entropy coding. In information theory, entropy coding is a lossless data compression method used to convert symbol values ​​into binary representations, which is a revertible process. Also, "Q" in the diagram corresponds to the quantization operation described above in connection with FIG. 4 and is further described in the "Quantization" section. Also, the quantization operation and the corresponding quantization unit as part of component 413 or 415 are not necessarily present and / or may be replaced by other units.

[0100] Also shown in Figure 4 is a decoder including upsampling layers 407-412. A further layer 420 is provided between upsampling layers 411 and 410 in the input processing order, which is implemented as a convolutional layer but does not provide upsampling to the received input. A corresponding convolutional layer 430 is also shown for the decoder. Such a layer may be provided in a neural network to perform operations on the input that do not change the size of the input but do change certain characteristics. However, such a layer is not necessarily provided.

[0101] In terms of the processing order of bitstream 2 through the decoder, the upsampling layers are performed in reverse order, from upsampling layer 412 to upsampling layer 407. Here, each upsampling layer is shown to provide upsampling with an upsampling ratio of 2, indicated by ↑. Of course, it is not necessary for all upsampling layers to have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8, etc. may also be used. Layers 407-412 are implemented as convolutional layers (conv). Specifically, an upsampling layer may be intended to provide an operation on the input that is the inverse of the encoder's operation, such that the upsampling layer can apply a deconvolution operation to the received input, thereby increasing its size by a factor corresponding to the upsampling ratio. However, the present disclosure is not generally limited to deconvolution, and upsampling may be performed in any other manner. By bilinear interpolation between two adjacent samples, or by nearest neighbor sample copying, etc.

[0102] In the first sub-network, several convolutional layers (401-403) are followed by generalized divisive normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second sub-network, the activation function applied is ReLu. Note that the present disclosure is not limited to such implementation and, in general, other activation functions can be used instead of GDN or ReLu.

[0103] Cloud solutions for machine tasks Video Coding for Machines (VCM) is another popular direction in computer science these days. The main idea behind this approach is to transmit a coded representation of image or video information intended for further processing by computer vision (CV) algorithms, such as object segmentation, detection, and recognition. In contrast to traditional image and video encoding, which are intended for human perception, the quality characteristic is not the reconstructed quality, but rather the performance of the computer vision task, e.g., object detection accuracy. This is illustrated in Figure 5.

[0104] Machine-oriented video coding, also known as collaborative intelligence, is a relatively new paradigm for efficient deployment of deep neural networks across mobile cloud infrastructures. By dividing the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload, thereby minimizing the overall energy and / or latency of the system. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed among two or more different computational nodes, e.g., devices, but generally, any functionally defined node. Here, the term “node” does not refer to the nodes of the neural network described above. Rather, a (computational) node here refers to a separate device / module (physically or at least logically) that implements parts of the neural network. Such devices can be a mixture of different servers, different end-user devices, servers and / or user devices, and / or clouds, and / or processors, etc. In other words, computational nodes can be considered as nodes that belong to the same neural network and communicate with each other to transfer coded data within / for the neural network. For example, to enable the execution of complex computations, one or more layers may be executed on a first device (such as a device on the mobile side 510) and one or more layers may be executed on another device (such as a cloud server on the cloud side 590). However, the distribution may also be finer, and a single layer may be executed on multiple devices. In this disclosure, the term "plurality" refers to two or more. In some existing solutions, part of the neural network function is executed on a device (such as a user device or an edge device) or multiple such devices, and then the output (feature map) is passed to the cloud.The cloud is a collection of processing or computing systems located outside the device that run parts of the neural network. The concept of collaborative intelligence extends to model training as well, where data flows both ways: from cloud to mobile during the backward propagation phase of training, and from mobile to cloud during the forward pass of training (shown in Figure 5), as well as for inference.

[0105] Some studies have presented semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization was demonstrated, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile side 510 to the cloud 590 rather than sending the compressed natural image data to the cloud and then performing object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520 for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps is useful for image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, such as arithmetic encoding, are a common approach to compressing deep features (i.e., feature maps).

[0106] Nowadays, video content contributes to over 80% of Internet traffic, and this proportion is expected to further increase. Therefore, it is important to build an efficient video compression system and generate higher-quality frames within a given bandwidth budget. In addition, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of the compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, techniques in video compression are also useful for action recognition and model compression. However, in the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancies in video sequences, as mentioned above. Although each module is well-designed, the entire compression system has not been optimized end-to-end. It is desirable to further improve video compression performance by jointly optimizing the entire compression system.

[0107] End-to-end image or video compression DNN-based image compression methods can utilize large-scale end-to-end training and highly nonlinear transformations not used in conventional approaches. However, directly applying these techniques to build end-to-end learning systems for video compression is not trivial. First, learning how to generate and compress tailored motion information for video compression remains an open problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0108] A straightforward solution is to use learning-based optical flow to represent motion information. However, current learning-based optical flow approaches aim to generate a flow field that is as accurate as possible. Accurate optical flow is often suboptimal for a particular video task. In addition, the amount of optical flow data increases significantly compared to motion information in conventional compression systems, and directly applying existing compression approaches to compress optical flow values ​​significantly increases the number of bits required to store motion information. Second, it is unclear how to build a DNN-based video compression system by minimizing a rate-distortion-based objective for both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher-quality reconstructed frames (i.e., less distortion) given the number of bits (or bitrate) for compression. RDO is critical for video compression performance. To take advantage of the power of end-to-end training for learning-based compression systems, an RDO strategy that optimizes the entire system is needed.

[0109] In Guo Lu, Wanli Ouyang, Dong Xu, Xiaoyun Zhang, Chunlei Cai, and Zhiyong Gao, "DVC: An End-to-end Deep Video Compression Framework," in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 11006–11015, the authors propose an end-to-end deep video compression (DVC) model that jointly learns motion estimation, motion compression, and residual encoding.

[0110] Such an encoder is shown in Figure 6A. In particular, Figure 6A shows the overall structure of an end-to-end trainable video compression framework. To compress motion information, we use the optical flow v t to the corresponding representation m t A CNN was specified to convert the image to a motion vector (MV) compression network. Specifically, an autoencoder-type network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 6B. The network architecture is somewhat similar to that of Figure 4. a / g s In particular, optical flow v t is fed to a series of convolution operations and nonlinear transformations including GDN and IGDN. The number of output channels for convolution (deconvolution) c is illustratively 128 here, except for the last deconvolution layer, which is equal to 2 in this example. The kernel size is k, e.g., k=3. Given an optical flow with size M×N×2, the MV encoder generates a motion representation m of size M / 16×N / 16×128. t The motion representation is then quantized (Q) and entropy coded to produce

number

number

[0111] Figure 6C shows the structure of the motion compensator, where the previous reconstructed frame x t-1Using the reconstructed motion information, a warping unit generates a warped frame (usually with the help of an interpolation filter, such as a bilinear interpolation filter). A separate CNN with three inputs then generates a predicted picture. The architecture of the motion-compensated CNN is also shown in Figure 6C.

[0112] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to convert the residual into a corresponding latent representation. Compared with the discrete cosine transform in traditional video compression systems, this approach can better utilize the power of nonlinear transforms and achieve higher compression efficiency.

[0113] From the above overview, we can see that CNN-based architectures can be applied for both image and video compression, considering different parts of the video framework, including motion estimation, motion compensation, and residual encoding. Entropy coding is a common method used for data compression, which has been widely adopted by the industry, and is also applicable to feature map compression for either human perception or computer vision tasks.

[0114] Video encoding for machines Video Coding for Machines (VCM) is another computer science direction that has recently become popular. The main idea behind this approach is to transmit an encoded representation of image or video information intended for further processing by computer vision (CV) algorithms such as object segmentation, detection, and recognition. In contrast to traditional image and video encoding, which are intended for human perception, the quality characteristic is not the reconstructed quality but the performance of the computer vision task, e.g., object detection accuracy.

[0115] Recent research has proposed a new deployment paradigm called collaborative intelligence, whereby deep models are split between mobile and cloud. Extensive experiments under various hardware configurations and wireless connection modes have revealed that the optimal operating point in terms of energy consumption and / or computational latency often involves splitting the model at a deep point in the network. It turns out that today's common solutions, where the model is located entirely in the cloud or entirely on the mobile, are rarely (if ever) optimal. The concept of collaborative intelligence has also been extended to model training, where data flows both ways: from cloud to mobile during the backpropagation phase of training, and from mobile to cloud during the forward pass of training, as well as inference.

[0116] In the context of recent deep models for object detection, lossy compression of deep feature data has been investigated based on HEVC intra-coding. A degradation in detection performance with increasing compression level has been noted, and compression-augmented training has been proposed to minimize this loss by producing models that are more robust to quantization noise in feature values. However, this remains a suboptimal solution because the employed codec is highly complex and optimized for natural scene compression rather than deep feature compression.

[0117] The problem of deep feature compression for collaborative intelligence has been addressed by an approach for object detection tasks using the popular YOLOv2 network to study the tradeoff between compression efficiency and recognition accuracy. Here, the term "deep features" has the same meaning as feature map. The word "deep" comes from the idea of ​​collaborative intelligence when the output feature maps of several hidden (deep) layers are captured and transferred to the cloud to perform inference. This appears to be more efficient than sending compressed natural image data to the cloud and then using the reconstructed images to perform object detection.

[0118] Efficient compression of feature maps is useful for image and video compression and reconstruction, both for human perception and machine vision. The above-mentioned drawbacks of state-of-the-art autoencoder-based approaches to compression are also valid for machine vision tasks.

[0119] The quality metrics in the JPEG AI common training and test conditions (CTTC) are selected based on their correlation with human perception. Not a single, but seven different metrics, each sensitive to different types of artifacts, are used by JPEG AI. Most of the quality metrics are calculated in the YUV color space, and some of them have been noted to use only the luma component. The JPEG AI trainable image codec is built under the assumption that one color component contains the majority of the information and influences human perception more than the others. To extract this strongest color component, RGB inputs first undergo color conversion. The default color conversion is RGB->YUV BT.709 (full range). Adaptive color conversion using a signaling color transformation matrix is ​​supported.

[0120] Figure 7A shows one exemplary AI-trainable image codec architecture. Primary and secondary color components are encoded separately using networks with the same architecture but different numbers of channels. In Figure 7A, neural sub-networks / data flows used only on the encoder side are marked by dashed boxes / lines. Solid boxes indicate sub-networks used on the decoder (or both the decoder and encoder sides). All boxes with the same name are sub-networks with the same architecture, differing only in the size of the input-output tensors and the number of channels. Sub-networks are also referred to as nets in this disclosure.

[0121] The number of channels in the latent tensor can vary from layer to layer in a neural network. Figure 7B shows some typical cases of convolution and inverse convolution where the number of channels in the input and output tensors are different. Typically, neural network algorithms use C in (number of channels in the input tensor) and C out (the number of channels in the output tensor), which is often expressed as the ratio C out =p*C in / q.

[0122] In learnable image encoding, the input signal to be encoded is denoted as x, the latent space tensor at the bottleneck of the variational auto-encoder is y, and the hyperparameters (tensors at the bottleneck of the hyper-encoder) are z. A prediction fusion net generates a prediction μ for the latent space tensor y. The residual signal r=y-μ is assumed to be Gaussian with zero mean and variance σ, which is the output of the hyperscale decoder net:

number

[0123] The second-order components are coded jointly. The second-order component joint and associated tensors are marked with "UV" in Figure 7A. The second-order components are coded at a reduced resolution. Then, the second-order tensor color components and the first-order tensor color components before the analytical transformation net pass through a downsampling module. The input of the analytical transformation net for the second-order components is the concatenated tensor of the three color components at the reduced resolution, and the size of this tensor is W / 2 x W / 2 x 3. The output of the analytical transformation net for the second-order components is C s ×h YUV ×w UV The number of channels allocated for secondary component coding is C s = 64. The output of the hyperencoder net for the second-order component is C s ×h hpUV ×w hpUV is.

[0124] The prediction sub-networks, e.g., the prediction fusion net in Figure 7A, operate independently and have the same architecture for the first-order and second-order components. The prediction sub-network outputs a tensor of the same size w × h × C as the latent space tensor y. The prediction sub-network receives a tensor of the same spatial size, but with four times more channels: w × h × 4C. Half of the channels come as the output of the context model net, and the other half are the output of the hyper-decoder net.

[0125] Convolution with downsampling stride is part of the JPEG AI trainable codec architecture. Before a convolution with downsampling stride, a padding layer is inserted to ensure that the tensor size is always a multiple of the downsampling stride.

[0126] Figure 8 is a block diagram illustrating an example encoder-side subnetwork. For example, an analytic transform net may include a sequence of four convolutions with a downsampling stride of 2. The sizes of tensors in different parts of the analytic transform net are shown in Figure 8 as an example. As can be seen, the peak memory usage occurs after the first convolution layer.

[0127] As shown in Figure 8, each of these convolutions is preceded by a padding layer, which modifies the tensor size as follows: h t =ceil(h t-1 ,s),w t =ceil(w t-1 , s), where s is the downsampling convolution stride (s=2). The nonlinear activation layer in the analysis-transformation net is a residual nonlinear unit (ResAU) with an attention mechanism (e.g., ResAU block in Figure 10). ResAU consists of an element-wise nonlinear operation (ReLU), a convolution layer, an element-wise multiplication operation (tanh), and a residual connection. The detailed structure of ResAU used in the analysis-transformation net (also called a transformation module) is shown in Figure 10.

[0128] A residual non-local attention block (RNAB) is used in the analysis-transformation net. The RNAB is placed between the first downsampling convolution and the second downsampling convolution and plays the role of an attention mechanism. Because the RNAB (e.g., RNAB in Figure 10) includes one downsampling and one upsampling convolution, a series of residual blocks (e.g., RB shown in Figure 10), a padding layer is inserted before the RNAB. The RNAB also includes a series of residual blocks (e.g., RB shown in Figure 10).

[0129] The encoder side may also include a hyperencoder net. For example, as shown in Figure 8, the hyperencoder net may include two convolutions with a downsampling stride of 2, three convolutions without tensor resizing, and a sequence of ReLUs (e.g., ReLUs in Figure 10) as activations. The sizes of tensors at different parts of the analytic transform are shown in Figure 8. Similar to the analytic transform net, for each downsampling, the convolutions are preceded by a padding layer.

[0130] Bitstream #1 and bitstream #1 (in FIG. 7A) are encoded / decoded using a fixed probability density model. The discretized cumulative distribution function is stored in a predefined fixed table and used to analyze the quantized hyper prior latent ẑ. The quantized hyper prior latent ẑ is then processed by a hyper-scale decoder net, a NN-based sub-network used to generate the Gaussian distribution σ. After that, the quantized residual latent sample

number

number

[0131] Figure 9a is a block diagram illustrating the decoder-side subnetwork. The hyperscale decoder net 910 includes a sequence of two deconvolutions with upsampling stride 2, two convolutions without tensor resizing, and leaky ReLU as the tensor activation size in different parts of the analytic transform (e.g., leaky ReLU in Figure 10), shown below the block diagram in Figure 9. In contrast to the hyperencoder, each upsampling convolution layer is followed by a cropping operation.

[0132] The entire process is shown in Figure 9 as a Prediction Fusion Net 920, which is also called a Prediction Gather Net. At the start of the latent sample prediction process, an inverse transform operation is performed on the hyper-prior latent ẑ by the Hyper-Decoder Net 910. The output of this process is concatenated with the output of the context model sub-network and then processed by the Prediction Fusion Net 920 to generate a prediction value μ. The prediction value is then added to the quantized residual sample r̂ to obtain the quantized latent sample ẑ. The Hyper-Decoder Net (Figure 9) contains two deconvolutions with an upsampling stride of 2, three convolutions without tensor resizing, and a sequence with leaky ReLU as activations. In contrast to the Hyper-Encoder, each upsampling convolution layer is followed by a cropping operation. The predictive fusion net 920 (FIG. 9) consists of a sequence of convolutions with a decreasing number of channels.

[0133] As an example shown in Figure 9, the Synthesis transform Net contains a sequence of four convolutions with an upsampling stride of 2. The sizes of tensors in different parts of the Synthesis transform Net are shown in Figure 9. It can be noticed that the peak memory usage (which is also the maximum amount of multiplication operations) occurs before the last upsampling convolution layer. For a first-component synthesis transform, the number of channels is:in =128; C out =2 The number of channels for the quadratic component synthesis transform is: C=64; C in =128+64; C out =2

[0134] Note that the latent sample prediction process is an autoregressive process. However, the quantized latent samples in different rows

number

[0135] In JPEG AI VMuC-2.0, for example, in the "Hyper Scale Decoder Net" and "Prediction Fusion (Gather) Net," the tensor size in the channel dimension is fractional. In actual implementations, the fraction is truncated and data is lost.

[0136] This problem is specific to the CCS architecture adopted in JPEG AI VMuC-2.0 because the number of channels in the primary component pipeline (Cp) and secondary component pipeline (Cs) is not a multiple of 3. Almost all papers use a channel count of C=192 (which is a multiple of 3 because the three RGB colors are encoded together).

[0137] The number of channels in a neural network (NN) sub-network often varies from layer to layer. One typical example is a gather sub-network, which acts as a fusion of multiple hypotheses generated by preceding sub-networks. In this disclosure, the terms "gather" and "fusion" have the same meaning.

[0138] According to an embodiment, the NN design principles include at least one of the following conditions: 1. If the number of channels in a subnetwork varies from layer to layer, it must be an integer in all layers. C in The subnetwork design is only possible if is a multiple of q. out =p*C in / q, which may prevent data loss. In one example, q is equal to 2. 2. When the number of channels changes within a sub-network, it does not decrease and then increase within one sub-network. ●If the tensor sizes are (in processing order) C0, C1, …, Ck-1, Ck, …, CN, ● 0>C1>…>Ck-1>Ck>…>CN means the tensor size is reduced after the input data is used.

number

[0139] This can provide efficient NPU / GPU resource utilization.

[0140] To realize the above-mentioned NN design principles, an embodiment of the present invention provides a neural network, for example, as shown in FIGS. 1-10. The neural network includes a first neural network layer and a second neural network layer. The neural network layer receives a first channel number (C in ) and the second channel number (C out ) can be output, where the number of first channels is different from the number of second channels, and C out =p*C in / q and C in is a multiple of q, and C in , C out , p, and q are integers. The second neural network layer receives as input the second channel number (C out ) is configured to retrieve the

[0141] As one example, the first neural network layer can be a hyperscale decoder net as shown in Figures 7 and 9. Alternatively, the first neural network layer can be a predictive fusion (gather) net as shown in Figures 7 and 9.

[0142] The first neural network layer includes data paths for the first-order and / or second-order components. The second neural network layer has a similar structure to the first neural network layer. The neural network may further include a third neural network layer, the second neural network layer further configured to output a third channel number (C'), and the third neural network layer configured to obtain the third channel number (C'), where C'=p'*C out / q', C' is a multiple of q', and C', p', and q' are integers. The third neural network layer may have a similar structure to the first neural network layer. As described above, the third channel number is smaller than the second channel number and the second channel number is smaller than the first channel number, or the third channel number is smaller than the second channel number and the second channel number is greater than the first channel number, or the third channel number is greater than the second channel number and the second channel number is greater than the first channel number. However, it is not permitted for the third number of channels to be greater than the second number of channels and for the second number of channels to be less than the first number of channels. In a possible implementation, the first channel number (C in ) and the number of second channels (C out ) is a multiple of chunk size 8. Chunk size can be 16 or 32.

[0143] To realize an embodiment of the present invention, two examples are provided in Figures 11 and 12. As shown in Figures 11 and 12, in the conventional method, the output of Conv 5 / 3C'3 in the hyperscale decoder net is 213.3333 when Cp = 128 and 106.6667 when Cs = 64. Therefore, the output of the channel number is a non-integer. In the embodiment provided in Figure 11, the output of Conv 5 / 2C'3'3 in the hyperscale decoder net is 320 when Cp = 128 and 160 when Cs = 64. Therefore, the output of the channel number is an integer.

[0144] Similarly, in a predictive fusion (gather) net, in the conventional method, the output of Conv 10 / 3C'1'1 in the hyperscale decoder net would be 426.6667 when Cp=128 and 213.3333 when Cs=64. Therefore, the output of the number of channels is a non-integer. In the embodiment provided in FIG. 11, the output of Conv 7 / 2C'1'1 would be 448 when Cp=128 and 224 when Cs=64. Therefore, the output of the number of channels is an integer.

[0145] FIG. 11 is an optional design for the hyperscale decoder net and the predictor fusion (gather) net. FIG. 12 provides another selection for the hyperscale decoder net and the predictor fusion (gather) net. The hyperscale decoder net in FIG. 11 can also operate together with the hyperscale decoder net and / or the predictor fusion (gather) net in FIG. 12. Similarly, the predictor fusion (gather) net in FIG. 11 can also operate together with the hyperscale decoder net and / or the predictor fusion (gather) net. The present invention does not impose any limitations on the combination of these nets. The nets are also referred to as neural network layers or neural network subnetworks.

[0146] The order of execution in Figures 11 and 12 is from right to left.

[0147] In the example hyperscale decoder net in Figure 11, p / q = 2 and p' / q' = 5 / 2. In the example hyperscale decoder net in Figure 12, p / q = 5 / 2 and p' / q' = 2.

[0148] As an example shown in Figure 11, the tensor sizes for the number of channels (C) in the hyperscale decoder net may include 3 / 2C, 2C, 5 / 2C, and 3 / 2C, in that order. As an example shown in Figure 12, the tensor sizes for the number of channels (C) in the hyperscale decoder net may include 3 / 2C, 5 / 2C, 2C, and 3 / 2C, in that order.

[0149] In the example of the predictive fusion (gather) net in Figure 11, p / q = 7 / 2 and p' / q' = 3. In the example of the predictive fusion (gather) net in Figure 12, p / q = 9 / 2 and p' / q' = 7 / 2.

[0150] As shown in the example of Figure 11, the tensor sizes of the number of channels (C) in the predictive fusion (gather) net may include 7 / 2C, 3C, and 5 / 2C, in that order. As shown in the example of Figure 12, the tensor sizes of the number of channels (C) in the predictive fusion (gather) net may include 9 / 2C, 7 / 2C, and 5 / 2C, in that order.

[0151] In the present disclosure, when the number of channels in a subnetwork is variable, maintaining the number of channels as an integer at every layer can prevent data loss. When the number of channels in a subnetwork is variable, maintaining the number of channels as a multiple of the chunk size at every layer can result in efficient NPU / GPU resource utilization. Furthermore, it is not allowed for the third channel number to be greater than the second channel number and for the second channel number to be less than the first channel number. This can prevent data loss and further data replay.

[0152] Functional modules Variable Bitrate Module An encoder can output bitstreams at different bitrates. Thus, in some methods, the output of the encoding network is scaled (e.g., each channel is multiplied by a corresponding scaling factor, also called a target gain value), and the input of the decoding network is inversely scaled (e.g., each channel is multiplied by a corresponding reciprocal scaling factor, also called a target inverse gain value), as shown in FIG. 13. The scaling factors may be preset. Different quality levels or quantization parameters correspond to different target gain values. If the output of the encoding network is scaled to a smaller value, the bitstream size may be reduced. Otherwise, the bitstream size may be increased.

[0153] Color Format Conversion RGB and YUV are common color spaces, and conversion between RGB and YUV can be performed according to formulas specified in standards such as CCIR 601 and BT.709.

[0154] Separation structure for luma and chroma As shown in FIG. 14, some VAE-based codecs use the YUV color space as the encoder input and decoder output. The Y component represents luma, and the UV component represents chroma. The resolution of the UV component may be the same as or lower than that of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted to a feature map F_Y through a network, and an entropy coding module generates a Y component bitstream based on the feature map F_Y. The UV component is converted to a feature map F_UV through another network, and an entropy coding module generates a UV component bitstream based on the feature map F_UV. Under this structure, the Y component feature map and the UV component feature map can be independently quantized, allowing for flexible bit allocation between luma and chroma. For example, for color-sensitive images, the feature map of the UV component may be less quantized and the amount of bitstream bits for the UV component may be increased to improve the reconstruction quality of the UV component.

[0155] In some other methods, the encoder concatenates the Y component and the UV component, and then sends it to the UV component processing module (to convert the image information into a feature map). In addition, the decoder concatenates the reconstructed feature map of the Y component and the reconstructed feature map of the UV component, and then sends it to the UV component processing module 2 (to convert the feature map into image information). In this method, the correlation between the Y component and the UV component can be used to reduce the bitstream of the UV component.

[0156] Although operations are shown in the figures in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown, or sequential order, or that all of the described operations be performed, to achieve desirable results. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems may typically be integrated together in a single software product or packaged into multiple software products.

[0157] Certain embodiments of the subject matter have been described. Other embodiments are within the scope of the following claims. For example, the actions recited in the claims can be performed in a different order and still achieve desirable results. As one example, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In certain embodiments, multitasking and parallel processing may be advantageous.

[0158] A corresponding system in which the above-described encoder / decoder processing chain may be deployed is shown in FIG. 15. FIG. 15 is a schematic block diagram illustrating an example coding system, e.g., a video, image, audio, and / or other coding system (or short coding system), that may utilize the techniques of the present application. The video encoder 20 (or short encoder 20) and the video decoder 30 (or short decoder 30) of the video coding system 10 represent examples of devices that may be configured to perform techniques in accordance with various examples described herein. For example, the video encoding and decoding may be distributed and employ neural networks that may apply the above-described bitstream parsing and / or bitstream generation to communicate feature maps between distributed computational nodes (two or more).

[0159] As shown in FIG. 15, encoding system 10 includes a source device 12 configured to provide encoded picture data 21, for example, to a destination device 14 for decoding encoded picture data 13.

[0160] The source device 12 includes an encoder 20 and may additionally, i.e., optionally, include a picture source 16, a preprocessor (or preprocessing unit) 18, e.g., a picture preprocessor 18, and a communication interface or communication unit 22.

[0161] Picture source 16 may include or be any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any type of other device for obtaining and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). Picture source may also be any type of memory or storage device that stores any of the foregoing pictures.

[0162] Unlike the pre-processor 18 and the processing performed by the pre-processing unit 18, the picture or picture data 17 is also referred to as a raw picture or raw picture data 17.

[0163] The pre-processor 18 is configured to receive (raw) picture data 17 and perform pre-processing on the picture data 17 to obtain a pre-processed picture 19 or pre-processed picture data 19. The pre-processing performed by the pre-processor 18 may include, for example, cropping, color format conversion (e.g., from RGB to YCbCr), color correction, or noise removal. It may be understood that the pre-processing unit 18 may be an optional component. Note that the pre-processing may also employ a neural network (such as any of FIGS. 1-7) that uses presence indicator signaling.

[0164] Video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21 .

[0165] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or any further processed version thereof) via the communication channel 13 to another device, such as the destination device 14 or any other device, for storage or direct reconstruction.

[0166] The destination device 14 includes a decoder 30 (e.g., a video decoder 30) and may additionally, i.e., optionally, include a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device.

[0167] The communications interface 28 of the destination device 14 is configured to receive the encoding picture data 21 (or any further processed version thereof), for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoding picture data storage device, and to provide the encoding picture data 21 to the decoder 30.

[0168] The communication interface 22 and the communication interface 28 may be configured to transmit or receive the encoding picture data 21 or the encoding data 13 via a direct communication link between the source device 12 and the destination device 14, e.g., a direct wired or wireless connection, or via any type of network, e.g., a wired or wireless network, or any combination thereof, or any type of private and public network, or any combination thereof.

[0169] The communications interface 22 may be configured, for example, to package the encoded picture data 21 into a suitable format, e.g., packets, and / or process the encoded picture data using any type of transmission encoding or processing for transmission over a communications link or communications network.

[0170] The communications interface 28, which forms the counterpart of the communications interface 22, may be configured to receive the transmitted data and process the transmitted data using any kind of corresponding transmission decoding or processing and / or depackaging, for example to obtain the encoded picture data 21.

[0171] Both communication interface 22 and communication interface 28 may be configured as unidirectional communication interfaces, as indicated by the arrow pointing from source device 12 to destination device 14 for communication channel 13 in Figure 15, or as bidirectional communication interfaces, and may be configured to send and receive messages, for example, to set up a connection, acknowledge, and exchange any other information related to the communication link and / or data transmission, e.g., encoding picture data transmission. Decoder 30 is configured to receive encoding picture data 21 and provide decoded picture data 31 or decoded pictures 31.

[0172] Post-processor 32 of destination device 14 is configured to post-process decoded picture data 31 (also referred to as reconstructed picture data), e.g., decoded picture 31, to obtain post-processed picture data 33, e.g., post-processed picture 33. The post-processing performed by post-processing unit 32 may include, for example, color format conversion (e.g., from YCbCr to RGB), color correction, cropping, or resampling, or any other processing to prepare decoded picture data 31 for display by, for example, display device 34.

[0173] Display device 34 of destination device 14 is configured to receive post-processed picture data 33, for example, to display the picture to a user or viewer. Display device 34 may be or include any type of display for displaying the reconstructed picture, such as an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light emitting diode (OLED) display, a plasma display, a projector, a micro LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0174] 15 depicts source device 12 and destination device 14 as separate devices, an embodiment of the device may also include functionality of both, i.e., source device 12 or corresponding functionality, and destination device 14 or corresponding functionality. In such an embodiment, source device 12 or corresponding functionality and destination device 14 or corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or by any combination thereof.

[0175] As will be apparent to those skilled in the art based on this description, the functions of different units, or the presence and (exact) division of functions within source device 12 and / or destination device 14 as shown in FIG. 15, may vary depending on the actual device and application.

[0176] Encoder 20 (e.g., video encoder 20), or decoder 30 (e.g., video decoder 30), or both encoder 20 and decoder 30 may be implemented via processing circuitry, such as one or more microprocessors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. Encoder 20 may be implemented via processing circuitry 46 to implement various modules, including a neural network or portions thereof. Decoder 30 may be implemented via processing circuitry 46 to implement any encoding system or subsystem described herein. The processing circuitry may be configured to perform various operations, such as those described below. If the techniques are implemented partially in software, a device may store instructions for the software on a suitable, non-transitory computer-readable storage medium and may execute the instructions in hardware using one or more processors to perform the techniques of this disclosure. Either video encoder 20 and video decoder 30 may be integrated as part of a combined encoder / decoder (codec) in a single device, for example, as shown in FIG.

[0177] Source device 12 and destination device 14 may include any of a wide range of devices, including any type of handheld or stationary device, such as a notebook or laptop computer, a mobile phone, a smartphone, a tablet or tablet computer, a camera, a desktop computer, a set-top box, a television, a display device, a digital media player, a video game console, (such as a content service server or content distribution server), a video streaming device, a broadcast receiver device, a broadcast transmitter device, etc., and may use no operating system or any type of operating system. In some cases, source device 12 and destination device 14 may be equipped for wireless communication. Thus, source device 12 and destination device 14 may be wireless communication devices.

[0178] In some instances, the video coding system 10 shown in FIG. 15 is merely an example, and the techniques of the present application may be applied to video coding settings (e.g., video encoding or video decoding) that do not necessarily include data communication between an encoding device and a decoding device. In other examples, data may be retrieved from local memory, streamed over a network, etc. A video encoding device may encode data and store it in memory, and / or a video decoding device may retrieve data from memory and decode it. In some examples, encoding and decoding are performed by devices that do not communicate with each other but simply encode data to memory and / or retrieve data from memory and decode it.

[0179] 17 is a schematic diagram of a video coding device 8000 according to an embodiment of the present invention. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In one embodiment, the video coding device 8000 may be a decoder, such as the video decoder 30 of FIG. 15, or an encoder, such as the video encoder 20 of FIG. 15.

[0180] The video coding device 8000 includes an ingress port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing the data, a transmitter unit (Tx) 8040 and an egress port 8050 (or output port 8050) for transmitting the data, and a memory 8060 for storing the data. The video coding device 8000 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, and the egress port 8050 for the egress or ingress of optical or electrical signals.

[0181] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., multi-core processors), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the ingress port 8010, the receiver unit 8020, the transmitter unit 8040, the egress port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the disclosed embodiments described above. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. The inclusion of the neural network-based codec 8070 therefore provides substantial improvements to the functionality of the video coding device 8000 and results in the transformation of the video coding device 8000 into different states. Alternatively, the neural network-based codec 8070 is implemented as instructions stored in memory 8060 and executed by the processor 8030.

[0182] The memory 8060 may include one or more disks, tape drives, and solid state drives, and may be used as an overflow data storage device, storing programs when they are selected for execution, and storing instructions and data read during program execution. The memory 8060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content addressable memory (TCAM), and / or static random access memory (SRAM).

[0183] FIG. 18 is a schematic block diagram of an apparatus that may be used as either or both of source device 12 and destination device 14 from FIG. 15, according to an example embodiment.

[0184] The processor 9002 in the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or devices now existing or later developed that can manipulate or process information. While the disclosed implementations may be implemented using a single processor as shown, e.g., processor 9002, advantages in speed and efficiency may be achieved using two or more processors.

[0185] The memory 9004 in the apparatus 9000, in one implementation, may be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device may be used as the memory 9004. The memory 9004 may include code and data 9006, which is accessed by the processor 9002 using the bus 9012. The memory 9004 may further include an operating system 9008 and application programs 9010. The application programs 9010 include at least one program that enables the processor 9002 to perform the methods described herein. For example, the application programs 9010 may include applications 1-N, which further include a video coding application that performs the methods described herein.

[0186] The apparatus 9000 may also include one or more output devices, such as a display 9018. The display 9018, in one example, may be a touch-sensitive display that combines a display with touch-sensitive elements operable to sense touch input. The display 9018 may be coupled to the processor 9002 via the bus 9012.

[0187] Although shown here as a single bus, the bus 9012 of the device 9000 may be comprised of multiple buses. Additionally, the secondary storage device may be directly coupled to other components of the device 9000 or may be accessed over a network, and may comprise a single integrated unit such as a memory card, or multiple units such as multiple memory cards. The device 9000 may therefore be implemented in a wide variety of configurations.

[0188] FIG. 19 is a block diagram of a video encoding system 10000 according to an embodiment of the present disclosure.

[0189] A neural network 2000 according to an embodiment of the present disclosure is shown in Figure 20. The neural network 2000 includes a first neural network layer 2010 and a second neural network layer 2020. The first neural network 2010 receives as input a first number of channels C in and the second channel number C out where the first number of channels is different from the second number of channels, and C out =p*C in / q, where C in is a multiple of q, and C in , C out , p and q are (positive) integers. The second neural network layer 2020 has a second number of channels C out It is configured to receive as input. For example, C in The input channels may be or include color components, for example, RGB or YGB color components. out The output channels exist in the latent space. Condition C out =p*C in / q is also the C of the first neural network layer 2010 out C output channels in 2000. According to one embodiment, this condition may also be satisfied for the output channels of the second neural network layer 2020, which receives as input channels C in Input channel and C out Generally, the number of channels or output channels can be a multiple of 16 or 32.

[0190] Restricting the number of variable channels to an integer significantly improves the accuracy and speed of the calculations compared to the prior art.

[0191] Neural network 2000 may be configured with one of the example neural network architectures shown in FIGS. 1-10 or one of the devices shown in FIGS. 15-19.

[0192] A method 2100 of operating a neural network with a variable number of channels in a neural network layer is shown in Figure 21. For example, method 2100 is for operating neural network 2000 shown in Figure 20, which includes first neural network layer 2010 and second neural network layer 2020. Method 2100 receives as input by the first neural network layer a first number of channels C in and a step 2110 of obtaining (receiving) a second number of channels C out where the first number of channels is different from the second number of channels and C out =p*C in / q and C in is a multiple of q and C in , C out , p, and q are (positive) integers. The second neural network layer takes as input the second number of channels C out and a step 2130 of obtaining (receiving) C'. According to one embodiment, the method 2100 further includes the step of obtaining (receiving) C'=p'*C'. out / q', where C out is a multiple of q', and C', p', and q' are (positive) integers and are channels through the second neural network layer. A video encoding or decoding method may advantageously include a method 2100 shown in FIG. 21.< / ck>

Claims

1. A neural network, Number of first channels C in as input, and the second channel number C out a first neural network layer configured to output the first number of channels is different from the second number of channels, and C out =p*C in / q, C in is a multiple of q and C in , C out , p, and q are integers, a first neural network layer; The second channel number C out a second neural network layer configured to receive as input Including neural networks.

2. q is equal to 2, 2. The neural network of claim 1.

3. the neural network further includes a third neural network layer; the second neural network layer is further configured to output a third number of channels C′; the third neural network layer is configured to obtain the third number of channels C′; C'=p'*C out / q' and C out is a multiple of q', and C', p', and q' are integers; 2. The neural network of claim 1.

4. the third number of channels is less than the second number of channels and the second number of channels is less than the first number of channels, or the third number of channels is less than the second number of channels and the second number of channels is greater than the first number of channels; or the third number of channels is greater than the second number of channels, and the second number of channels is greater than the first number of channels; 4. The neural network of claim 3.

5. The neural network the third number of channels is not allowed to be greater than the second number of channels and the second number of channels is not allowed to be less than the first number of channels; 5. The neural network according to claim 3 or 4.

6. p / q=5 / 2 and p' / q'=2, or p / q=2 and p' / q'=5 / 2. A neural network according to any one of claims 3 to 5.

7. the first channel number C in and the second channel number C out is a multiple of the chunk size, A neural network according to any one of claims 1 to 6.

8. the chunk size is 16 or 32; 8. The neural network of claim 7.

9. the first neural network layer or the neural network subnet is a hyperscale decoder subnet; A neural network according to any one of claims 1 to 8.

10. The first neural network layer or the subnet of the neural network is a predictive fusion subnet. A neural network according to any one of claims 1 to 8.

11. the first neural network layer includes a data path for at least one of a first-order component and a second-order component; A neural network according to any one of claims 1 to 10.

12. the neural network includes at least one neural network subnet consisting of successive neural network layers; For the at least one neural network subnet, at least one of the following conditions is satisfied for successive neural network layers where the number of channels varies from one neural network layer to another: d) each successive neural network layer is configured to output, in processing order, only a number of channels that is less than or greater than the number of channels received from a previous neural network layer in the successive neural network layer; e) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; iii) each of the successive neural network layers of the first subset is configured to output, in processing order, only a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the first subset; and iv) each of the successive neural network layers of the second subset is configured to output, in processing order, only a number of channels that is less than the number of channels received from a previous one of the successive neural network layers of the second subset; f) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; v) each of the successive neural network layers of the first subset is configured to output, in processing order, only a number of channels that is less than the number of channels received from a previous one of the successive neural network layers of the first subset; vi) none of the successive neural network layers of the second subset is configured to output, in processing order, a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the second subset; and vii) the first neural network layer in the second subset of successive neural network layers in processing order is configured to output only a smaller number of channels than the number of channels received from the last neural network layer in the first subset of successive neural network layers in processing order; 3. A neural network according to claim 1 or 2.

13. each successive neural network layer is configured to output a number of channels that is a multiple of 16 or 32; 13. The neural network of claim 12.

14. the at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet; 14. A neural network according to claim 12 or 13.

15. 1. A method of operating a neural network having a variable number of channels associated with a neural network layer, comprising: The first neural network layer generates the first number of channels C in as input; The first neural network layer generates a second number of channels C out and outputting the first number of channels is different from the second number of channels, and C out =p*C in / q and C in is a multiple of q and C in , C out ,p,and q are integers, and, The second neural network layer generates the second number of channels C out as input; A method comprising:

16. q is equal to 2, 16. The method of claim 15.

17. The method further comprises: outputting a third number of channels C' by the second neural network layer; obtaining the third number of channels C′ by a third neural network layer; C'=p'*C out / q' and C out is a multiple of q', and C', p', and q' are integers; 17. The method of claim 15 or 16, comprising:

18. the third number of channels is less than the second number of channels and the second number of channels is less than the first number of channels, or the third number of channels is less than the second number of channels and the second number of channels is greater than the first number of channels; or the third number of channels is greater than the second number of channels, and the second number of channels is greater than the first number of channels; 18. The method of claim 17.

19. The neural network the third number of channels is not allowed to be greater than the second number of channels, and the second number of channels is not allowed to be less than the first number of channels; 19. The method of claim 17 or 18.

20. p / q=5 / 2 and p' / q'=2, or p / q=2 and p' / q'=5 / 2.

20. The method of any one of claims 17 to 19.

21. the first channel number C in and the second channel number C out is a multiple of the chunk size, 21. The method of any one of claims 15 to 20.

22. the chunk size is 16 or 32; 22. The method of claim 21.

23. the first neural network layer or the neural network subnet is a hyperscale decoder subnet; 23. The method of any one of claims 15 to 22.

24. The first neural network layer or the subnet of the neural network is a predictive fusion subnet.

23. The method of any one of claims 15 to 22.

25. the first neural network layer includes a data path for a first-order component and / or a second-order component; 25. The method of any one of claims 15 to 24.

26. the neural network includes at least one neural network subnet consisting of successive neural network layers; For the at least one neural network subnet, at least one of the following conditions is satisfied for successive neural network layers where the number of channels varies from one neural network layer to another: c) each successive neural network layer outputs, in processing order, only a number of channels that is less than or greater than the number of channels it receives from a previous neural network layer in the successive neural network layer; d) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; i) each of the successive neural network layers of the first subset outputs, in processing order, only a number of channels that is greater than the number of channels received from a previous one of the successive neural network layers of the first subset; and ii) each of the successive neural network layers of the second subset outputs, in processing order, only a number of channels that is less than the number of channels it receives from a previous one of the successive neural network layers of the second subset; c) the successive neural network layers consist of a first subset of successive neural network layers followed in processing order by a second subset of successive neural network layers; j) each of the successive neural network layers of the first subset outputs, in processing order, only a number of channels that is less than the number of channels it receives from a previous one of the successive neural network layers of the first subset; ii) none of the successive neural network layers of the second subset outputs, in processing order, a number of channels greater than the number of channels received from a previous one of the successive neural network layers of the second subset; and viii) the first neural network layer in the second subset of successive neural network layers in processing order outputs only a smaller number of channels than the number of channels received from the last neural network layer in the first subset of successive neural network layers in processing order; 17. The method of claim 15 or 16.

27. each successive neural network layer is configured to output a number of channels that is a multiple of 16 or 32; 27. The method of claim 26.

28. the at least one neural network subnet is one of a hyperscale decoder subnet and a predictive fusion subnet; 28. The method of claim 26 or 27.

29. 1. A method of encoding data, comprising: A method for operating a neural network according to any one of claims 15 to 28, comprising the steps of: method.

30. 1. A method of decoding encoded data, comprising: A method for operating a neural network according to any one of claims 15 to 28, comprising the steps of: method.

31. 1. A computer program product comprising program code stored on a non-transitory medium, The program, when executed on one or more processors, performs the method of any one of claims 15 to 28. Computer program products.

32. 1. An apparatus for encoding data, comprising: comprising processing circuitry configured to perform the steps of the method of any one of claims 15 to 28, Device.

33. 1. An apparatus for decoding data, comprising: comprising processing circuitry configured to perform the steps of the method of any one of claims 15 to 28, Device.

34. 1. An apparatus for decoding at least a portion of an encoded image, comprising: a processing circuit configured to provide an entropy model; Carrying out the steps of the method according to any one of claims 15 to 28, processing the bitstream with a neural network based on the provided entropy model to obtain latent tensors representing components of the image; and processing the latent tensors to obtain tensors representing components of the image; 1. An apparatus comprising:

35. 1. An apparatus for encoding at least a portion of an image, comprising:

15. A neural network comprising the neural network of any one of claims 1 to 14. Device.

36. 1. An apparatus for decoding at least a portion of an image, comprising:

15. A neural network comprising the neural network of any one of claims 1 to 14. Device.

Images