Method and apparatus for video frame synthesis

Abstract

A method for synthesizing video frames using a convolutional neural network is provided, comprising an encoder including at least one first mask unit and a decoder including at least one second mask unit. The method includes the steps of: generating first data corresponding only to a first sub-part of a first video frame captured at a first time point and second data corresponding only to a second sub-part of a second video frame captured at a second time point using at least one first mask unit; generating a first feature pyramid based on the first data and a second feature pyramid based on the second data using the encoder; and generating a third video frame synthesized for a third time point between the first and second time points using at least one second mask unit, based on the generated feature pyramids, using the decoder.

Description

[Technical Field]

[0001] Embodiments of this disclosure generally relate to the field of encoding and decoding data based on neural network architectures. In particular, some embodiments relate to methods and apparatus for synthesizing video frames based on convolutional neural networks. [Background technology]

[0002] Hybrid image and video codecs have been used for decades to compress image and video data. In such codecs, signals are typically encoded in blocks by predicting blocks and then further coding only the difference between the original blocks and their predictions. In particular, such coding may include transformation, quantization, and bitstream generation, and usually includes some form of entropy coding. Typically, the three components of a hybrid coding method—transformation, quantization, and entropy coding—are optimized separately. Modern video compression standards such as High-Efficiency Video Coding (HEVC), Versatile Video Coding (VVC), and Essential Video Coding (EVC) also use transformed representations to code the residual signal after prediction.

[0003] Recently, neural network architectures have been applied to image and / or video coding. Generally, these neural network (NN)-based methods can be applied to image and video coding in a variety of different ways. For example, several end-to-end optimized image or video coding frameworks have been discussed. Furthermore, deep learning is being used to determine or optimize certain parts of the end-to-end coding framework, such as the selection or compression of prediction parameters. In addition, several neural network-based methods are also being discussed for use in hybrid image and video coding frameworks, for example, as trained deep learning models for intra or inter-prediction in image or video coding.

[0004] The end-to-end optimized image or video coding applications discussed above share the commonality of generating several feature map data that should be transmitted between the encoder and the decoder.

[0005] A neural network is a machine learning model that uses one or more layers of nonlinear units, on which it can predict an output for an incoming input. Some neural networks include one or more hidden layers in addition to an output layer. A corresponding feature map may be provided as the output of each hidden layer. Such a corresponding feature map of each hidden layer may be used as input to a subsequent layer in the network, i.e., a subsequent hidden layer or output layer. Each layer of the network generates an output from an incoming input according to the current values ​​of its respective set of parameters. In a neural network that is split between devices, for example, between an encoder and a decoder, between a device and the cloud, or between different devices, the feature map at the output of the split location (e.g., the first device) is compressed and transmitted to the remaining layers of the neural network (e.g., to the second device).

[0006] Further improvements in encoding and decoding using trained network architectures may be desirable. [Overview of the initiative]

[0007] This disclosure provides, in particular, a method and apparatus for improving the synthesis of video frames for video compression or frame rate upconversion.

[0008] The above-mentioned and other objectives are achieved by the subject matter of the independent claim. Further means of implementation are evident from the dependent claims, specification and drawings.

[0009] Specific embodiments are outlined in the attached independent claims, and other embodiments are outlined in the dependent claims.

[0010] According to a first aspect, the present disclosure relates to a method for synthesizing video frames using a convolutional neural network. The convolutional neural network includes an encoder and a decoder, the encoder includes at least one first mask unit, and the decoder includes at least one second mask unit. The method is A step of generating, using at least one first mask unit, first data corresponding only to one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding only to one or more second sub-parts of a second video frame captured at a second time point different from the first time point, The steps include generating a first feature pyramid based on the first data and a second feature pyramid based on the second data using an encoder, The decoder generates a third video frame synthesized for a third time point between the first and second time points, using at least one second mask unit based on the first and second feature pyramids generated. Includes.

[0011] Each feature pyramid contains features at different levels, each with a different number of feature channels, and individual mask units may be used for features at each level. Note that the mask units may be applied to feature tensors (in which case the first and second data contain feature tensors), or they may be applied to the first and second video frames on a pixel-by-pixel basis.

[0012] According to the method of the first embodiment, intermediate video frame synthesis using synthesized video frames generated by a neural network is combined with dynamic / conditional convolution. The use of a mask unit enables dynamic / conditional convolution in which the convolutional filters of the convolutional neural network are applied only to sub-parts of the first and second video frames, rather than to all regions (pixels) of the first and second video frames. This significantly speeds up the overall processing and reduces the memory space requirements.

[0013] For example, in the context of interpredictive coding, regions of the video frame to be processed that do not show significant motion can be ignored based on the application of a mask unit. Generally, based on the content, the mask unit may provide only the sub-parts of the video frame of interest for the convolution process, for example, excluding static background portions.

[0014] The above embodiments and other embodiments may each optionally include one or more of the following features, either individually or in combination.

[0015] In a possible implementation, at least one first and second mask unit is trained to acquire a target sparsity for first data relating to a first video frame and second data relating to a second video frame. Target sparsity defines the reduction of information from the entire video frame to the data to be processed by the neural network's convolutional filter. For example, at least one first and second mask unit is trained based on the content of the training image, so that content-based conditional convolution can be achieved and such sub-parts of the video frame to be processed that are not relevant to a particular practical use (e.g., static objects and background) can be ignored.

[0016] According to the implementation, the first embodiment and the method of the implementation further include the step of determining a target sparsity based on quantization parameters used to compress first and second video frames. The same or different quantization parameters may be used for the first and second video frames. In the context of video compression, the quantization parameters represent a measure of the complexity of the partitions of the video frame, with larger quantization parameters being used to code partitions with relatively low complexity and smaller quantization parameters being used to code partitions with relatively high complexity. Thus, quantization parameters can be appropriately used to control the reduction of data to be processed by the convolutional filters of a neural network.

[0017] According to the implementation, the target sparsity depends on at least one of the following: the number of objects present in the first and second video frames, the degree of object movement between the first and second video frames, the number of image partitions into which the first and second video frames are partitioned, and the compression levels of the first and second video frames. Such quantities can also be appropriately used to control the reduction of data to be processed by the convolutional filters of the neural network.

[0018] The first embodiment and any implementation thereof can be implemented (and carried out) in a video encoder device and a video decoder device. Here, a) an encoder and video encoder device and b) a decoder and video decoder device must be distinguished from each other. The video encoder device transmits encoded data to the video decoder device, and the video decoder device decodes the encoded data. Each of the video encoder device and the video decoder device includes a copy of a convolutional neural network, which includes an encoder for feature extraction and a decoder for frame synthesis.

[0019] According to the implementation, the method further includes the steps of: selecting a target sparsity from a plurality of predefined target sparsity levels (ratios) using a video encoder device that includes a copy of a convolutional neural network for each of the predefined target sparsity levels; signaling the selected target sparsity to a video decoder device that includes a copy of a convolutional neural network for each of the predefined target sparsity levels; and processing first and second video frames based on the signaled target sparsity using the video decoder device. According to this implementation, a pre-trained set of synthetic neural networks is used by the video encoder device and the video decoder device, respectively, and the video decoder device is explicitly notified by the video encoder device of the sparsity level to be used to decode the encoded data transmitted by the video encoder device. Signaling the target sparsity enables the application of the method in a wide variety of video compression schemes.

[0020] According to the alternative implementation method, signaling of the target sparsity is not required. Rather, the method further includes the step of determining the target sparsity by both a video encoder device including a copy of the convolutional neural network for each of the predefined target sparsity levels and a video decoder device including a copy of the convolutional neural network for each of the predefined target sparsity levels, based on at least one of a) quantization parameters used for compression of the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned.

[0021] For example, the target sparsity level SR estimated is determined by the video encoder device and the video decoder device based on the formula

Number

[0022] In the above implementation method, the multiple predefined target sparsity levels may include multiple sparsity levels between 10% and 90% of the sparsity of the first data for the first video frame and the second data for the second video frame, and adjacent sparsity levels are spaced apart from each other by at least 10% of the sparsity of the first data for the first video frame and the second data for the second video frame. Thus, a wide range of sparsity levels can be covered, and intervals considerably smaller than 10% do not increase the accuracy of frame synthesis.

[0023] According to the third alternative implementation, the target sparsity is neither signaled nor estimated based on the quantization parameters and / or the number of partitioned blocks, and there is no need to provide multiple pre-trained copies of the neural network for different sparsity levels. Rather, the method includes the steps of receiving at least one of a) quantization parameters used to compress the video of the first and second frames and b) the number of blocks into which the first and second video frames are partitioned by at least one first mask unit, and determining the target sparsity based on at least one of a) quantization parameters used to compress the video of the first and second frames and b) the number of blocks into which the first and second video frames are partitioned by at least one first mask unit.

[0024] Therefore, according to this implementation, the neural network is conditioned by further input channels (control parameters) in the form of quantization parameters and / or the number of partition blocks. This type of implementation allows the neural network to operate without the need to update its weights on the fly and enables the adaptation of sparsification to the actual content of the reference frame.

[0025] According to the implementation, the step of generating a synthesized third video frame includes a step of jointly refining the intermediate flow fields on both sides, together with a first intermediate feature pyramid reconstructed based on a first feature pyramid and a second intermediate feature pyramid reconstructed based on a second feature pyramid. Thus, video frame synthesis may appropriately be based on flow-based video frame interpolation, which can yield effective and accurate synthesis results.

[0026] The first embodiment and any implementation thereof may be appropriately used, for example, in the context of video compression and frame rate upconversion (on both the video encoder device side and the video decoder device side).

[0027] According to a second embodiment, a video compression method is provided which includes a step of the method according to the first embodiment and any implementation thereof, wherein the synthesized third video frame is stored as an S-frame in a decoding picture buffer or added to a list of reference pictures used for intra-prediction or inter-prediction.

[0028] A third embodiment provides a frame rate upconversion method comprising the steps of the first embodiment and any implementation thereof, wherein the synthesized third video frame is used to increase the frame rate of a transmitted video comprising the first and second video frames.

[0029] According to a fourth aspect, a computer program stored on a non-temporary medium is provided, which, when executed on one or more processors, includes code that performs the steps of the method according to the first aspect and any implementation thereof, or the method according to the second aspect, or the method according to the third aspect.

[0030] According to the fifth aspect, a processing device for video frame synthesis (e.g., a video encoder device or a video decoder device) is provided, which includes one or more processors and a non-temporary computer-readable storage medium coupled to the one or more processors and storing a program for execution by the one or more processors, wherein the device is configured to perform the method according to the first aspect and any implementation thereof, or the method according to the second aspect, or the method according to the third aspect, when the program is executed by the one or more processors.

[0031] According to a sixth aspect, a processing device for video frame synthesis (e.g., a video encoder device or a video decoder device) including a convolutional neural network is provided. The convolutional neural network includes an encoder and a decoder, the encoder including at least one first mask unit, and the decoder including at least one second mask unit. The at least one first mask unit is configured to generate first data corresponding only to one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding only to one or more second sub-parts of a second video frame captured at a second time point different from the first time point. The encoder is configured to generate a first feature pyramid based on the first data and a second feature pyramid based on the second data, and the decoder is configured to generate a third video frame synthesized for a third time point between the first and second time points using at least one second mask unit based on the generated first and second feature pyramids.

[0032] The processing apparatus according to the sixth embodiment and any implementation thereof provides the same or similar advantages as those described above with reference to the method according to the first embodiment and any implementation thereof.

[0033] According to the implementation of the processing apparatus of the first embodiment, at least one first and second mask unit is trained to acquire the target sparsity of first data relating to a first video frame and second data relating to a second video frame.

[0034] For example, at least one first and second mask unit is trained based on the content of the training image. The target sparsity may be determined based on the quantization parameters used to compress the first and second video frames. The target sparsity may depend on at least one of the following: the number of objects present in the first and second video frames, the degree of object movement between the first and second video frames, the number of image partitions into which the first and second video frames are partitioned, and the compression levels of the first and second video frames.

[0035] According to another implementation, the decoder of the processing unit is configured to generate a synthesized third video frame by jointly refining the intermediate flow fields on both sides, along with a first intermediate feature pyramid reconstructed based on a first feature pyramid and a second feature intermediate pyramid reconstructed based on a second feature pyramid.

[0036] According to the seventh aspect, a video encoder device is provided which includes a copy of a processing device according to the sixth aspect or any implementation thereof, each copy including a convolutional neural network for one of a predefined target sparsity level, the video encoder device being configured to select a target sparsity from a plurality of predefined target sparsity levels and to signal the selected target sparsity to a video decoder device.

[0037] According to the eighth aspect, a video decoder device is provided which includes a copy of a processing device according to the sixth aspect or any implementation thereof, each copy including a convolutional neural network for one of a predefined target sparsity level, and the video decoder device is configured to process first and second video frames based on the target sparsity received from an encoder device.

[0038] According to the ninth aspect, a video encoder device is provided which includes a copy of a processing device according to the sixth aspect or any implementation thereof, each copy including a convolutional neural network for one of a predefined target sparsity level, the video encoder device is configured to determine the target sparsity based on at least one of a) quantization parameters used to compress the video of first and second frames, and b) the number of blocks into which the first and second video frames are partitioned. According to the implementation, the video encoder device is provided which of the following equations

number

[0039] According to the tenth embodiment, a video decoder device is provided which includes a copy of a processing device according to the sixth embodiment or any of its implementations, each copy including a convolutional neural network for one of a predefined target sparsity level, the video encoder device being configured to determine the target sparsity based on at least one of a) quantization parameters used to compress the video of first and second frames, and b) the number of blocks into which the first and second video frames are partitioned. According to the implementation, the video decoder device is configured with the formula

number

[0040] In other implementations, at least one first mask unit of the processing according to the sixth embodiment or any of the implementations thereof is configured to receive at least one of i) a) quantization parameters used to compress the video of the first and second frames and b) the number of blocks into which the first and second video frames are partitioned, and to determine the target sparsity based on at least one of the received i) a) quantization parameters used to compress the video of the first and second frames and b) the number of blocks into which the first and second video frames are partitioned.

[0041] According to the eleventh aspect, a video compression device is provided which includes a processing device according to the sixth aspect or any implementation thereof, a video encoder device according to the seventh or ninth aspect or any implementation thereof, or a video decoder device according to the eighth or tenth aspect or any implementation thereof, and is configured to store the synthesized third video frame as an S-frame in a decoded picture buffer, or to add the synthesized third video frame to a list of reference pictures used for intra-prediction or inter-prediction.

[0042] According to the twelfth aspect, a video frame rate upconversion device is provided which includes a processing device according to the sixth aspect or any implementation thereof, or a video decoder device according to the eighth or tenth aspect or any implementation thereof, and is configured to increase the frame rate of transmitted video including the first and second video frames using a synthesized third video frame.

[0043] According to the 13th aspect, the present disclosure relates to a video stream decoding device including a processor and memory. The memory stores instructions causing the processor to perform a method according to the first aspect or any implementation thereof.

[0044] According to the fourteenth aspect, the disclosure relates to a video stream encoding apparatus including a processor and memory. The memory stores instructions causing the processor to perform a method according to the first aspect or any implementation thereof.

[0045] According to the 15th aspect, a computer-readable storage medium is proposed that stores instructions, when executed, cause one or more processors to encode video data. The instructions cause one or more processors to execute a method according to the first or second aspect, or any possible embodiment of the first aspect, or any implementation thereof.

[0046] Details of one or more embodiments are described in the accompanying drawings and the following description. Other features, purposes, and advantages will become apparent from the description, drawings, and claims. [Brief explanation of the drawing]

[0047] The embodiments of this disclosure will be described in more detail below with reference to the attached drawings. [Figure 1] This is a schematic diagram showing the channels processed by the layers of a neural network. [Figure 2] This is a schematic diagram illustrating an autoencoder type of neural network. [Figure 3A] This is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperprior model. [Figure 3B] This is a schematic diagram showing a typical network architecture on the encoder side, including a hyperplier model. [Figure 3C] This is a schematic diagram showing a typical network architecture on the decoder side, including a hyperplier model. [Figure 4] This is a schematic diagram showing an exemplary network architecture for the encoder and decoder sides, including a hyperplier model. [Figure 5]This block diagram shows the structure of a cloud-based solution for machine-based tasks such as machine vision tasks. [Figure 6A] A block diagram showing a neural network-based end-to-end video compression framework. [Figure 6B] This block diagram shows some illustrative details of the application of neural networks for motion field compression. [Figure 6C] This block diagram shows some illustrative details of the application of neural networks for motion compensation. [Figure 7] This document describes a convolutional neural network architecture for intermediate video frame synthesis using the aforementioned technology. [Figure 8] This document presents a convolutional neural network architecture that enables dynamic convolution using the aforementioned technology. [Figure 9] An embodiment is shown which includes a convolutional neural network architecture for intermediate video frame synthesis and a processing unit that enables dynamic convolution. [Figure 10] Figure 9 shows an exemplary method for achieving dynamic convolution in the processing unit. [Figure 11A] The present invention includes a convolutional neural network architecture for intermediate video frame synthesis based on a hybrid video codec, and describes a processing unit that enables dynamic convolution. [Figure 11B] Another embodiment includes a convolutional neural network architecture for intermediate video frame synthesis based on a hybrid video codec, and a processing unit that enables dynamic convolution is also shown. [Figure 11C] Another embodiment includes a convolutional neural network architecture for intermediate video frame synthesis based on a hybrid video codec, and a processing unit that enables dynamic convolution is also shown. [Figure 12]This is a flowchart illustrating a method for synthesizing video frames using a convolutional neural network including an encoder and decoder, according to an embodiment. [Figure 13] This block diagram shows an example of a processing apparatus according to an embodiment. [Figure 14] This block diagram shows an example of a video coding system configured to implement embodiments of the present disclosure. [Figure 15] Block diagram shows another example of a video coding system configured to implement embodiments of the present disclosure. [Figure 16] This is a block of examples of encoding or decoding devices. [Figure 17] This is a block of other examples of encoding or decoding devices. [Figure 18] This is a block of other examples of encoding or decoding devices.

[0048] Similar reference numerals and names in different drawings may refer to the same elements. [Modes for carrying out the invention]

[0049] The following description refers to the accompanying drawings, which form part of the Disclosure and illustrate specific aspects of the embodiments of the Disclosure or specific aspects in which the embodiments of the Disclosure may be used. It is understood that the embodiments of the Disclosure may be used in other aspects and may include structural or logical modifications not shown in the drawings. Therefore, the following detailed description should not be constrained to mean limitingly, and the scope of the Disclosure is defined by the accompanying claims.

[0050] For example, disclosure relating to a method described may also apply to a corresponding device or system configured to perform the method, and vice versa. For example, if one or more specific method steps are described, the corresponding device may include one or more units, e.g., functional units, to perform the described method steps, even if one or more such units are not explicitly described or shown in the drawings (e.g., one unit performs one or more steps, or multiple units each perform one or more of the steps). On the other hand, for example, if a particular device is described based on one or more units, e.g., functional units, the corresponding method may include one step for performing the function of one or more units, even if one or more such steps are not explicitly described or shown in the drawings (e.g., one step performs the function of one or more units, or multiple steps each perform one or more of the functions of the units). Furthermore, it is understood that the various exemplary embodiments and / or features described herein may be combined with each other unless otherwise specified.

[0051] The following provides an overview of some of the technical terms and frameworks used in embodiments of this disclosure.

[0052] Artificial neural networks Artificial neural networks (ANNs), or connectionist systems, are computing systems vaguely inspired by the biological neural networks that make up animal brains. Such systems typically "learn" to perform tasks by considering examples, rather than being programmed with task-specific rules. For example, in image recognition, a system might learn to identify images containing cats by analyzing exemplary images manually labeled as "cat" or "no cat," and then using the results to identify cats in other images. This is done without prior knowledge of cats, such as having fur, a tail, whiskers, and a cat-like face. Instead, it automatically generates discriminative characteristics from the examples it processes.

[0053] ANNs are based on a collection of connected units or nodes called artificial neurons, which roughly model neurons in the biological brain. Each connection can transmit signals to other neurons, just like synapses in the biological brain. The receiving artificial neuron can then process the signals and signal them to the neurons it is connected to.

[0054] In an ANN implementation, the "signals" in a connection are real numbers, and the output of each neuron is calculated by some nonlinear function of the sum of its inputs. Connections are called edges. Neurons and edges typically have weights that adjust as learning progresses. The weights increase or decrease the strength of the signal in the connection. Neurons may have thresholds such that a signal is transmitted only when the aggregated signal exceeds a threshold. Typically, neurons are aggregated into layers. Different layers may perform different transformations on their inputs. The signal travels from the first layer (input layer) to the last layer (output layer), possibly traversing multiple layers.

[0055] The original goal of ANN methods was to solve problems in the same way the human brain does. Over time, attention has shifted to performing specific tasks, leading to deviations from biology. ANNs are used in a wide range of tasks, including computer vision, speech recognition, machine translation, social network filtering, playing board and video games, and medical diagnosis, and even in activities traditionally thought to be reserved for humans, such as painting.

[0056] The name "convolutional neural network" (CNN) indicates that the network uses a mathematical operation called convolution. Convolution is a special type of linear operation. A convolutional network is a neural network that uses convolution instead of general matrix multiplication in at least one of its layers.

[0057] Figure 1 schematically illustrates the general concept of processing by a neural network such as a CNN. A convolutional neural network consists of input and output layers and multiple hidden layers. The input layer is the layer to which the input (e.g., a portion of the input image shown in Figure 1) is provided for processing. The hidden layers of a CNN typically consist of a series of convolutional layers that convolve by multiplication or other dot products. The result of the layers is one or more feature maps (indicated by empty solid rectangles), sometimes also called channels. Resampling (such as subsampling) may be present, involving some or all of the layers. As a result, the feature maps may become smaller, as shown in Figure 1. It should be noted that convolutions with stride may also reduce (resample) the size of the input feature maps. The activation function in a CNN is usually a ReLU (Rectified Linear Unit) layer or Leaky ReLU, followed by further convolutions such as pooling layers, fully connected layers, and normalization layers, whose inputs and outputs are masked by the activation function and the final convolution, and are therefore called hidden layers. The layer is colloquially called a convolution, but this is merely a convention. Mathematically, it is technically a sliding dot product or cross-correlation. This is important for indices in a matrix in that it affects how weights are determined at a particular index point.

[0058] As shown in Figure 1, when programming a CNN to process images, the input is a tensor with shape (number of images) × (image width) × (image height) × (image depth). It should be recognized that the image depth can be composed of the channels of the image. After passing through the convolutional layer, the image is abstracted into a feature map with shape (number of images) × (feature map width) × (feature map height) × (feature map channels). The convolutional layer in the neural network should have the following attributes: The convolutional kernel is defined by its width and height (hyperparameters). The number of input and output channels (hyperparameters). The depth of the convolutional filter (input channels) must be equal to the number of channels (depth) of the input feature map.

[0059] Traditionally, multilayer perceptron (MLP) models have been used for image recognition. However, due to the full connectivity between nodes, these models suffer from high dimensionality and do not scale well with higher resolution images. A 1000x1000 pixel image with RGB color channels has 3 million weights, which is too high to process efficiently and efficiently at a scale with full connectivity. Furthermore, such network architectures do not take into account the spatial structure of the data, treating distant input pixels in the same way as pixels that are close to each other. This ignores locality of reference in image data, both computationally and semantically. Therefore, for purposes like image recognition, which are governed by spatially local input patterns, the full connectivity of neurons is redundant.

[0060] Convolutional neural networks (CNNs) are a biologically inspired variation of multilayer perceptrons specifically designed to emulate the behavior of the visual cortex. These models mitigate the challenges posed by MLP architectures by leveraging the strong spatially local correlations present in natural images. Convolutional layers are the core building blocks of a CNN. The layer parameters consist of a set of learnable filters (kernels, as described above), which have small receptive fields but extend across the full depth of the input volume. During the forward path, each filter is convolved across the width and height of the input volume, calculating the dot product between the filter's entry and the input to generate a two-dimensional activation map of that filter. As a result, the network learns which filters activate when it detects some specific type of feature at some spatial location in the input.

[0061] Stacking the activation maps of all filters along the depth dimension forms the complete output volume of the convolutional layer. Therefore, every entry in the output volume can also be interpreted as the output of a neuron that looks at a small region in the input and shares parameters with the neuron in the same activation map. A feature map, or activation map, is the output activation for a given filter. Feature map and activation have the same meaning. In some literature, it is called an activation map because it is a mapping corresponding to the activations of different parts of an image, and also called a feature map because it is a mapping of where certain types of features are found in the image. High activation means that a particular feature has been found.

[0062] Another important concept in CNNs is pooling, a form of nonlinear downsampling. Several nonlinear functions exist to implement pooling, with max pooling being the most common. This partitions the input image into a set of non-overlapping rectangles and outputs the maximum value for each such sub-region.

[0063] Intuitively, the precise location of a feature is less important than its approximate location relative to other features. This is the idea behind the use of pooling in convolutional neural networks. Pooling layers function to progressively reduce the spatial size of representations, thereby reducing the number of parameters, memory footprint, and computational complexity within the network, and thus also to control overfitting. In CNN architectures, it is common to periodically insert pooling layers between consecutive convolutional layers. Pooling operations provide another form of translational invariance.

[0064] A pooling layer operates independently for all depth slices of the input, spatially resizing them. The most common form is a pooling layer with a 2x2 filter applied with a stride of 2 for all depth slices in the input, along both width and height, discarding 75% of activations. In this case, all maximum operations exceed 4. The depth dimension remains unchanged. In addition to max pooling, pooling units can use other functions such as average pooling or l2-norm pooling. Average pooling has historically been frequently used, but recently it has lost support compared to max pooling, which actually often performs better. There is a recent trend toward using smaller filters or discarding pooling layers altogether due to the aggressive reduction in size of representation. "Region of Interest" pooling (also known as ROI pooling) is a variation of max pooling where the output size is fixed and the input rectangle is parameterized. Pooling is a key component of convolutional neural networks for object detection based on the fast R-CNN architecture.

[0065] The above ReLU stands for Normalized Linear Unit and applies a non-saturated activation function. This effectively removes negative values ​​from the activation map by setting negative values ​​to zero. This increases the nonlinear properties of the decision function and the overall network without affecting the receptive field of the convolutional layer. Other functions, such as the saturated hyperbolic tangent and sigmoid functions, are also used to increase nonlinearity. ReLU is often preferred over other functions because it trains neural networks several times faster with little penalty to generalized accuracy.

[0066] Leaky Rectified Linear Unit (Leaky ReLU) is a type of activation function based on ReLU, but for negative values, it has a small slope instead of a flat slope. The gradient coefficient is determined before training, i.e., it is not learned during training. This type of activation function is common in tasks with sparse gradients, such as training generative adversarial networks. Leaky ReLU applies an element-wise function.

number

[0067] Of these, the parameters are as follows: negative_slope - Controls the angle of the negative slope. The default is 1e-2. The `inplace` property allows you to perform calculations at any location. The default is `false`.

[0068] After several convolutional and max-pooling layers, high-level inference in a neural network is performed via fully connected layers. Neurons in fully connected layers have connections to all activations in the previous layer, as seen in typical (non-convolutional) artificial neural networks. Therefore, these activations can be computed as affine transformations, followed by matrix multiplication and bias offsets (vector addition of learned or fixed bias terms).

[0069] The "loss layer" (which includes the calculation of the loss function) specifies how training penalizes deviations between predicted (output) labels and true labels, and is typically the final layer of a neural network. Different loss functions may be used that are suitable for different tasks. Softmax loss is used to predict a single class out of K mutually exclusive classes. Sigmoid cross-entropy loss is used to predict K independent probability values ​​in [0,1]. Euclidean loss is used to regress to real-valued labels.

[0070] In short, Figure 1 shows the data flow in a typical convolutional neural network. First, the input image passes through a convolutional layer and is abstracted into a feature map containing several channels corresponding to the number of filters in the set of learnable filters in that layer. The feature map is then subsampled, for example, using a pooling layer to reduce the dimensionality of each channel in the feature map. Next, the data enters another convolutional layer, which may have a different number of output channels. As mentioned above, the number of input and output channels are hyperparameters of the layer. To establish network connectivity, these parameters need to be synchronized between two connected layers so that the number of input channels for the current layer is equal to the number of output channels for the previous layer. For the first layer processing input data, e.g., an image, the number of input channels is typically equal to the number of channels in the data representation, e.g., 3 channels for an RGB or YUV representation of an image or video, or 1 channel for a grayscale image or video representation. The channels obtained by one or more convolutional layers (and possibly resampling layers) may be passed to an output layer. Such an output layer may be convolutional or resampling in some implementations. In an exemplary and non-restrictive implementation, the output layer is a fully connected layer.

[0071] Autoencoders and unsupervised learning An autoencoder is a type of artificial neural network used to learn efficient data coding in an unsupervised manner. A schematic diagram of it is shown in Figure 2. The autoencoder includes an encoder side 210 with input x which is fed into the input layer of the encoder subnetwork 220, and a decoder side 250 with output x' which is output from the decoder subnetwork 260. The purpose of the autoencoder is to learn a representation (encoded) 230 for a set of data x, typically for dimensionality reduction, by training the networks 220, 260 to ignore signal "noise". Along with the reduction (encoder) side subnetwork 220, the reconstruction (decoder) side subnetwork 260 is learned, and the autoencoder attempts to generate a representation x' from the reduced encoding 230 that is as close as possible to its original input x, and therefore to its name. In its simplest form, given one hidden layer, the encoder stage of the autoencoder takes input x and maps it to h as follows: h = σ(Wx + b)

[0072] This image h is typically called code 230, latent variable, or latent representation, where σ is an element-wise activation function such as a sigmoid function or normalized linear unit. W is the weight matrix and b is the bias vector. The weights and biases are typically initialized randomly and then iteratively updated during training through backpropagation. The decoder stage of the autoencoder then maps h to a reconstructed x' of the same shape as x, as follows: x'=σ'(W'h'+b')

[0073] Here, σ', W', and b' for the decoder may be independent of the corresponding σ, W, and b for the encoder.

[0074] Variational autoencoder models make strong assumptions about the distribution of latent variables. They use variational methods for latent representation learning, which results in a further loss component and a specific estimator for the training algorithm called the Stochastic Gradient Variational Bayes (SGVB) estimator. This is used when the data is directed to a graphical model p θ (x|h) is generated, and the encoder is the posterior distribution p θ Approximation q to (h|x) φ Assuming that (h|x) has been learned, where φ and θ These represent the parameters of the encoder (recognition model) and decoder (generation model), respectively. The probability distribution of the latent vector of a VAE typically matches the probability distribution of the training data much more closely than a standard autoencoder. The purpose of a VAE is as follows:

number

[0075] Here, D KL This represents the Kullback-Leibler divergence. The prior distribution for the latent variables is typically a central isotropic multivariate Gaussian p- θ (h) is set to N(0,I). Generally, the shapes of the variational and likelihood distributions are chosen such that they are Gaussian distributions factorized as follows. q φ (h|x)=N(ρ(x),ω 2 (x)I) p φ (x|h)=N(μ(h),σ 2 (h)I)

[0076] Here, ρ(x) and ω 2 (x) is the encoder output, while μ(h) and σ 2 (h) is the decoder output.

[0077] Recent advances in the field of artificial neural networks, particularly convolutional neural networks, have enabled researchers to apply neural network-based techniques to the tasks of image and video compression. For example, end-to-end optimized image compression using networks based on variational autoencoders has been proposed.

[0078] Therefore, data compression is considered a fundamental and well-studied problem in engineering, generally formulated with the goal of designing a code for a given discrete data ensemble that has minimum entropy. This solution relies heavily on knowledge of the probabilistic structure of the data, and thus the problem is closely related to probabilistic source modeling. However, since all actual codes must have finite entropy, continuous-value data (such as vectors of image pixel intensity) must be quantized into a finite set of discrete values, which introduces errors.

[0079] In this situation, as is known as the irreversible compression problem, a trade-off must be made between two competing costs: the entropy (rate) of the discrete representation and the error (distortion) resulting from quantization. Different compression applications, such as data storage or transmission over channels with limited capacity, require different rate-distortion trade-offs.

[0080] Simultaneous optimization of rate and distortion is difficult. Without further constraints, the general problem of optimal quantization in high-dimensional spaces is difficult to solve. For this reason, most existing image compression methods operate by linearly transforming the data vector into a suitable continuous-value representation, independently quantizing its elements, and then encoding the resulting discrete representation using a reversible entropy code. This method is called transformative coding due to the central role of the transformation.

[0081] For example, JPEG uses the discrete cosine transform for blocks of pixels, while JPEG2000 uses multiscale orthogonal wavelet decomposition. Typically, the three components of the transform coding method—the transform, quantization, and entropy code—are optimized separately (often through manual parameter tuning). Modern video compression standards such as HEVC, VVC, and EVC also use transformed representations to code the residual signal after prediction. For this purpose, several transforms are used, such as the discrete cosine transform and sine transform (DCT, DST), as well as the low-frequency non-separable manually optimized transform (LFNST).

[0082] Variational image compression The Variational Autoencoder (VAE) framework can be considered a nonlinear transformation coding model. The transformation process can be divided into four main parts, as illustrated in Figure 3A, which shows the VAE framework.

[0083] The transformation process can be divided into four main parts. Figure 3A illustrates the VAE framework. In Figure 3A, the encoder 101 maps the input image x to a latent representation (indicated by y) via the function y=f(x). This latent representation may also be referred to below as part of the “latent space” or a point within it. The function f() is a transformation function that converts the input signal x to a more compressible representation y. The quantizer 102,

number

number

number

[0084] The latent space can be understood as a compressed representation of data where similar data points are closer to each other within the latent space. The latent space is useful for learning data features and finding a simpler representation of the data for analysis. Quantized latent representation T,

number

number

number

number

number

number

number

[0085] In Figure 3A, component AE 105 is an arithmetic coding module and a quantized latent representation

number

number

number

[0086] Arithmetic decoding (AD) 106 is the process of reversing the binarization process, where the binary numbers are converted back to sample values. Arithmetic decoding is provided by the arithmetic decoding module 106.

[0087] It should be noted that this disclosure is not limited to this particular framework. Furthermore, this disclosure is not limited to image or video compression, but can also be applied to object detection, image generation, and recognition systems.

[0088] In Figure 3A, there are two interconnected subnetworks. In this context, a subnetwork is a logical division between parts of the overall network. For example, in Figure 3A, modules 101, 102, 104, 105, and 106 are called the "encoder / decoder" subnetwork. The "encoder / decoder" subnetwork is responsible for encoding (generating) and decoding (parse) the first bitstream, "bitstream 1". The second network in Figure 3A, including modules 103, 108, 109, 110, and 107, is called the "hyperencoder / decoder" subnetwork. The second subnetwork is responsible for generating the second bitstream, "bitstream 2". The two subnetworks have different purposes.

[0089] The first subnetwork is • Transformation of input image x into latent representation y (which is easier to compress than x). • Latent expression y is a quantized latent expression

number

number

number

[0090] The purpose of the second subnetwork is to obtain the statistical properties of the samples in "bitstream 1" (e.g., mean, variance, and correlation between samples in bitstream 1) so that the compression of bitstream 1 by the first subnetwork becomes more efficient. The second subnetwork generates a second bitstream, "bitstream 2," which contains this information (e.g., mean, variance, and correlation between samples in bitstream 1).

[0091] The second network is a quantized latent representation

number

number

number

number

number

number

number

number

number

number

number

number

number

[0092] Figure 3A illustrates an example of a VAE (Variational Autoencoder), the details of which may differ in different implementations. For example, in a particular implementation, additional components may exist to more efficiently obtain the statistical characteristics of the samples of bitstream 1. In one such implementation, there may be a context modeler aimed at extracting cross-correlation information of bitstream 1. The statistical information provided by the second subnetwork may be used by the AE (Arithmetic Encoder) 105 and AD (Arithmetic Decoder) 106 components.

[0093] Figure 3A shows an encoder and decoder in a single drawing. As will be apparent to those skilled in the art, the encoder and decoder may be incorporated into different devices, and very often are.

[0094] Figure 3B shows the encoder, and Figure 3C shows the decoder components of the VAE framework separated. According to some embodiments, the encoder receives a picture as input. The input picture may include one or more channels, such as a color channel or other types of channels, such as a depth channel or a motion information channel. The outputs of the encoder (shown in Figure 3B) are bitstream 1 and bitstream 2. Bitstream 1 is the output of the encoder's first subnetwork, and bitstream 2 is the output of the encoder's second subnetwork.

[0095] Similarly, in Figure 3C, two bitstreams, namely bitstream 1 and bitstream 2, are received as input, and the reconstructed (decoded) image is

number

[0096] Specifically, as shown in Figure 3B, the encoder includes an encoder 121 that converts input x into a signal y, which is then provided to a quantizer 322. The quantizer 122 provides the information to an arithmetic coding module 125 and a hyperencoder 123. The hyperencoder 123 provides the bitstream 2 discussed above to a hyperdecoder 147, which then provides the information to an arithmetic coding module 105 (125).

[0097] The output of the arithmetic coding module is bitstream 1. Bitstreams 1 and 2 are the outputs of the coded signals, which are then provided (transmitted) to the decoding process. Unit 101 (121) is called the “encoder,” but the entire subnetwork shown in Figure 3B can also be called the “encoder.” The coding process generally refers to a unit (module) that converts an input into a coded (e.g., compressed) output. From Figure 3B, it can be seen that unit 121 can actually be considered the core of the entire subnetwork, as it performs the conversion of input x to y, which is a compressed version of x. Compression in encoder 121 may be achieved, for example, by applying a neural network, or any processing network having one or more layers in general. In such a network, compression may be performed by cascading processes that include downsampling to reduce the size and / or number of channels of the input. Thus, the encoder may be called, for example, a neural network (NN) based encoder.

[0098] The remaining parts in the diagram (quantization unit, hyperencoder, hyperdecoder, arithmetic encoder / decoder) are all responsible for improving the efficiency of the encoding process or for converting the compressed output y into a sequence of bits (bitstream). Quantization may be provided to further compress the output of the NN encoder 121 by lossy compression. The AE125 may perform binarization, which can further compress the signal quantized by lossless compression, in combination with the hyperencoder 123 and hyperdecoder 127 used to constitute the AE125. Thus, the entire subnetwork in Figure 3B can also be called the "encoder".

[0099] Most deep learning (DL)-based image / video compression systems reduce the dimensionality of a signal before converting it to binary (bits). For example, in the VAE framework, the encoder, which is a nonlinear transformation, maps the input image x to y, where y has smaller width and height than x. Since y has smaller width and height, and therefore smaller size, the dimensionality (size) of the signal is reduced, and thus it becomes easier to compress the signal y. It should be noted that, generally, the encoder does not necessarily have to reduce the size in both (or generally all) dimensions. Rather, some exemplary implementations may provide an encoder that reduces the size in only one dimension (or generally a subset thereof).

[0100] J. Balle, L. Valero Laparra, and EP Simoncelli (2015). “Density Modeling of Images Using a Generalized Normalization Transformation”, In: arXiv e-prints, 4 thAt the Int. Conf. for Learning Representations, 2016 (hereinafter referred to as "Balle"), the authors proposed a framework for end-to-end optimization of image compression models based on nonlinear transformations. The authors optimize for mean squared error (MSE) but use a more flexible transformation constructed from a cascade of linear convolution and nonlinearity. In particular, the authors use generalized divisive normalization (GDN) coupled nonlinearity, conceived from a neuronal model in biological visual systems and proven effective in Gaussianizing image density. This cascaded transformation is followed by uniform scalar quantization (i.e., each element is rounded to the nearest integer), effectively realizing a parametric form of vector quantization in the original image space. The compressed image is reconstructed from these quantized values ​​using an approximate parametric nonlinear inverse transform.

[0101] An example of such a VAE framework is shown in Figure 4, which utilizes six downsampling layers labeled 401-406. The network architecture includes a hyperplier model. (g) a ,g s ) shows the image autoencoder architecture, and on the right (h a ,h s ) supports an autoencoder that realizes hyperplier. The factored plier model is analyzed and synthesized g a and g s The same architecture is used for this purpose. Q represents quantization, and AE and AD represent the arithmetic encoder and arithmetic decoder, respectively. The encoder takes the input image x as g a This results in a response y (latent representation) with a spatially varying standard deviation. aThe encoding involves multiple convolutional layers with subsampling and generalized divisive normalization (GDN) as the activation function.

[0102] The response is h a It is supplied to summarize the distribution of the standard deviation at z. Then z is quantized, compressed, and transmitted as side information. The encoder then outputs the quantized vector

number

number

number

number

number

number

number

[0103] Layers that include downsampling are indicated by a downward arrow in the layer description. The layer description "Conv N,k1,2↓" means that the layer is a convolutional layer with N channels and a convolution kernel of size k1×k1. For example, k1 may be equal to 5 and k2 may be equal to 3. As above, 2↓ means that downsampling is performed in this layer by a coefficient of 2. Downsampling by a coefficient of 2 results in one of the dimensions of the input signal being reduced by half in the output. In Figure 4, 2↓ indicates that both the width and height of the input image are reduced by a coefficient of 2. Since there are six downsampling layers, if the width and height (also shown as x) of the input image 414 are given by w and h, then the output signal

number

[0104] Figure 4 also shows a decoder including upsampling layers 407-412. A further layer 420, implemented as a convolutional layer but not providing upsampling to the received input, is provided between the upsampling layers 411 and 410 in the input processing order. A corresponding convolutional layer 430 is also shown for the decoder. Such layers can be provided within the NN to perform operations on inputs that do not change the size of the input but change certain characteristics. However, such layers are not required.

[0105] Viewed in the order of processing bitstream 2 as it passes through the decoder, the upsampling layers are executed in reverse order, i.e., from upsampling layer 412 to upsampling layer 407. Here, each upsampling layer is shown to provide upsampling with an upsampling ratio of 2, indicated by ↑. Of course, it is not necessarily required that all upsampling layers have the same upsampling ratio, and other upsampling ratios such as 3, 4, 8, etc., may also be used. Layers 407-412 are implemented as convolutional layers (conv). Specifically, since they may be intended to provide an operation on the input that is the reverse of the encoder operation, the upsampling layers may apply an inverse convolution operation to the received input such that their size increases by a coefficient corresponding to the upsampling ratio. However, this disclosure is not generally limited to inverse convolution, and upsampling may be performed in any other way, such as by bilinear interpolation between two adjacent samples, or by nearest neighbor sample copying, etc.

[0106] In the first subnetwork, several convolutional layers (401-403) are followed by generalized divisive normalization (GDN) on the encoder side and inverse GDN (IGDN) on the decoder side. In the second subnetwork, the activation function applied is ReLU. It should be noted that this disclosure is not limited to this implementation, and in general, other activation functions may be used instead of GDN or ReLU.

[0107] Cloud solutions for machine tasks Video coding for machines (VCM) is another direction in computer science that is gaining popularity today. The main idea behind this technique is to transmit coded representations of image or video information that are intended for further processing by computer vision (CV) algorithms such as object segmentation, detection, and recognition. In contrast to traditional image and video coding that targets human perception, the quality characteristic is not reconstruction quality, but rather performance on computer vision tasks, such as object detection accuracy. This is illustrated in Figure 5.

[0108] Video coding for machines is also called collaborative intelligence, which is a relatively new paradigm for the efficient deployment of deep neural networks across mobile-cloud infrastructure. By splitting the network between the mobile side 510 and the cloud side 590 (e.g., cloud servers), it is possible to distribute the computational workload so that the overall energy and / or latency of the system is minimized. In general, collaborative intelligence is a paradigm in which the processing of a neural network is distributed among two or more different computing nodes, e.g., devices, generally any functionally defined node. Here, the term “node” does not refer to the neural network nodes described above. Rather, a (computation) node here refers to a separate device / module (physically or at least logically) that implements a part of the neural network. Such devices may be different servers, different end-user devices, a mix of servers and / or user devices and / or the cloud and / or processors, etc. In other words, computing nodes may be thought of as nodes that belong to the same neural network and communicate with each other to transmit coded data within / for the neural network. For example, to enable the execution of complex calculations, one or more layers may run on a first device (such as the device in mobile-side 510), and one or more layers may run on other devices (such as the cloud server in cloud-side 590). However, the distribution may also be finer-grained, with a single layer running on multiple devices. In this disclosure, the term “multiple” refers to two or more. In some existing solutions, a portion of the neural network functionality runs on a device (such as a user device or edge device) or multiple such devices, and the output (feature map) is then passed to the cloud. The cloud is a collection of processing or computing systems located outside the devices on which the portion of the neural network is running. The concept of collaborative intelligence is similarly extended to model training.In this case, data flows bidirectionally, that is, from the cloud to mobile during backpropagation in training, and from mobile to the cloud during the forward pass in training (as shown in Figure 5), and the same applies to inference.

[0109] Several studies have proposed semantic image compression by encoding deep features and then reconstructing the input image from them. Compression based on uniform quantization has been demonstrated, followed by context-based adaptive arithmetic coding (CABAC) from H.264. In some scenarios, it may be more efficient to transmit the output of the hidden layer (deep feature map) 550 from the mobile unit 510 to the cloud 590 rather than sending compressed natural image data to the cloud and performing object detection using the reconstructed image. Therefore, it may be advantageous to compress the data (features) generated by the mobile side 510, which may include a quantization layer 520, for this purpose. Correspondingly, the cloud side 590 may include an inverse quantization layer 560. Efficient compression of feature maps benefits image and video compression and reconstruction for both human perception and machine vision. Entropy coding methods, such as arithmetic coding, are common techniques for compressing deep features (i.e., feature maps).

[0110] Today, video content contributes more than 80% of internet traffic, and this percentage is expected to increase further. Therefore, it is crucial to build efficient video compression systems that can produce higher quality frames within a given bandwidth budget. Furthermore, most video-related computer vision tasks, such as video object detection or video object tracking, are sensitive to the quality of compressed video, and efficient video compression can benefit other computer vision tasks. Meanwhile, techniques in video compression are also useful for motion recognition and model compression. However, for the past few decades, video compression algorithms have relied on handcrafted modules, such as block-based motion estimation and discrete cosine transform (DCT), to reduce redundancy in video sequences, as described above. While each module may be well-designed, the overall compression system is not end-to-end optimized. It is desirable to further improve video compression performance by optimizing the entire compression system together.

[0111] End-to-end image or video compression DNN-based image compression methods can utilize large-scale end-to-end training and highly nonlinear transformations not used in conventional methods. However, directly applying these techniques to build an end-to-end learning system for video compression is no trivial matter. Firstly, learning how to generate and compress motion information in accordance with video compression remains an unresolved problem. Video compression methods rely heavily on motion information to reduce temporal redundancy in video sequences.

[0112] A simple solution is to represent motion information using learning-based optical flow. However, current learning-based optical flow methods aim to generate the most accurate flow field possible. Accurate optical flow is often not optimal for a particular video task. Furthermore, the amount of data in optical flow is considerably larger compared to motion information in conventional compression systems, and directly applying existing compression methods to compress optical flow values ​​significantly increases the number of bits required to store motion information. Secondly, it is unclear how to build a DNN-based video compression system by minimizing rate-distortion-based objectives for both residual and motion information. Rate-distortion optimization (RDO) aims to achieve higher quality (i.e., less distortion) of reconstructed frames given the number of bits (or bitrate) for compression. RDO is critical to video compression performance. To leverage the end-to-end training capabilities for learning-based compression systems, RDO strategies need to optimize the entire system.

[0113] In "DVC: An End-to-end Deep Video Compression Framework," Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp.11006-11015, the authors propose an end-to-end deep video compression (DVC) model that learns motion estimation, motion compression, and residual coding together.

[0114] Such an encoder is shown in Figure 6A. In particular, Figure 6A shows the overall structure of an end-to-end trainable video compression framework. To compress motion information, the CNN uses optical flow v t This corresponds to a more suitable representation for better compression. t It is shown to convert to . Specifically, an autoencoder-style network is used to compress the optical flow. The motion vector (MV) compression network is shown in Figure 6B. The network architecture is somewhat similar to ga / gs in Figure 4. In particular, the optical flow v t This is fed into a series of convolution operations and nonlinear transformations, including GDN and IGDN. The number of output channels c for convolution (deconvolution) is 128 here exemplarily, except for the last deconvolution layer which is equal to 2 in this example. The kernel size is k, for example, k=3. Given an optical flow of size M×N×2, the MV encoder generates a motion representation m of size M / 16×N / 16×128. t This generates the motion representation, which is then quantized (Q) and entropy coded.

number

number

number

[0115] Figure 6C shows the structure of the motion compensation unit. Here, the previously reconstructed frame x t-1Using the reconstructed motion information, the warp unit generates warped frames (usually with the help of interpolation filters such as bilinear interpolation filters). A separate CNN with three inputs then generates the predicted picture. The architecture of the motion-compensated CNN is also shown in Figure 6C.

[0116] The residual information between the original frame and the predicted frame is encoded by a residual encoder network. A highly nonlinear neural network is used to convert the residuals into their corresponding latent representations. Compared to the discrete cosine transform in conventional video compression systems, this method can better utilize the capabilities of nonlinear transformations and achieve higher compression efficiency.

[0117] From the above overview, it can be seen that CNN-based architectures can be applied to the compression of both images and videos, taking into account different parts of the video framework, including motion estimation, motion compensation, and residual coding. Entropy coding is a common method used for data compression, widely adopted by the industry, and can also be applied to feature map compression for either human perception or computer vision tasks.

[0118] Video coding for machines Video coding for machines (VCM) is another direction in computer science that is gaining popularity today. The main idea behind this technique is to transmit coded representations of image or video information that are intended for further processing by computer vision (CV) algorithms such as object segmentation, detection, and recognition. In contrast to traditional image and video coding that targets human perception, the quality characteristic is not reconstruction quality, but rather performance on computer vision tasks, such as object detection accuracy.

[0119] Recent research proposes a new developmental paradigm called collaborative intelligence, in which deep models are partitioned between mobile and cloud. Extensive experimentation under various hardware configurations and wireless connectivity modes has revealed that the optimal operating point in terms of energy consumption and / or computational latency is typically involved in partitioning the model at a point deep in the network. It has been found that today's common solutions, where the model resides either entirely in the cloud or entirely on mobile, are rarely optimal (if any exist). The concept of collaborative intelligence is similarly extended to model training. In this case, data flows bidirectionally, i.e., from cloud to mobile during backpropagation in training and from mobile to cloud during the forward pass in training, and similarly during inference.

[0120] Lossy compression of deep feature data has been studied based on HEVC intracoding in the context of recent deep modeling for object detection. Degradation of detection performance has been noted with increasing compression levels, and compression-extension training has been proposed to minimize this loss by generating models more robust to quantization noise in feature values. However, this remains a suboptimal solution, as the codecs used are highly complex and optimized for natural scene compression rather than deep feature compression.

[0121] The problem of deep feature compression for collaborative intelligence is addressed by techniques for object detection tasks using the common YOLOv2 network for studying the trade-off between compression efficiency and recognition accuracy. Here, the term "deep feature" is synonymous with "feature map." The word "deep" comes from the idea of ​​collaborative intelligence where the output feature maps of some hidden (deep) layer are captured and sent to the cloud for inference. This appears to be more efficient than sending compressed natural image data to the cloud and performing object detection using the reconstructed images.

[0122] Efficient feature map compression benefits both human perception and machine vision in the compression and reconstruction of images and videos. The shortcomings of state-of-the-art autoencoder-based compression methods, as described above, also apply to machine vision tasks.

[0123] Network Training Loss function The loss function may include multiple components. For image coding tasks, loss components related to reconstruction quality generally include L1 loss, L2 loss (also called MSE loss), MS-SSIM loss, VGG loss, LPIPS loss, GAN loss, etc., and further include loss components related to bitstream size.

[0124] The L1 loss is calculated by averaging the errors between points to obtain the L1 loss value. The L1 loss function can better evaluate the reconstruction quality of structured regions within an image.

[0125] Mean squared error (MSE) loss: A function for measuring the distance between two data points. In this embodiment of this application, the MSE loss is also called the L2 loss function. To obtain the L2 loss value, the mean of the squares of the errors between the points is calculated. The MSE loss may also be used to calculate the PSNR. The L2 loss is also a pixel-level loss. The L2 loss function can also better assess the reconstruction quality of structured regions in an image. When the L2 loss function is used to optimize an image coding and decoding network, the optimized image coding and decoding network can achieve a higher PSNR.

[0126] The structural similarity index measure (SSIM) is an objective criterion for evaluating image quality. A higher SSIM indicates better image quality. In this embodiment of this application, the structural similarity between two images on a certain scale is calculated to obtain an SSIM loss value. The SSIM loss is a loss based on artificial features. Compared to the L1 and L2 loss functions, the SSIM loss function can evaluate image reconstruction quality more objectively, i.e., it can evaluate the structured and unstructured regions of an image in a more balanced manner. When the SSIM loss function is used to optimize an image coding and decoding network, the optimized image coding and decoding network can achieve a higher SSIM.

[0127] Multiscale Structural Similarity Index (multi-scale SSIM, MS-SSIM): An objective criterion for evaluating image quality. A higher SSIM indicates better image quality. Multi-layer low-pass filtering and downsampling are performed separately on two images to obtain image pairs at multiple scales. Contrast maps and structural information are extracted from the image pairs at each scale, and SSIM loss values ​​at the corresponding scales are obtained based on the contrast maps and structural information. Luminance information is extracted from the image pair at the smallest scale, and luminance loss values ​​at the smallest scale are obtained based on the luminance information. The SSIM loss values ​​and luminance loss values ​​at multiple scales are then aggregated using a method for obtaining MS-SSIM loss values, for example, the aggregation method in equation (1).

number

[0128] In equation (1), the loss values ​​at all scales are aggregated using exponential power weighting and multiplication. Here, x and y represent two images separately, l represents the loss value based on luminance information, c represents the loss value based on the contrast map, and s represents the loss value based on structural information. The subscripts j=1, ..., M represent M scales corresponding to a total of M downsamplings, with j=1 representing the largest scale and j=M representing the smallest scale. The superscripts α, β, and γ represent the exponential powers of the corresponding terms, respectively.

[0129] The MS-SSIM loss function and the SSIM loss function have similar but better image evaluation effects. Compared to L1 loss and L2 loss, the MS-SSIM loss for optimization can improve the subjective experience of the human eye and meet objective evaluation metrics. When the MS-SSIM loss function is used to optimize image coding and decoding networks, the optimized image coding and decoding networks can achieve higher MS-SSIM.

[0130] Visual Geometry Group (VGG) Loss: VGG is the name of the organization that designed the CNN network and named it the VGG network. The image loss value determined based on the VGG network is called the VGG loss value. The process of determining the VGG loss value is essentially as follows: Features of the original image before compression and features of the decompressed reconstructed image (e.g., feature maps obtained through convolution calculations in a certain layer) are extracted separately using the VGG network, and then the distance between the features of the original image and the features of the reconstructed image at this scale is calculated to obtain the VGG loss value. This process can be thought of as the process of determining the VGG loss value according to the VGG loss function. The VGG loss function focuses on improving the texture reconstruction quality.

[0131] Learned perceptual image patch similarity (LPIPS): This is an enhanced VGG loss. Multiscale properties are introduced into the process of determining the LPIPS loss value. The process of determining the LPIPS loss value is essentially as follows: By using a VGG network, features of two images at multiple scales are extracted separately, then the distance between the features of the two images at each scale is calculated to obtain multiple VGG loss values, and then a weighted sum is performed on the multiple VGG loss values ​​to obtain the LPIPS loss value. This process can be considered a process of determining the LPIPS loss value according to the LPIPS loss function. Similar to the VGG loss function, the LPIPS loss function also focuses on improving the texture reconstruction quality.

[0132] Generative Adversarial Network Loss: By using the discriminators (also called discriminators) included in the GAN, features from two images are extracted separately, the distance between the features of the two images is calculated, and the Generative Adversarial Network Loss value is obtained. This process can be thought of as the process of determining the GAN loss value according to the GAN loss function. The GAN loss function also focuses on improving the texture reconstruction quality. The GAN loss includes at least one of the following: standard GAN loss, relative GAN loss, relative mean GAN loss, least squares GAN loss, etc.

[0133] Perceptual loss: There is broad and narrow perceptual loss. In this embodiment of this application, narrow perceptual loss is used as an example for illustrative purposes. VGG loss and LPIPS loss may be considered narrow perceptual loss. However, in other embodiments, loss calculated based on depth features extracted from the image may be considered broad perceptual loss. Broad perceptual loss may include narrow perceptual loss and may further include losses, such as the GAN loss described above. The perceptual loss function may cause the reconstructed image to better satisfy the subjective experience of the human eye, but may reduce PSNR and MS-SSIM.

[0134] Functional module Variable bitrate module An encoder can output a bitstream at different bitrates. Therefore, in some cases, the output of the encoding network is scaled (e.g., each channel is multiplied by a corresponding scaling factor, also called the target gain), and the input of the decoding network is descaled (e.g., each channel is multiplied by the reciprocal of the corresponding scaling factor, also called the target inverse gain). The scaling factor may be preset. Different quality levels or quantization parameters correspond to different target gain values. If the output of the encoding network is scaled to a smaller value, the bitstream size may decrease; otherwise, the bitstream size may increase.

[0135] Color format conversion RGB and YUV are common color spaces. Conversion between RGB and YUV may be performed according to formulas specified in standards such as CCIR601 and BT.709.

[0136] Separate structures for luma and chroma Some VAE-based codecs use the YUV color space as the input to the encoder and the output to the decoder. The Y component represents the lumens, and the UV component represents the chromens. The resolution of the UV component may be the same as or lower than the resolution of the Y component. Typical formats include YUV4:4:4, YUV4:2:2, and YUV4:2:0. The Y component is converted to a feature map F_Y through a network, and an entropy coding module generates a bitstream of the Y component based on the feature map F_Y. The UV component is converted to a feature map F_UV through another network, and an entropy coding module generates a bitstream of the UV component based on the feature map F_UV. Under this structure, the feature maps of the Y component and the UV component may be quantized independently so that bits can be flexibly allocated to the lumens and chromens. For example, for color-sensitive images, the feature map of the UV component may be less quantized, increasing the amount of bits in the UV component bitstream, improving the reconstruction quality of the UV component, and achieving a better visual effect.

[0137] In some other methods, the encoder concatenates the Y and UV components and then sends them to a UV component processing module (for converting the image information into a feature map). Furthermore, the decoder concatenates the reconstructed feature map of the Y component with the reconstructed feature map of the UV component and then sends them to a UV component processing module 2 (for converting the feature map into image information). In this method, the correlation between the Y and UV components may be used to reduce the bitstream of the UV component.

[0138] In this specification, “parameters” are values ​​used in the computational processes of each layer forming a neural network, and may include, for example, weights used when input values ​​are applied to a particular computational expression. Here, parameters may be represented in matrix form. Parameters are values ​​set as a result of training and may be updated as needed through separate training data.

[0139] This disclosure presents a method and apparatus for video frame synthesis based on a convolutional neural network. Due to conditional / dynamic convolution, the coding process can be accelerated and simplified compared to the prior art. In particular, according to embodiments, optical flow-based video frame interpolation can be combined with conditional / dynamic convolution. The application of dynamic convolution in the context of video frame synthesis is introduced herein for the first time. Optical flow-based video frame interpolation includes estimating the optical flow between a target frame and an input frame, warping the input frame or corresponding feature frame (tensor) by the predicted flow field for spatial alignment, and refining the warped input frame or feature frame to generate the target frame by the synthesis network. The present invention is achievable in any neural network for frame synthesis. In contrast to a general frame synthesis network in which all operations are performed at all spatial locations, this spatial operation is performed selectively based on the input (content). During inference, the proposed neural network can adapt to the content and provide different levels of computational effort.

[0140] In certain embodiments, the encoder-decoder-based network described as IFRNet in the paper “IFRNet: Intermediate Feature Refine Network for Efficient Frame Interpolation”, Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, June 22, pages 1969-1978 by L. Kong et al. can be combined with conditional / dynamic convolution. The concept of conditional / dynamic convolution is described in the context of human pose estimation in the paper “Dynamic Convolutions: Exploiting Spatial Sparsity for Faster Inference”, IEEE Conference on Computer Vision and Pattern Recognition, June 2020, pages 2320-2329 by T. Verelst and T. Tuytelaars.

[0141] Figure 7 shows an IFRNet architecture 700 that can be used in the embodiment. The IFRNet takes two frames I0 and I1 as input, having dimensions W×H×3 and time steps 0 and t=1, and outputs an intermediate frame having dimensions W×H×3 at an intermediate time step t∈[0,1]. The IFRNet architecture 700 includes a pyramidal encoder 710 and a coarse-fine decoder 720, D 1 ~D 4 This includes the pyramidal encoder 710, which has the characteristic Φ0 1 ~Φ0 4 and Φ1 1 ~Φ1 4 The pyramids are extracted from the input (reference) frames I0 and I1, respectively. Each pyramid level is characterized by a different resolution, i.e., the number of feature channels corresponding to the spatial size that is progressively removed. Intermediate flow field F 1 t→0~F 3 t→0 and F 1 t→1 ~F 3 t→1 are gradually improved by backward warping of a plurality of decoders 720, D 1 ~D 4 through the feature Φ0 1 ~Φ0 4 and Φ1 1 ~Φ1 4 and are gradually improved by backward warping to F 1 t→0 ~F 3 t→0 and F 1 t→1 ~F 3 t→1 and generate intermediate features

Number

Number

[0142] Figure 8 shows a dynamic convolution architecture 800 introduced by T. Verelst and T. Tuytelaars. The convolution process in the spatial domain is conditionally applied. The dynamic convolution includes a residual block with a learnable mask unit 810 that learns which spatial positions should be processed. The gating decision that functions as an execution mask is end-to-end trained, and the level of sparsity can be added to the loss function along with the output fidelity. The convolution is dynamically applied conditionally to the input image. First, the mask unit 810 generates a mask (gating decision) based on the input. The gating decision indicates the positions where the spatial 3×3 convolution should be applied. The mask is morphologically extended to avoid gaps. Then, the gather operation copies the elements corresponding to the active positions within the mask to a new intermediate tensor. Non-spatial operations such as activation functions and per-point 1×1 convolutions can be efficiently executed on the intermediate tensor, but spatial operations such as normal 3×3 convolutions require a small adaptation to map back to the original spatial pixel positions. A scatter operation is used to copy the results back to the original positions before performing the residual sum. Further details can be found in the literature by T. Verelst and T. Tuytelaars et al. cited above.

[0143] Figure 9 shows the architecture of a processing device 900 according to an embodiment. The processing device 900 may be a video encoder device or a video decoder device. The processing device 900 includes a convolutional neural network architecture for intermediate video frame synthesis as shown in FIG. 7 and implements dynamic convolution as shown in FIG. 8. The resulting architecture may be treated as a sparsified IFRNet architecture.

[0144] The processing device 900 takes as input two video frames I0 and I1 having dimensions W×H×3 and time steps 0 and t = 1, and outputs an intermediate frame having dimensions W×H×3 at intermediate time steps t∈[0,1]. The processing device 900 includes a (sparse) pyramid encoder 910 and a (sparse) decoder 920 including a coarse and fine decoder for each pyramid level. Each pyramid level is characterized by a number of feature channels corresponding to a different resolution, i.e., a spatial size that is gradually removed. For example, four pyramid levels having 32, 48, 72, and 96 feature channels respectively are used. Each encoder level may be composed of two 3×3 convolutional blocks having strides 2 and 1. After each convolution, a PReLU activation function follows. The intermediate flow fields F 1 t→0 ~F 3 t→0 and F 1 t→1 ~F 3 t→1 are gradually refined by backward warping the features provided by the pyramid decoder 910 through the decoder 920, and intermediate features are generated according to F 1 t→0 ~F 3 t→0 and F 1 t→1 ~F 3 t→1 The bilateral intermediate flow fields F

Number

Number

Number

[0145] Final output frame I t This can be placed, for example, in the decoded picture buffer. For example, the buffered output frame I t This can be used for intra-prediction or inter-prediction coding. According to another example, output frame I t It can be used for frame rate upconversion purposes, for example, for frame rate upconversion by a video decoder device.

[0146] On the decoder side 920, the intermediate flow is gradually refined by backward warping the extracted pyramidal features. Each of the coarse and fine decoders of decoder 920 outputs higher-level reconstructed intermediate features and flow fields on both sides. Each level of decoder 920 may consist of blocks of six 3x3 convolutions and one 4x4 deconvolution with strides of 1 and 1 / 2 and residual connections. The residual connection is added between the output of the first convolution and the output of the sixth convolution of each block. Each convolution is followed by a PReLU activation function.

[0147] Unlike the conventional architecture 700 shown in Figure 7, some convolutional blocks in the encoder 710 and decoder 720 are replaced to sparsify the data to be processed in the processing unit 900 of the embodiment shown in Figure 9. For example, every other convolutional block used in IFRNet may be replaced with a dynamic convolutional block in the encoder 910 of the processing unit 900, and in the decoder 920, the convolution of each residual block (the second to sixth convolutions) is replaced with three consecutive dynamic convolutions. For example, each dynamic convolutional block includes a mask unit, a set unit, a 1x1 convolutional unit, a 3x3 depth convolutional unit, another 1x1 convolutional unit, and a distributed operation unit. Furthermore, the convolution is followed by a batch norm ReLU activation function (see Figure 8 and its description).

[0148] In principle, a (content-based) trained mask unit may be applied pixel by pixel to input video frames I0 and I1, or in latent space (to a feature tensor), and the application of the mask unit results in sparse data in which irrelevant sub-parts of input video frames I0 and I1 (e.g., static objects or background) are removed from further processing.

[0149] A set of networks can be trained to optimize for different sparsity levels, each representing a relative reduction in data compared to the original input frame. The sparsity level may represent the overall proportion of convolutional positions to be skipped, e.g., the number of relative zeros within the sparsification mask units. Since trained sparsified IFRNets have the same structure, switching between different sparsity levels can be achieved by changing the weights within the network using one of the pre-trained weight sets. The network set can be trained to skip certain convolutional positions / filter applications depending on the content of the input data. During inference, the optimal sparsity level can be selected for the actual application / data input to appropriately adapt the computational load of the overall coding process. Based on the selected sparsity level (target sparsity), the corresponding pre-trained neural network can be used.

[0150] The decision regarding the sparsity level to be used must be recognized by both the video encoder device and the video decoder device. Different embodiments provide explicit sparsity level selection, estimated sparsity level selection, and implicit sparsity level selection.

[0151] In the case of explicit sparsity level selection and estimated sparsity level selection, a set of neural networks, including neural networks 910 and 1010 with mask units as shown in Figures 9 and 10 (left), is trained for different sparsity levels. A sparsity level between 10% and 90% of the data to be further processed for an input video frame, for example between 20% and 80%, may be selected based on empirical knowledge, with adjacent sparsity levels spaced at least 10% apart from each other. A wide range of sparsity levels can be covered, and intervals considerably smaller than 10% do not increase the accuracy of frame synthesis, assuming limited matching of target sparsity levels during the training process. In the case of explicit sparsity level selection, the video encoder device determines the target sparsity to be used and signals the target sparsity to the video decoder device along with the normal video stream metadata. The determination and signaling of the target sparsity may be performed frame by frame. Upon receiving the signaled target sparsity, the video decoder device knows how to synthesize the intermediate frames and which set of weights should be used in the sparsified network for synthesis. Based on explicit selection, the sparse IFRNet architecture can be applied to a wide variety of video compression schemes.

[0152] According to estimated sparsity level selection, a set of neural networks, including neural networks 910 and 1010 with mask units as shown in Figures 9 and 10 (left), is also trained for different sparsity levels. However, the target sparsity actually used is not selected by the video encoder device, nor is it signaled to the video decoder device; rather, it is estimated by both the video encoder device and the video decoder device. Since both the video encoder and video decoder devices use the same estimation algorithm, they will get the same result.

[0153] According to one embodiment, the target sparsity is estimated based on the quantization parameters and / or the number of partition blocks into which the input video frame is partitioned.

[0154] The larger the quantization parameter, the coarser the compression, resulting in a smoother image. Smoother images are easier to process, require fewer samples, and can be reliably processed using higher sparsity levels. During compression, the image is divided into blocks, which can have variable sizes depending on the content. Smoother regions are typically encoded using larger blocks, while high-density, high-contrast textures typically require partitioning into smaller blocks. Thus, fewer blocks indicate a smoother and easier image to process, allowing for higher sparsity levels.

[0155] According to a particular embodiment, the target sparsity is given by the formula

number

[0156] The coefficient k is experimentally selected according to the proportional contribution of the average quantization parameter and the average number of partitioned blocks for two input reference frames. Since the quantization parameter and block partitioning change from frame to frame, a list of previously used quantization parameters and block partitions may be maintained for each reference picture. Estimated sparsity level selection allows for simple computation and does not require signaling of the target sparsity.

[0157] According to implicit sparsity level selection, the target sparsity is also determined on both the video encoder and video decoder sides, without signaling, based on the implicitly sparsified neural network 1020 (see the right side of Figure 10). In this case, different sparsity levels are achieved not by training a separate network for each sparsity level, but rather by providing control parameters as further inputs and training the neural network to adapt on the fly to the selected control parameters (also called tuning the network). In this case as well, the previously calculated quantization parameters and block partition numbers remain attached to each reference picture from the reference picture list. To achieve tuning, NB0 n , NB1 n , QP0 n , QP1 n This is provided as a side channel to each dynamic convolution, as shown in Figure 10 (right side). This method is more difficult to train compared to the explicit and inferred sparsity level selection described above, but it allows it to operate without the need to update the neural network weights on the fly. Furthermore, the network can adapt its sparsification to the actual content of the reference frame, not just the mean value.

[0158] Figures 11A to 11C illustrate embodiments of a processing unit (e.g., a video encoder or video decoder device) that includes a convolutional neural network architecture for intermediate video frame synthesis based on a hybrid video codec, and implements dynamic convolution according to explicit (Figure 11A), estimated (Figure 11B), and implicit (Figure 11C) sparsity level selection, respectively.

[0159] All of the processing units 1100a, 1100b, and 1100c shown in Figures 11A, 11B, and 11C include a partitioning unit 1110, a transformation unit 1120, a quantization unit 1130, an inverse quantization unit 1140, an inverse transformation unit 1150, an entropy coding unit 1160, a loop filter 1170, a motion estimation unit 1180, and a prediction unit 1190 for selective intra or inter prediction.

[0160] The processing unit 1100a shown in Figure 11A includes an explicitly sparsified IFRNet 1111a, i.e., an IFRNet that implements the explicit sparsity level selection described above. Based on an encoder decision on the video encoder device side (or a decision by different units of the video encoder device), a specific sparsity level / ratio is selected and signaled to the video decoder device. The explicitly sparsified IFRNet 1111a is one of a set of pre-trained neural networks (trained for multiple different sparsity levels) that are selected based on the specific sparsity level / ratio selected by the encoder. A reference frame is input to the explicitly sparsified IFRNet 1111a and processed to obtain an intermediate video frame synthesized as described above, which is stored in the decoded picture buffer and / or reference picture list for further use (e.g., prediction or frame rate upconversion).

[0161] The processing device 1100b shown in FIG. 11B includes an estimated sparsified IFRNet 1111b that implements the above-described estimated sparsity level selection. The sparsity level is estimated based on a sparsity formula, for example, the above SR estimated The target sparsity actually used is selected based on the estimation result. The estimated sparsified IFRNet 1111b is one of a set of pre-trained neural networks (trained for multiple different sparsity levels) selected based on a specific target sparsity level / ratio resulting from the estimation. The reference frame is input to the estimated sparsified IFRNet 1111b, processed to obtain the intermediate video frame synthesized as described above, and stored in the decoded picture buffer and / or reference picture list for further use (e.g., prediction or frame rate up-conversion).

[0162] The processing device 1100c shown in FIG. 11C includes an implicitly sparsified IFRNet 1111c that implements the above-described implicit sparsity level selection. For each input frame, the quantization parameter and the number of partitioned blocks are input to the implicitly sparsified IFRNet as a side channel for on-the-fly training / provision of an appropriate target sparsity. The reference frame is input to the implicitly sparsified IFRNet 1111c, processed to obtain the intermediate video frame synthesized as described above, and stored in the decoded picture buffer and / or reference picture list for further use (e.g., prediction or frame rate up-conversion).

[0163] Figure 12 is a flowchart illustrating a method 1200 for video frame synthesis using a convolutional neural network according to an embodiment. The convolutional neural network includes an encoder and a decoder, the encoder including at least one first mask unit, and the decoder including at least one second mask unit. Method 1200 includes the steps of generating first data corresponding to only one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding to only one or more second sub-parts of a second video frame captured at a second time point different from the first time point, using at least one first mask unit; generating a first feature pyramid based on the first data and a second feature pyramid based on the second data using the encoder; and generating a third video frame synthesized for a third time point between the first and second time points, using at least one second mask unit, using the generated first and second feature pyramids, with the decoder. Method 1200 may be implemented in a video encoder device or a video decoder device. In particular, Method 1200 can be implemented in any of the processing units 900, 1100a, 1100b, and 1100c shown in Figures 9, 11A, 11B, and 11C, respectively. For example, a system configured to perform image compression, such as the neural network in Figure 1, can perform the steps of Method 1200.

[0164] Furthermore, method 1200 may be implemented in a processing device 1300 for video frame synthesis shown in Figure 13. The processing device 1300 (e.g., a video encoder device or a video decoder device) includes a convolutional neural network 1310. The convolutional neural network 1310 includes an encoder 1320 and a decoder 1330, wherein the encoder 1320 includes at least one first mask unit 1325, and the decoder includes at least one second mask unit 1335. The at least one first mask unit 1325 is configured to generate first data corresponding only to one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding only to one or more second sub-parts of a second video frame captured at a second time point different from the first time point. The encoder 1320 is configured to generate a first feature pyramid based on first data and a second feature pyramid based on second data, and the decoder 1330 is configured to generate a third video frame synthesized for a third time point between the first and second time points, using at least one second mask unit 1335 based on the generated first and second feature pyramids.

[0165] The processing unit 13 may include any of the processing units 900, 1100a, 1100b, and 1100c shown in Figures 9, 11A, 11B, and 11C, respectively.

[0166] While the operations are shown in a specific order in the drawings, this should not be understood as requiring that such operations be performed in a specific or sequential order as shown, or that all shown operations be performed, in order to achieve the desired result. In certain situations, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system modules and components in the above embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and systems can generally be integrated together into a single software product or packaged into multiple software products.

[0167] Specific embodiments of the subject matter are described. Other embodiments are within the scope of the following claims. For example, the operations described in the claims may be performed in a different order and still achieve the desired results. As an example, the processes shown in the accompanying drawings do not necessarily require the specific order or sequence shown to achieve the desired results. In certain implementations, multitasking and parallel processing may be advantageous.

[0168] Several exemplary implementations in hardware and software A corresponding system in which the above encoder-decoder processing chain can be deployed is shown in Figure 14. Figure 14 is a schematic block diagram showing an exemplary coding system that may utilize the technology of this application, e.g., a video, image, audio and / or other coding system (or simply a coding system). The video encoder device 20 (or simply abbreviated as encoder 20 hereafter) and video decoder device 30 (or simply abbreviated as decoder 30 hereafter) of the video coding system 10 represent examples of devices that may be configured to perform the technology described in various examples in this application. For example, video coding and decoding may use a neural network that may be distributed and which may apply the above bitstream parsing and / or bitstream generation to transmit feature maps between distributed computing nodes (two or more).

[0169] As shown in Figure 14, the coding system 10 includes a source device 12 configured to provide the encoded picture data 21 to a destination device 14 for decoding the encoded picture data 13, for example.

[0170] The source device 12 includes an encoder 20 and may also optionally include a picture source 16, a preprocessor (or preprocessing unit) 18, for example, a picture preprocessor 18 and a communication interface or communication unit 22.

[0171] The picture source 16 may include, or may include, any type of picture capture device, e.g., a camera for capturing real-world pictures, and / or any type of picture generation device, e.g., a computer graphics processor for generating computer-animated pictures, or any other type of device for acquiring and / or providing real-world pictures, computer-generated pictures (e.g., screen content, virtual reality (VR) pictures), and / or any combination thereof (e.g., augmented reality (AR) pictures). The picture source may also be any type of memory or storage for storing any of the above pictures.

[0172] To distinguish it from the processing performed by the preprocessor 18 and the preprocessing unit 18, the picture or picture data 17 may also be called the raw picture or raw picture data 17.

[0173] The preprocessor 18 is configured to receive (raw) picture data 17, perform preprocessing on the picture data 17, and obtain a preprocessed picture 19 or preprocessed picture data 19. The preprocessing performed by the preprocessor 18 may include, for example, cropping, color format conversion (e.g., RGB to YCbCr), color correction, or noise reduction. It is understood that the preprocessing unit 18 may consist of any optional components. It should also be noted that the preprocessing may use a neural network that uses presence indicator signaling (as in any of Figures 1 to 7).

[0174] The video encoder 20 is configured to receive pre-processed picture data 19 and provide encoded picture data 21.

[0175] The communication interface 22 of the source device 12 may be configured to receive the encoded picture data 21 and transmit the encoded picture data 21 (or a further processed version thereof) over the communication channel 13 to another device, for example, the destination device 14 or any other device, for storage or direct reconstruction.

[0176] The destination device 14 includes a decoder 30 (e.g., a video decoder 30), and may also include, optionally, a communication interface or communication unit 28, a post-processor 32 (or post-processing unit 32), and a display device 34.

[0177] The communication interface 28 of the destination device 14 is configured to receive encoded picture data 21 (or any further processed version thereof) from, for example, directly from the source device 12 or from any other source, for example, a storage device, for example, an encoded picture data storage device, and to provide the encoded picture data 21 to the decoder 30.

[0178] Communication interfaces 22 and 28 may be configured to transmit or receive encoded picture data 21 or encoded data 13 via a direct communication link between the source device 12 and the destination device 14, for example, via a direct wired or wireless connection, or via any type of network, for example, a wired or wireless network or a combination thereof, or any type of private and public network, or a combination thereof.

[0179] The communication interface 22 may be configured, for example, to package the encoded picture data 21 into an appropriate format, such as a packet, and / or to process the encoded picture data using any type of transmission encoding or processing for transmission over a communication link or communication network.

[0180] A communication interface 28 forming a corresponding communication interface 22 may be configured, for example, to receive transmitted data and process the transmitted data using any of the corresponding types of transmission decoding or processing and / or depackaging to obtain encoded picture data 21.

[0181] Both communication interfaces 22 and 28 may be configured as unidirectional or bidirectional communication interfaces, as indicated by the arrows of the communication channel 13 in Figure 14 pointing from the source device 12 to the destination device 14, and may be configured to acknowledge and exchange any other information relating to the communication link and / or data transmission, such as encoded picture data transmission, for example, to send and receive messages, for example, to set up a connection. The decoder 30 is configured to receive encoded picture data 21 and provide decoded picture data 31 or decoded picture 31.

[0182] The post-processor 32 of the destination device 14 is configured to post-process the decoded picture data 31 (also called reconstructed picture data), for example, the decoded picture 31, to obtain post-processed picture data 33, for example, the post-processed picture 33. The post-processing performed by the post-processing unit 32 may include, for example, color format conversion (e.g., YCbCr to RGB), color correction, cropping or resampling, or any other processing to prepare the decoded picture data 31 for display by, for example, the display device 34.

[0183] The display device 34 of the destination device 14 is configured to receive post-processed picture data 33 for, for example, displaying the picture to a user or viewer. The display device 34 may include any type of display for representing the reconstructed picture, such as an integrated or external display or monitor. The display may include, for example, a liquid crystal display (LCD), an organic light-emitting diode (OLED) display, a plasma display, a projector, a micro-LED display, a liquid crystal on silicon (LCoS), a digital light processor (DLP), or any other type of display.

[0184] Although Figure 14 shows the source device 12 and the destination device 14 as separate devices, the device embodiment may also include the functionality of both the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality. In such embodiments, the source device 12 or its corresponding functionality and the destination device 14 or its corresponding functionality may be implemented using the same hardware and / or software, or by separate hardware and / or software, or any combination thereof.

[0185] As will become apparent to those skilled in the art based on the description, the functions of the different units shown in Figure 14, or the presence and (exact) division of functions within the source device 12 and / or destination device 14, may vary depending on the actual device and application.

[0186] The encoder 20 (e.g., video encoder 20) or the decoder 30 (e.g., video decoder 30), or both the encoder 20 and the decoder 30, may be implemented via processing circuits such as one or more microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), discrete logic, hardware, dedicated video coding, or any combination thereof. The encoder 20 may be implemented via processing circuit 46 to implement various modules, including a neural network or a part thereof. The decoder 30 may be implemented via processing circuit 46 to implement any coding system or subsystem described herein. The processing circuits may be configured to perform various operations, as discussed below. Where the technology is partially implemented in software, the device may store instructions for the software in a suitable non-temporary computer-readable storage medium, and may use one or more processors to execute the instructions in hardware to perform the technology of this disclosure. Either the video encoder 20 or the video decoder 30 may be integrated as part of a composite encoder / decoder (codec) within a single device, for example, as shown in Figure 15.

[0187] The source device 12 and destination device 14 may include any of a wide range of devices, including handheld or fixed devices of any kind, such as notebook or laptop computers, mobile phones, smartphones, tablets or tablet computers, cameras, desktop computers, set-top boxes, televisions, display devices, digital media players, video game consoles, video streaming devices (such as content service servers or content distribution servers), broadcast receiver devices, broadcast transmitter devices, etc., and may or may not use an operating system. In some cases, the source device 12 and destination device 14 may be equipped for wireless communication. Therefore, the source device 12 and destination device 14 may also be wireless communication devices.

[0188] In some cases, the video coding system 10 shown in Figure 14 is merely an example, and the technology of this application may be applied to video coding configurations (e.g., video coding or video decoding) that do not necessarily involve any data communication between the coding device and the decoding device. In other examples, the data may be retrieved from local memory or streamed over a network. The video coding device may code the data and store it in memory, and / or the video decoding device may retrieve the data from memory and decode it. In some examples, coding and decoding are performed by devices that do not communicate with each other but simply code the data into memory and / or retrieve the data from memory and decode it.

[0189] Figure 16 is a schematic diagram of a video coding device 8000 according to an embodiment of the present disclosure. The video coding device 8000 is suitable for implementing the disclosed embodiments as described herein. In embodiments, the video coding device 8000 may be a decoder, such as the video decoder 30 in Figure 14, or an encoder, such as the video encoder 20 in Figure 14.

[0190] The video coding device 8000 includes an inlet port 8010 (or input port 8010) and a receiver unit (Rx) 8020 for receiving data, a processor, logic unit, or central processing unit (CPU) 8030 for processing data, a transmitter unit (Tx) 8040 and an exit port 8050 (or output port 8050) for transmitting data, and a memory 8060 for storing data. The video coding device 8000 may also include optical-to-electrical (OE) and electrical-to-optical (EO) components coupled to the inlet port 8010, receiver unit 8020, transmitter unit 8040, and exit port 8050 for optical or electrical signal input or output.

[0191] The processor 8030 is implemented by hardware and software. The processor 8030 may be implemented as one or more CPU chips, cores (e.g., as a multi-core processor), FPGAs, ASICs, and DSPs. The processor 8030 communicates with the inlet port 8010, the receiver unit 8020, the transmitter unit 8040, the exit port 8050, and the memory 8060. The processor 8030 includes a neural network-based codec 8070. The neural network-based codec 8070 implements the embodiments disclosed above. For example, the neural network-based codec 8070 implements, processes, prepares, or provides various coding operations. Thus, including the neural network-based codec 8070 provides a substantial improvement to the functionality of the video coding device 8000, resulting in the conversion of the video coding device 8000 to different states. Alternatively, the neural network-based codec 8070 is implemented as instructions stored in the memory 8060 and executed by the processor 8030.

[0192] Memory 8060 may include one or more disks, tape drives, and solid-state drives, and may be used as an overflow data storage device to store such programs when a program is selected for execution, and to store instructions and data read during program execution. Memory 8060 may be, for example, volatile and / or non-volatile, and may be read-only memory (ROM), random access memory (RAM), ternary content-addressable memory (TCAM), and / or static random-access memory (SRAM).

[0193] Figure 17 is a simplified block diagram of a device that can be used as either or both of the source device 12 and destination device 14 from Figure 14, according to an exemplary embodiment.

[0194] The processor 9002 within the device 9000 may be a central processing unit. Alternatively, the processor 9002 may be any other type of device or multiple devices currently existing or to be developed that are capable of manipulating or processing information. The disclosed implementation can be carried out with a single processor, e.g., processor 9002, as shown in the figures, but advantages in speed and efficiency can be achieved using two or more processors.

[0195] The memory 9004 within the device 9000 may, in its implementation, be a read-only memory (ROM) device or a random access memory (RAM) device. Any other suitable type of storage device can be used as memory 9004. Memory 9004 may contain code and data 9006 accessed by the processor 9002 using the bus 9012. Memory 9004 may further contain an operating system 9008 and an application program 9010, the application program 9010 including at least one program that enables the processor 9002 to perform the method described herein. For example, the application program 9010 may include applications 1 to N, the applications 1 to N further including a video coding application that performs the method described herein.

[0196] The device 9000 may also include one or more output devices, such as a display 9018. The display 9018 may, in one example, be a touch-sensitive display combining a display with a touch-sensitive element capable of sensing touch input. The display 9018 can be coupled to the processor 9002 via the bus 9012.

[0197] Although shown here as a single bus, the bus 9012 of device 9000 can consist of multiple buses. Furthermore, secondary storage can be directly coupled to other components of device 9000, or it can be accessed via a network, and may include a single integrated unit such as a memory card, or multiple units such as multiple memory cards. Thus, device 9000 can be implemented in a wide variety of configurations.

[0198] Figure 18 is a block diagram of a video coding system 10000 according to an embodiment of the present disclosure.

[0199] Platform 10002 within System 10000 can be a server or a local server. Alternatively, Platform 10002 can be any other type of device or multiple devices capable of computation, storage, transcoding, encryption, rendering, decryption, or encoding. The disclosed implementation can be carried out on a single platform as shown in the figure, e.g., Platform 10002, but advantages in speed and efficiency can be achieved by using two or more platforms.

[0200] The content delivery network (CDN) 10004 within system 10000 can be a group of geographically distributed servers. Alternatively, CDN 10004 can be any other type of device or a group of devices capable of accelerating web content delivery by data buffering, scheduling, distribution, or bringing web content closer to the user's location. The disclosed implementation can be carried out with a single CDN, e.g., CDN 10004, as shown in the figure, but the advantages in speed and efficiency can be achieved by using two or more CDNs.

[0201] Terminal 10006 within device 10000 can be a mobile phone, computer, television, laptop, or camera. Alternatively, terminal 10006 can be any other type of device or multiple devices capable of displaying video or images.

Claims

1. A method for synthesizing video frames using a convolutional neural network (1310) including an encoder (1320) and a decoder (1330), wherein the encoder (1320) includes at least one first mask unit (1325), and the decoder (1330) includes at least one second mask unit (1335), Step (S1210) of generating first data corresponding to only one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding to only one or more second sub-parts of a second video frame captured at a second time point different from the first time point, using the at least one first mask unit (1325), The encoder (1320) generates a first feature pyramid based on the first data and a second feature pyramid based on the second data (S1220), Step (S1230) of generating a third video frame synthesized for a third time point between the first time point and the second time point using the decoder (1330) based on the first and second feature pyramids generated, and the at least one second mask unit (1335). A method including (1200).

2. The method according to claim 1 (1200), wherein the at least one first and second mask unit (1325, 1335) is trained to acquire target sparsity of the first data relating to the first video frame and the second data relating to the second video frame.

3. The method according to claim 2 (1200), wherein the at least one first and second mask unit (1325, 1335) is trained based on the content of a training image.

4. The method according to claim 2 or 3 (1200), further comprising the step of determining the target sparsity based on quantization parameters used to compress the first and second video frames.

5. The method according to any one of claims 2 to 4 (1200), wherein the target sparsity depends on at least one of the number of objects present in the first and second video frames, the degree of movement of objects between the first and second video frames, the number of image partitions into which the first and second video frames are partitioned, and the compression levels of the first and second video frames.

6. A video encoder device (20) including a copy of the convolutional neural network (1310) for each of a plurality of predefined target sparsity levels selects the target sparsity from the predefined target sparsity levels; The steps include signaling the selected target sparsity to a video decoder device (30) which includes a copy of the convolutional neural network (1310) for each of the predefined target sparsity levels, The video decoder device processes the first and second video frames based on the signaled target sparsity. The method according to any one of claims 2 to 5, further comprising (1200).

7. The method (1200) of any one of claims 2 to 5, further comprising the step of determining the target sparsity by a video encoder device (20) including a copy of the convolutional neural network (1310) for each of the predefined target sparsity levels, based on at least one of a) quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned, using a video encoder device (20) including a copy of the convolutional neural network (1310) for each of the predefined target sparsity levels, and a video decoder device (30) including a copy of the convolutional neural network (1310) for each of the predefined target sparsity levels.

8. The aforementioned target sparsity level SR estimated is, formula [Math 1] Determined by the video encoder device (20) and the video decoder device (30) based on this, where NB 0 n , NB 1 n is the number of blocks into which the first and second video frames are each partitioned, NB max is the maximum possible number of blocks into which the first and second video frames can be partitioned depending on the resolution and possible block sizes of the first and second video frames, SR max is a predefined maximum target sparsity level d where QP 0 n + QP 1 n are quantization parameter values for the first and second video frames respectively, and QP max is a predefined maximum quantization parameter value, and k is a predefined weighting factor, the method (1200) according to claim 7.

9. The method according to any one of claims 6 to 8 (1200), wherein the plurality of predefined target sparsity levels include a plurality of sparsity levels between 10% and 90% of the sparsity of the first data relating to the first video frame and the second data relating to the second video frame, and adjacent sparsity levels are spaced apart from each other by an interval of at least 10% of the sparsity of the first data relating to the first video frame and the second data relating to the second video frame.

10. The steps include receiving at least one of the following by the at least one first mask unit (1325): a) quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned. The steps include determining the target sparsity based on at least one of the received a) the quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned, using the at least one first mask unit (1325); The method according to any one of claims 2 to 5, further comprising (1200).

11. The method according to any one of claims 1 to 10 (1200), wherein the step of generating the synthesized third video frame includes the step of jointly improving the intermediate flow fields on both sides together with a first intermediate feature pyramid reconstructed based on the first feature pyramid and a second intermediate feature pyramid reconstructed based on the second feature pyramid.

12. A video compression method comprising the steps of the method (200) according to any one of claims 1 to 11, A method wherein the synthesized third video frame is either stored as an S-frame in a decoded picture buffer or added to a list of reference pictures used for intra-prediction or inter-prediction.

13. A method for frame rate upconversion comprising the steps of the method (200) described in any one of claims 1 to 11, A method wherein the synthesized third video frame is used to increase the frame rate of a transmitted video including the first and second video frames.

14. A computer program stored on a non-temporary medium, which includes code that, when executed on one or more processors, performs the steps of the method according to any one of claims 1 to 13.

15. A processing device for video frame synthesis, One or more processors, A non-temporary computer-readable storage medium coupled to one or more processors and storing a program for execution by the one or more processors. An apparatus comprising, wherein the programming, when performed by the one or more processors, configures the apparatus to perform the method according to any one of claims 1 to 13.

16. A processing device for video frame synthesis including a convolutional neural network (1310), The convolutional neural network includes an encoder (1320) and a decoder (1330), wherein the encoder (1320) includes at least one first mask unit (1325), and the decoder (1330) includes at least one second mask unit (1325). The at least one first mask unit (1325) is configured to generate first data corresponding only to one or more first sub-parts of a first video frame captured at a first time point, and second data corresponding only to one or more second sub-parts of a second video frame captured at a second time point different from the first time point. The encoder (1320) is configured to generate a first feature pyramid based on the first data and a second feature pyramid based on the second data. The decoder (1330) is configured to generate a third video frame synthesized for a third time point between the first time point and the second time point, using the at least one second mask unit (1335) based on the generated first and second feature pyramids, and is a processing unit (900, 1300).

17. The processing apparatus (900, 1300) according to claim 16, wherein the at least one first and second mask unit (1325, 1335) is trained to acquire target sparsity of the first data relating to the first video frame and the second data relating to the second video frame.

18. The processing apparatus (900, 1300) according to claim 17, wherein the at least one first and second mask unit (1325, 1335) is trained based on the content of a training image.

19. The processing apparatus (900,1300) according to claim 17 or 18, wherein the target sparsity is determined based on quantization parameters used to compress the first and second video frames.

20. The processing apparatus (900,1300) according to any one of claims 17 to 19, wherein the target sparsity depends on at least one of the number of objects present in the first and second video frames, the degree of movement of objects between the first and second video frames, the number of image partitions into which the first and second video frames are partitioned, and the compression levels of the first and second video frames.

21. The processing apparatus (900,1300) according to any one of claims 17 to 20, wherein the decoder is configured to generate the synthesized third video frame by jointly improving the intermediate flow fields on both sides together with a first intermediate feature pyramid reconstructed based on the first feature pyramid and a second feature intermediate pyramid reconstructed based on the second feature pyramid.

22. A video encoder device comprising a copy of the processing apparatus (900, 1300) described in any one of claims 17 to 21, Each of the aforementioned copies includes the convolutional neural network (1310) for one of a predefined target sparsity level, and the video encoder device (20) is configured to select the target sparsity from a plurality of predefined target sparsity levels and signal the selected target sparsity to the video decoder device (30).

23. A video decoder device comprising a copy of the processing apparatus (900, 1300) described in any one of claims 17 to 21, Each of the copies comprises the convolutional neural network (1310) for one of a predefined target sparsity level, and the video decoder device (30) is configured to process the first and second video frames based on the target sparsity received from the encoder device (20).

24. A video encoder device comprising a copy of the processing apparatus (900, 1300) described in any one of claims 17 to 21, Each of the copies comprises a convolutional neural network (1310) for one of a predefined target sparsity level, and the video encoder device (20) is configured to determine the target sparsity based on at least one of a) quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned.

25. formula [Math 2] Based on the above, the target sparsity level SR estimated It is configured to determine, where NB 0 n , NB 1 n NB is the number of blocks into which the first and second video frames are partitioned, respectively. max SR is the maximum number of possible blocks that the first and second video frames can be partitioned into, depending on the resolution and possible block size of the first and second video frames. max This is a predefined maximum target sparsity level, QP 0 n +QP 1 n These are the quantization parameter values ​​for the first and second video frames, respectively, and QP max The video encoder device (20) according to claim 24, wherein k is a predefined maximum quantization parameter value and k is a predefined weighting coefficient.

26. A video decoder device comprising a copy of the processing apparatus (900, 1300) described in any one of claims 17 to 21, Each of the copies comprises the convolutional neural network (1310) for one of a predefined target sparsity level, and the video decoder device (20) is configured to determine the target sparsity based on at least one of a) quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned.

27. formula [Math 3] Based on the above, the target sparsity level SR estimated It is configured to determine, where NB 0 n , NB 1 n NB is the number of blocks into which the first and second video frames are partitioned, respectively. max SR is the maximum number of possible blocks that the first and second video frames can be partitioned into, depending on the resolution and possible block size of the first and second video frames. max This is a predefined maximum target sparsity level, QP 0 n +QP 1 n These are the quantization parameter values ​​for the first and second video frames, respectively, and QP max The video decoder device according to claim 26, wherein k is a predefined maximum quantization parameter value and k is a predefined weighting coefficient.

28. The at least one first mask unit (1325) is, a) receive at least one of the quantization parameters used to compress the video of the first and second frames, and b) receive at least one of the number of blocks into which the first and second video frames are partitioned. A processing apparatus (900,1300) according to any one of claims 17 to 21, configured to determine the target sparsity based on at least one of the received a) quantization parameters used to compress the video of the first and second frames, and b) the number of blocks into which the first and second video frames are partitioned.

29. A video compression device comprising a processing device (900, 1300) according to any one of claims 17 to 21 or 28, or a video encoder device (20) according to any one of claims 22, 24 or 25, or a video decoder device according to any one of claims 23, 26 and 27, configured to store the synthesized third video frame as an S-frame in a decoded picture buffer, or to add the synthesized third video frame to a list of reference pictures used for intra-prediction or inter-prediction.

30. A video frame rate upconversion device comprising a processing device (900, 1300) according to any one of claims 17 to 21 or 28, or a video decoder device according to any one of claims 23, 26 and 27, configured to increase the frame rate of a transmitted video including the first and second video frames using the synthesized third video frame.

Images