Temporal Structure Based Conditional Convolutional Neural Networks for Video Compression

Abstract

Video encoding and decoding is performed using an autoencoder that uses luminance information to derive motion information for chrominance prediction. In one embodiment, a conditional convolution is used to encode the motion flow information. The current condition, e.g., GOP structure, is used as an input to a series of fully connected layers to perform the conditional convolution. In a related embodiment, two or more reference frames are used to encode the motion flow information.

Description

[Technical field]

[0001] At least one of the present embodiments generally relates to a method or apparatus for compressing images and videos using Artificial Neural Network (ANN) based tools. [Background technology]

[0002] The Joint Video Exploration Team (JVET) between ISO / MPEG and ITU is currently investigating tools to replace some modules of the current standard H.266 / VVC, as well as replacing the entire structure with an end-to-end autoencoder method. Summary of the Invention

[0003] At least one of the present embodiments generally relates to a method or apparatus in the context of image and video compression using a novel artificial neural network (ANN) based tool.

[0004] According to a first aspect, there is provided a method, comprising: receiving an input to a conditional convolutional layer, the input comprising a concatenated tensor of a current block and at least one reference block, performing at least one conditional convolution on the input based on data representative of a current condition, the conditional convolution comprising a series of fully connected layers, encoding a motion flow using an output from the at least one conditional convolution, the current condition being based on a GOP structure, and generating a bitstream comprising the encoded motion flow.

[0005] According to a second aspect, there is provided a method, the method comprising: entropy decoding a bitstream comprising motion flow data of a current block of video data, performing at least one constrained deconvolution on said bitstream to generate a reconstructed residual, said conditional deconvolution comprising a series of fully connected layers, and combining said reconstructed residual with a prediction of said current block to generate a decoded block.

[0006] According to another aspect, an apparatus is provided, the apparatus comprising a processor, the processor may be configured to implement the general aspects by performing any of the described methods.

[0007] According to another general aspect of at least one embodiment, a device is provided that includes an apparatus according to any of the decoding embodiments and at least one of: (i) an antenna configured to receive a signal, the signal including a video block; (ii) a band limiter configured to limit the received signal to a frequency band including the video block; or (iii) a display configured to display an output representing the video block.

[0008] According to another general aspect of at least one embodiment, a non-transitory computer-readable medium is provided that includes data content generated according to any of the described decoding embodiments or variations.

[0009] According to another general aspect of at least one embodiment, there is provided a signal including video data generated according to any of the described encoding embodiments or variations.

[0010] According to another general aspect of at least one embodiment, a bitstream is formatted to include data content generated according to any of the described decoding embodiments or variations.

[0011] According to another general form of at least one embodiment, there is provided a computer program product including instructions that, when executed by a computer, cause the computer to perform any of the decoding embodiments or variations described.

[0012] These and other aspects, features, and advantages of the general aspects will become apparent from the following detailed description of exemplary embodiments, which proceeds with reference to the accompanying drawings. [Brief description of the drawings]

[0013] [Figure 1] Here is a basic autoencoder chain: [Diagram 2] A random access structure with a Group of Pictures (GOP) of eight frames is shown. [Diagram 3] 1 shows an architecture by Agustsson et al. for compressing video frames. [Figure 4] 1 shows the proposed architecture using conditional convolution. [Diagram 5] 1 illustrates one embodiment of a method for encoding video using the present principles. [Figure 6] 1 illustrates one embodiment of a method for decoding video using the present principles. [Figure 7] 1 illustrates one embodiment of an apparatus for encoding or decoding using the present principles. [Figure 8] 1 illustrates a standard common video compression scheme. [Figure 9] 1 shows a standard common video decompression scheme. [Figure 10] 1 illustrates a processor-based system for encoding / decoding under a general described aspect. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] In recent years, novel image and video compression methods based on neural networks have been developed. In contrast to traditional methods that apply handcrafted prediction modes and transformations, ANN-based methods rely on parameters that are learned on large datasets during training by iteratively minimizing a loss function. In the case of compression, the loss function describes both an estimate of the bit rate of the encoded bitstream and an objective function for the decoded content. Traditionally, the quality of the reconstructed image is optimized based on, for example, a measure of signal distortion or an approximation of the visual quality perceived by humans.

[0015] An exemplary end-to-end compression system is shown in Figure 1. The inputs X to the encoder part of the network can consist of: - an image or frame from a video; -Part of the image - A tensor representing a group of images / frames - A tensor representing a portion (crop) of a group of images / frames.

[0016] In either case, the input may have one or more color components, for example monochrome, RGB, or YCbCr (YUV) components. 1. An input tensor X is fed into an encoder network, which is usually a sequence of convolutional layers with nonlinear activation functions. Convolutional or spatial-depth 1 A large stride in the operations can be used to reduce the spatial resolution while increasing the number of channels. The encoder network can be viewed as a learned analytical transform. 2. The output of the encoder network, the “feature map” or “latent” Z, is quantized and entropy coded (EC) as a binary stream (bitstream) for storage or transmission. 3. The bitstream is a quantized version of Z.

[0017]

number

[0018]

number

[0019]

number

[0020] Note that more sophisticated architectures exist, for example adding a "hyper-autoencoder" (hyper-prior) to the network to jointly learn the parameters of the latent distribution for efficient compression. The present invention is not limited to the use of autoencoders. Any end-to-end differentiable codec can be considered. 1. Reshaping and permutation. For example, a tensor of size (N, H, W) can be resized to (N * 2 * 2, H / / 2, W / / 2).

[0021] Video Coding Traditional Video Compression To remove redundancy between frames of a video sequence, conventional video coding methods include inter-prediction tools: an image is divided into blocks.

[0022] For each block to be coded, the motion estimator tries to find a predictor in a previously reconstructed reference picture. Conventionally, pure transformations are considered and the motion estimator analyzes blocks of the same shape in a window at the same position as the current block to be predicted. Once the best match is found, a motion vector, i.e. the horizontal and vertical coordinate difference between the position of the current block and its predictor, is coded and transmitted in the bitstream together with the index of the reference picture. The difference between the source block and the predictor, also called the residual, is typically transformed, quantized and entropy coded in the bitstream.

[0023] At the decoder, the motion information and the residual are decoded. The predictor is accessed in the signaled reference frame using the decoded motion vector. The decoded block is then constructed by adding the decoded residual to the predictor.

[0024] In this scheme, the encoder performs the complex task of finding the best matching block within a window in the reference frame centered on the position of the current block, and the decoder only needs to analyze the motion information.

[0025] temporal structure In the previous section, we described the search and transmission of motion vectors between the current frame to be predicted and the reference picture. Some temporal structures allow the encoder to select a reference frame from among previously decoded pictures.

[0026] A typical structure used in the broadcast ecosystem is termed the random access structure, which consists of a periodic group of pictures (GOP) that consists of a repeating minimal temporal time-frame structure.

[0027] Figure 2 shows such a structure for a GOP of 8 frames. The first frame is an intraframe, i.e. an I-frame, meaning that it does not depend on other frames to be decoded. It can then be used as a random access point from which the decoder can start decoding the sequence. In broadcast, they are typically separated by one second of video, which allows a TV viewer to switch channels, start decoding the new channel they have selected, and not wait too long for the video to start being displayed. However, these frames usually require more bits to transmit, since they are not predicted using previously decoded content. Between I-frames, other frames are predicted using previously decoded frames. In the structure of Figure 2, it can be noticed that the coding order is different from the display order. This allows the encoder to predict frames using previously reconstructed pictures from the past and future. These frames are therefore called B-frames, after bidirectional prediction. In this case, the structure follows a hierarchical pattern with frames of type B0, B1, B2, and B3. B0 of each GOP is the first frame to be coded and is predicted using the last key frame (I or B0) from the previous GOP, e.g., frame 8 in display order is predicted from frame 0. The next frame in coding order can be predicted using past and future frames, as depicted by the arrows. Frame B1 can use frame B0 of type I, frame B2 can be predicted from frames I, B0, and B1, etc. The distance between the current frame and the reference frames varies depending on the decision made by the encoder.

[0028] Other GOP structures exist, and this example was chosen to show possible dependencies between frames at different distances between the reference and the prediction, dramatically improving coding efficiency. In the above example, when the motion in the scene is not too fast and irregular and there are no scene cuts, the B3 picture requires very few bits since it is mostly interpolated from adjacent frames.

[0029] Video Compression Using Artificial Neural Networks This section details state-of-the-art video compression methods based on artificial neural networks, which serve as a basic model for the description of the present invention. They rely on an autoencoder architecture to code distinct elements and address the removal of inter-frame redundancy by computing optical flow, which estimates dense motion flow between frames, i.e., each pixel location is assigned a separate 2D motion vector (horizontal and vertical).

[0030] Combined with traditional block-based inter prediction and residual coding, optical flow has so far not been able to demonstrate advantages over previously described block-based methods at reasonable complexity, however, recent work using autoencoders to encode motion fields has shown promising results.

[0031] Figure 3 shows the general architecture presented in the conventional approach. The I-frame on the left represents the encoding of an independently (intra) coded keyframe. This block consists of a simple autoencoder, developed for ANN-based image compression, composed of a convolutional analysis and synthesis module. Inference uses an Entropy Coder (EC) and an Entropy Decoder (ED) to arithmetically remove redundancies.

[0032] The predicted frame (P) is encoded according to the process on the right. At the encoder, the input is the current frame x cur and the previously reconstructed reference frame

[0033]

number

[0034]

number

[0035]

number

[0036]

number

[0037]

number

[0038]

number

[0039] The bitstream, denoted by [1 0 0...1 1] in Figure 3, then consists of binary codes representing either the latent representation of the I-frame, or the latent representation of the motion field and residual in case of predicted frames.

[0040] The present invention aims to solve the problem of optimizing the compression of video by using motion compensation based on optical flow estimation and coding. The dependency that can be modeled between two pictures depends heavily on the distance in time between the two frames. This distance comes from two main parameters: the number of frames that separate the two reference frames and the frame rate at which the video was collected.

[0041] ANN-based optical flow estimators typically rely on convolutional layers to capture features related to texture motion and deformation between frames. Flow amplitude and accuracy vary dramatically depending on the above parameters that define the action in the scene, the camera motion, and the temporal distance between two frames. Therefore, the design of the convolutional model needs to be able to capture a wide variety of displacements for different temporal coding structures. In this paper, we propose to improve the performance of joint flow estimation and compression by conditioning the convolutional layers of the optical flow autoencoder on the distance between reference frames.

[0042] Conventional video coding methods rely on block-based motion estimation, as detailed in Section 1. This design relies on the assumption that motion between frames can be modeled by a simple translation of each block. The encoder selects a block partition that minimizes the residual that needs to be transmitted to correct errors caused by this modeling. For each block to be coded, the encoder finds the best matching predictor in a previously decoded reference picture. To capture large motion and be able to estimate motion between distant frames, conventional codecs define a maximum motion vector. Recent research has shown the promise of using deep neural networks to estimate and code dense motion flow between frames of video. However, existing methods typically rely on simple temporal structures, such as the simplest possible variant of a low-latency structure, where each frame depends only on the immediately preceding frame in the display order. In contrast to the random access presented in the above section, the display order and coding order are identical, which solves the problem of distance between predicted and reference frames, but limits coding efficiency, since deep learning-based motion models can also exploit complex bidirectional temporal structures.

[0043] Recent work has proposed a deep motion flow estimation based framework that leverages bidirectional prediction. It relies on a recurrent model that preserves the state of each frame for the next frame in the coding order. However, this approach does not appear to be scalable, as the model is learned for an entire group of pictures. If a different GOP structure needs to be used, the model needs to be adapted and retrained.

[0044] In the present invention, it is proposed to improve video compression based on artificial neural networks by specializing the motion flow estimator according to the temporal structure. The overall architecture is similar to the one depicted in Figure 3, where the reference pictures can be determined by the structure as shown in Figure 2.

[0045] In particular, we propose to use conditional (de)convolution within the network modules (instead of standard (de)convolution) to train a general network that can handle complex GOP structures with varying distances between reference frames.

[0046] Proposed general method In this document, it is proposed to adapt the model by using conditional convolutions that rely on interdependencies in the GOP structure. The following paragraphs explain the context and design of motion flow estimation and coding in which the present invention can be applied.

[0047] Autoencoders used to encode motion flow are usually based on convolutional layers that take current and reference frame pixel values ​​as input. As can be seen in Figure 3, the frames are concatenated along the component axis to fit the convolutional model (f a , f s ) into the tensor. For example, if images are represented in RGB format, they can be shaped as tensors of size 3×h×w, where h and w denote the height and width of the frame, respectively. The input to the flow autoencoder consists of 6×h×w. Successive convolutional layers lead to lower dimensional and compressible latent representations that are encoded. Typically, the latent contains more channels (N>6 in the RGB example) but has a smaller spatial dimension, e.g.

[0048]

number

[0049] It should be noted that additional components can be estimated, for example, as in the scale-space flow schemes proposed in conventional methods where additional channels are used to identify and blur regions where motion estimation fails when occlusion occurs and no matching texture can be found in the reference picture. These techniques are compatible with the proposed invention described below.

[0050] Such autoencoders need to be trained, i.e., the convolution weights and entropy bottleneck are optimized for a large dataset of videos. As explained in the previous section, an efficient GOP structure leads to predictions occurring between frames that are at different distances from each other and in different contexts (past, future content types, etc.).

[0051] The first obvious way to address this problem is to learn and code each possible mapping in a given GOP structure. However, this dramatically increases the size of the entire encoder / decoder model, since multiple flow autoencoders are required. In addition, this is difficult to scale and adapt when new types or sizes of GOPs are used. For example, in traditional video coding, thanks to improved motion estimation and coding, the random access GOP considered in standardization now contains 16 or 32 frames, whereas in the past it only had 8 frames, as shown in Figure 2.

[0052] Another way to solve this is to train only one model for different inter-frame prediction contexts. However, this solution leads to suboptimal motion flow constructions since the same weights are used in different conditions.

[0053] In this paper, we propose to adapt the model by using conditional convolutions that rely on interdependencies in the GOP structure. Figure 4 shows the overall architecture. On the right, we have a motion flow autoencoder (f a , f s ) is expressed in the same way as in Figure 3. However, f a and f s The convolution and deconvolution used respectively in are replaced by a conditional convolution, resp. deconvolution. A vector encoding the current condition c is used as input to a series of layers, e.g. the fully connected (FC) layers of FIG. 4, which generate a vector of size corresponding to the number of channels in the output tensor of the convolution. Each channel of the output of the convolution is then multiplied by the corresponding value in the vector output of a smaller network that propagates the condition. The function f can be any scaling function, e.g. an exponential function.

[0054] Note that we propose to replace at least one layer: for example, we could replace all convolutions, but we can also envisage replacing only some layers to find the right balance between model size and compression efficiency.

[0055] In the exemplary Fig. 4, a nonlinearity of type relu is represented. The present invention is not limited to the use of relu activations, and any type of nonlinearity can be taken into account during the conditional convolution as well as inside the conditional module.

[0056] The conditions can be encoded as one-hot encoding. For example, if we consider three possible distances between references, the condition vector has a length of 3 and the three values ​​can be encoded as [0, 0, 1], [0, 1, 0], [1, 0, 0]. Although other schemes can also be considered, one-hot encoding for conditional modules has proven to be very effective in deep learning applications.

[0057] Variations In this section, different variations, examples and embodiments of the proposed method are presented.

[0058] Extending conditions The first parameter that can be modified in different variants is the actual condition parameter that is the input of the convolution module. C can be a single value, for example the distance between the predicted frame and the reference frame, but it can also be a vector that includes several parameters. The following conditions can also be taken into account, either separately or in combination: - the distance between the current frame and the reference frame, whether the reference is in the past or future of the current frame in display order, - how many frames are used to predict the current frame, - the frame rate at which the sequence was shot, -Content type: Games, VR360 content, screen content

[0059] More conditional modules The main embodiment of this proposal focuses on conditioning the joint optical flow estimation / compression to different prediction structures, since it is the main module that is logically affected, but other parts of the model can be adapted.

[0060] For example, residual coding or reconstruction filters can be used to improve the prediction. These modules can also be conditioned based on "C", as described above, to learn different behaviors based on the current prediction type.

[0061] Affine conditional functions The main embodiment described the use of the following functions to modify the output of the convolution: Y = f(s(c).(W * X)) where X is the input, W is the convolution weights, f (also denoted as f in Figure 4) is an arbitrary activation function (e.g., exponential, softplus), and s is a set of scaling functions trained under condition c.

[0062] In this variant, it is proposed to include the bias in the calculation of the output as follows: Y = f(s(c).(WX)) + b(c) where b(c) is a bias vector that contains as many elements as there are channels in the output tensor.

[0063] Syntax Condition c is derived using metadata signaling different elements, e.g., the Picture Order Count (POC) of the reference and current frames used to calculate the distance between frames and the temporal direction of prediction.

[0064] To activate the proposed conditional modules, syntax elements can be added.

[0065] In addition, some metadata may be needed to optimize the encoding. For example, the frame rate of the video can be used to derive c. Frame rate is typically not part of the required high-level syntax transmitted to the decoder, since it is rarely used in traditional video compression and does not interfere with the decoding process. In this variant, this metadata should be transmitted if necessary to derive c.

[0066] In another variant, the information c can be transmitted to the decoder as part of a high-level syntax: at the sequence or frame / slice level. The encoder can make some decisions depending on the content or the temporal structure and adjust the condition c accordingly. At the decoder, the transmitted c is used to derive the appropriate conditional deconvolution.

[0067] FIG. 5 illustrates one embodiment of an encoding method 500. The method begins at start block 501 and proceeds to block 510 for receiving input to a conditional convolutional layer, said input including a concatenated tensor of a current block and at least one reference block. Control proceeds from block 510 to block 520 for performing at least one conditional convolution on said input based on data representative of a current condition, said conditional convolution including a series of fully connected layers. Control proceeds from block 520 to block 530 for encoding a motion flow using an output from the at least one conditional convolution, where the current condition is based on a GOP structure. Control proceeds from block 530 to block 540 for generating a bitstream including said encoded motion flow.

[0068] 6 illustrates one embodiment of a method 600 for decoding video data. The method begins at start block 601 and proceeds to block 610 for entropy decoding a bitstream including motion flow data for a current block of video data. Control proceeds from block 610 to block 620 for performing at least one constrained deconvolution on said bitstream to generate a reconstructed residual, said constrained deconvolution including a series of fully connected layers. Control proceeds from block 620 to block 630 for combining said reconstructed residual with a prediction of said current block to generate a decoded block.

[0069] 7 shows one embodiment of an apparatus 700 for compressing, encoding, or decoding video using the methods described above. The apparatus comprises a processor 710 and may be interconnected through at least one port to a memory 720. Both the processor 710 and the memory 720 may also have one or more additional interconnections to external connections.

[0070] Additionally, the processor 710 is configured to insert or receive information in a bitstream and compress, encode, or decode using the methods previously described.

[0071] The embodiments described herein include various aspects, including tools, features, examples, models, techniques, and the like. Many of these aspects have been specifically described and described in a manner that may often sound limiting, at least to indicate their individual characteristics. However, this is for purposes of clarity of description and does not limit the applicability or scope of the aspects. In fact, all of the different aspects can be combined and substituted to provide further aspects. Moreover, these aspects can also be combined and substituted with aspects described in previous applications as well.

[0072] The aspects described and contemplated in this application can be implemented in many different forms. Figures 8, 9, and 10 provide some embodiments, but other embodiments are contemplated, and the descriptions of Figures 8, 9, and 10 do not limit the scope of implementations. At least one of the aspects generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as a method, an apparatus, a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or a computer-readable storage medium having stored thereon a bitstream generated according to any of the described methods.

[0073] In this application, the terms "reconstructed" and "decoded" may be used interchangeably, the terms "pixel" and "sample" may be used interchangeably, and the terms "image", "picture", and "frame" may be used interchangeably. Usually, but not necessarily, the term "reconstructed" is used on the encoder side, while the term "decoded" is used on the decoder side.

[0074] Various methods are described herein, each of which includes one or more steps or actions for achieving the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined. In addition, terms such as "first", "second", etc. may be used in various embodiments to modify elements, components, steps, operations, etc., such as, for example, "first decoding" and "second decoding". The use of such terms does not imply a modified ordering of operations, unless specifically required. Thus, in this example, the first decoding need not be performed before the second decoding, but may occur, for example, before, during, or during an overlapping time with the second decoding.

[0075] Various methods and other aspects described in the present application may be used to modify modules of the video encoder 100 and the video decoder 200, such as the intra prediction module, the entropy coding module, and / or the decoding module (160, 360, 145, 330) as shown in Figures 8 and 9. Furthermore, aspects of the present disclosure are not limited to VVC or HEVC, but may be applied to other standards and recommendations, such as existing or future developments, and extensions of any such standards and recommendations, including VVC and HEVC. Unless otherwise specified or technically excluded, aspects described in the present application may be used individually or in combination.

[0076] In this application, various numerical values ​​are used. The specific values ​​are for illustrative purposes and the described aspects are not limited to these specific values.

[0077] 8 shows an encoder 100. Variations of this encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all possible variations.

[0078] Before being encoded, a video sequence may undergo pre-encoding processing (101), such as applying a color transformation to the input color picture (e.g., converting from RGB 4:4:4 to YCbCr 4:2:0) or performing a remapping of the input picture components to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata can be associated with the pre-processing and added to the bitstream.

[0079] In the encoder 100, a picture is coded by the encoder elements as described below. The picture to be coded is divided (102) into units, e.g., CUs, and processed. Each unit is coded, e.g., using either intra mode or inter mode. When a unit is coded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and motion compensation (170) are performed. The encoder decides (105) whether to use intra mode or inter mode to code the unit, and indicates the intra / inter decision, e.g., by a prediction mode flag. A prediction residual is calculated, e.g., by subtracting (110) the predicted block from the original image block.

[0080] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, as well as the motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass both the transform and quantization, i.e., the residual is coded directly without applying a transform or quantization process.

[0081] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the predicted block are combined (155) to reconstruct an image block. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, deblocking / Sample Adaptive Offset (SAO) filtering to reduce coding artifacts. The filtered image is stored in a reference picture buffer (180).

[0082] Figure 9 shows a block diagram of a video decoder 200, in which the bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs a decoding pass that is the reverse of the encoding pass as described in Figure 8. The encoder 100 also generally performs video decoding as part of encoding the video data.

[0083] In particular, the decoder's input includes a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other coded information. Picture partition information indicates how the picture is partitioned. The decoder may then partition the picture according to the decoded picture partition information (235). The transform coefficients are inverse quantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the predicted block to reconstruct an image block. A prediction block may be obtained (270) from intra prediction (260) or motion compensated prediction (i.e., inter prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer (280).

[0084] The decoded picture may further undergo post-decoding processing (285), such as an inverse color conversion (e.g., converting from YcbCr 4:2:0 to RGB 4:4:4) or inverse remapping that performs the inverse of the remapping process performed in the pre-encoding processing (101). The post-decoding processing may use metadata derived in the pre-encoding processing and signaled in the bitstream.

[0085] FIG. 10 illustrates a block diagram of an example of a system in which various aspects and embodiments may be implemented. System 1000 may be embodied as a device including various components described below and configured to perform one or more of the aspects described herein. Examples of such devices include, but are not limited to, various electronic devices such as personal computers, laptop computers, smartphones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected appliances, and servers. The elements of system 1000, alone or in combination, may be embodied in an integrated circuit (IC), multiple ICs, and / or separate components. For example, in at least one embodiment, the processing elements and encoder / decoder elements of system 1000 are distributed across multiple ICs and / or separate components. In various embodiments, system 1000 is communicatively coupled to one or more other systems or other electronic devices, for example, via a communication bus or through dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more of the aspects described herein.

[0086] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein, for example to implement various aspects described herein. The processor 1010 can include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which can include non-volatile memory and / or volatile memory, including but not limited to Electrically Erasable Programmable Read-Only Memory (EEPROM), Read-Only Memory (ROM), Programmable Read-Only Memory (PROM), Random Access Memory (RAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Flash, magnetic disk drives, and / or optical disk drives. Storage devices 1040 may include, by way of non-limiting examples, internal storage devices, attached storage devices (including removable and non-removable storage devices), and / or network-accessible storage devices.

[0087] The system 1000 includes an encoder / decoder module 1030 configured to process data to provide, for example, encoded or decoded video, and the encoder / decoder module 1030 may include its own processor and memory. The encoder / decoder module 1030 corresponds to a module that may be included in a device that performs encoding and / or decoding functions. As is known, a device may include one or both of an encoding module and a decoding module. Furthermore, the encoder / decoder module 1030 may be implemented as a separate element of the system 1000, or may be incorporated within the processor 1010 as a combination of hardware and software, as is known to those skilled in the art.

[0088] Program code loaded into the processor 1010 or the encoder / decoder 1030 to perform various aspects described herein may be stored in the storage device 1040 and subsequently loaded into the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during execution of the processes described herein. Such stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, expressions, operations, and operational logic.

[0089] In some embodiments, memory internal to the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be the memory 1020 and / or the storage device 1040, e.g., dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, a fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations such as MPEG-2 (MPEG stands for Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC stands for High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard being developed by JVET).

[0090] Inputs to the elements of system 1000 can be provided through various input devices, as shown in block 1130. Such input devices include, but are not limited to, (i) a Radio Frequency (RF) section that receives, for example, an RF signal transmitted throughout a broadcast by a broadcaster, (ii) a Component (COMP) input terminal (or set of COMP input terminals), (iii) a Universal Serial Bus (USB) input terminal, and / or (iv) a High Definition Multimedia Interface (HDMI) input terminal. Although not shown in FIG. 10, another example includes composite video.

[0091] In various embodiments, the input devices of block 1130 have associated respective input processing elements as known in the art. For example, the RF section may be associated with elements suitable for (i) selecting a desired frequency (also referred to as selecting a signal or band-limiting a signal to a band of frequencies), (ii) down-converting the selected signal, (iii) band-limiting again to a narrower band of frequencies to select a signal frequency band that may be referred to as a channel in a particular embodiment (for example), (iv) demodulating the down-converted band-limited signal, (v) performing error correction, and (vi) demultiplexing to select a desired stream of data packets. The RF section of various embodiments includes one or more elements that perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or a near-baseband frequency) or to baseband. In one embodiment of a set-top box, the RF section and its associated input processing elements receive RF signals transmitted over a wired (e.g., cable) medium and perform frequency selection by filtering, downconverting, and refiltering to a desired frequency band. Various embodiments rearrange the order of the above-described (and other) elements, remove some of these elements, and / or add other elements that perform similar or different functions. Adding elements may include inserting elements between existing elements, such as, for example, inserting amplifiers and analog-to-digital converters. In various embodiments, the RF section includes an antenna.

[0092] Additionally, the USB and / or HDMI terminals may include respective interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be appreciated that various aspects of the input processing, e.g., Reed-Solomon error correction, may be implemented, for example, in a separate input processing IC or implemented within the processor 1010, as desired. Similarly, aspects of the USB or HDMI interface processing may be implemented in a separate interface IC or within the processor 1010, as desired. The demodulated, error corrected, and demultiplexed stream is provided to various processing elements, including, for example, the processor 1010 and an encoder / decoder 1030 that operates in combination with memory and storage elements to process the data stream as desired for presentation on an output device.

[0093] The various elements of system 1000 may be provided within an integrated housing in which the various elements can be interconnected and transmit data between each other using suitable connection arrangements, e.g., internal buses known in the art, including the Inter-IC (I2C) bus, wiring, and printed circuit boards.

[0094] The system 1000 includes a communication interface 1050 that enables communication with other devices over a communication channel 1060. The communication interface 1050 may include, but is not limited to, a transceiver configured to transmit and receive data over the communication channel 1060. The communication interface 1050 may include, but is not limited to, a modem or a network card, and the communication channel 1060 may be implemented within a wired and / or wireless medium, for example.

[0095] Data is streamed or otherwise provided to the system 1000 in various embodiments using a wireless network such as a Wi-Fi network, e.g., IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers). The Wi-Fi signal in these embodiments is received via a communication channel 1060 and a communication interface 1050 that are adapted for Wi-Fi communication. The communication channel 1060 in these embodiments is typically connected to an access point or router that provides access to external networks, including the Internet, to enable streaming applications and other over-the-top communications. In other embodiments, streaming data is provided to the system 1000 using a set-top box that delivers data via an HDMI connection in the input block 1130. In yet other embodiments, streaming data is provided to the system 1000 using an RF connection in the input block 1130. As indicated above, various embodiments provide data in a non-streaming manner. Additionally, various embodiments use wireless networks other than Wi-Fi, e.g., a cellular network or a Bluetooth network.

[0096] The system 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light-emitting diode (OLED) display, a curved display, and / or a foldable display. The display 1100 can be for a television, a tablet, a laptop, a mobile phone, or another device. The display 1100 can also be integrated with other components (e.g., as in a smartphone) or can be separate (e.g., an external monitor for a laptop). The other peripheral devices 1120 include, in various examples of embodiments, one or more of a stand-alone digital video disc (or digital versatile disc, DVR) (for both terms), a disc player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functionality based on the output of the system 1000. For example, a disc player performs the function of playing the output of the system 1000 .

[0097] In various embodiments, control signals are communicated between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV.Link, Consumer Electronics Control (CEC), or other communication protocols that allow control between devices with or without user intervention. The output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, the output devices can be connected to the system 1000 using a communication channel 1060 via the communication interface 1050. The display 1100 and speaker 1110 can be integrated into one unit with other components of the system 1000 in an electronic device such as, for example, a television. In various embodiments, the display interface 1070 includes a display driver, such as, for example, a timing controller (T Con) chip.

[0098] For example, if the RF portion of input 1130 is part of a separate set-top box, display 1100 and speakers 1110 may alternatively be separate from one or more of the other components. In various embodiments where display 1100 and speakers 1110 are external components, the output signal may be provided via a dedicated output connection including, for example, an HDMI port, a USB port, or a COMP output.

[0099] The embodiments may be performed by computer software implemented by the processor 1010, or by hardware, or by a combination of hardware and software. As a non-limiting example, the embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technology environment, and may be implemented using any suitable data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memories, and rib-bubble memories. The processor 1010 may be of any type suitable for the technology environment, and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0100] Various implementations include decoding. As used herein, "decoding" can encompass all or part of the processing performed on a received encoded sequence to generate a final output suitable for display, for example. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as, for example, entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoders of the various implementations described herein.

[0101] As a further example, in one embodiment, "decoding" refers to entropy decoding only, in another embodiment, "decoding" refers to differential decoding only, and in another embodiment, "decoding" refers to a combination of entropy and differential decoding. Whether the phrase "encoding process" is intended to refer to a task subset specifically or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0102] Various implementations involve encoding. As used herein, "encoding" can encompass all or part of the processing performed on an input video sequence to generate an encoded bitstream, for example, as described above for "decoding." In various embodiments, such processing includes one or more of the processes typically performed by an encoder, such as, for example, partitioning, differential encoding, transforming, quantizing, and entropy encoding. In various embodiments, such processing also or alternatively includes the processes performed by the encoders of the various implementations described herein.

[0103] As a further example, in one embodiment, "encoding" refers to entropy encoding only, in another embodiment, "encoding" refers to differential encoding only, and in another embodiment, "encoding" refers to a combination of differential and entropy encoding. Whether the phrase "encoding process" is intended to refer to a task subset specifically or to the broader encoding process as a whole will be clear based on the context of the specific description and will be well understood by one of ordinary skill in the art.

[0104] It should be noted that the syntax elements used herein are descriptive terms, and therefore do not exclude the use of other syntax element names.

[0105] Where a figure is presented as a flow chart, it should be understood that the figure also provides a block diagram of the corresponding apparatus. Similarly, where a figure is presented as a block diagram, it should be understood that the figure also provides a flow chart of the corresponding method / process.

[0106] Various embodiments may relate to parametric models or rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is usually considered as the constraint on computational complexity. This can be measured through a Rate Distortion Optimization (RDO) metric, or through Least Mean Square (LMS), Mean of Absolute Errors (MAE), or other such measures. Rate-distortion optimization is usually formulated to minimize a rate-distortion function that is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches may be based on extensive testing of all encoding options, including all considered modes or coding parameter values, with a thorough evaluation of their encoding costs and associated distortion of the reconstructed signal after encoding and decoding. To keep the encoding complexity down, faster approaches can also be used, especially with the calculation of approximate distortion based on a prediction or prediction residual signal rather than on the reconstructed signal. A mixture of these two approaches can also be used, such as by using approximate distortion for only some of the possible encoding choices and full distortion for others. Other approaches evaluate only a subset of the possible encoding choices. More generally, many approaches employ any of a variety of techniques to perform an optimization that is not necessarily a complete evaluation of both the coding cost and the associated distortion.

[0107] The implementations and aspects described herein may be implemented, for example, in a method or process, an apparatus, a software program, a data stream, or a signal. Even if discussed in the context of only a single implementation (e.g., discussed only as a method), the implementation of the discussed features may also be implemented in other forms (e.g., an apparatus or a program). For example, an apparatus may be implemented in appropriate hardware, software, and firmware. The method may be implemented, for example, in a processor, which refers to a general processing device including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. Processors also include, for example, communication devices such as computers, mobile phones, portable / personal digital assistants ("PDAs"), and other devices that facilitate communication of information between end users.

[0108] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an implementation," as well as other variations thereof, means that a particular feature, structure, characteristic, etc. described in connection with that embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "an embodiment" or "in one implementation," as well as other variations thereof, appearing in various places throughout this application are not necessarily all referring to the same embodiment.

[0109] Additionally, the application may refer to "determining" various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0110] Additionally, the application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0111] Additionally, the application may refer to "receiving" various information. Receiving, like "accessing," is intended to be a broad term. Receiving information may include, for example, one or more of accessing information or retrieving information (e.g., from a memory). Furthermore, "receiving" generally involves in some way, for example, storing information, processing information, transmitting information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information.

[0112] For example, in the case of "A / B," "A and / or B," and "at least one of A and B," it should be understood that use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first listed alternative (A), or selection of only the second listed alternative (B), or selection of both alternatives (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such language is intended to encompass the selection of only the first listed alternative (A), or the selection of only the second listed alternative (B), or the selection of only the third listed alternative (C), or the selection of only the first and second listed alternatives (A and B), or the selection of only the first and third listed alternatives (A and C), or the selection of only the second and third listed alternatives (B and C), or the selection of all three alternatives (A and B and C). This may be expanded as many times as the number of items listed, as would be apparent to one of ordinary skill in the art and related arts.

[0113] Also, as used herein, the term "signaling" specifically means to indicate something to a corresponding decoder. For example, in a particular embodiment, an encoder signals a particular one of multiple transforms, coding modes, or flags. Thus, in an embodiment, the same transform, parameter, or mode is used at both the encoder and decoder sides. Thus, for example, an encoder can transmit a particular parameter to a decoder (explicit signaling) so that the decoder can use the same particular parameter. In contrast, if the decoder already has the particular parameter as well as other parameters, a non-transmitting signaling (implicit signaling) can be used to simply allow the decoder to know and select the particular parameter. By avoiding the transmission of any actual function, bit savings are realized in various embodiments. It will be understood that signaling can be achieved in various ways. For example, one or more syntax elements, flags, etc. are used in various embodiments to signal information to a corresponding decoder. The above relates to the verb form of the word "signal", which may also be used as a noun in this specification.

[0114] As will be apparent to one skilled in the art, implementations can result in a variety of signals formatted to carry information that can be, for example, stored or transmitted. Information can include, for example, instructions for performing a method or data generated by one of the described implementations. For example, a signal can be formatted to carry a bit stream of the described embodiments. For example, such a signal can be formatted as an electromagnetic wave (e.g., using a radio frequency portion of the spectrum) or as a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information that the signal carries can be, for example, analog information or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored in a processor-readable medium.

[0115] The preceding sections describe several embodiments across various claim categories and types. The features of these embodiments may be provided alone or in any combination. Furthermore, the embodiments may include one or more of the following features, devices, or aspects across various claim categories and types, alone or in any combination. · Encoding and decoding of video information using conditional convolutions. · Encoding and decoding of the above with conditional coding based on the GOP structure. · Encoding and decoding as above using luminance to determine motion flow. The encoding and decoding of the above implemented using a series of fully connected layers. Encoding and decoding of the above using transposed convolution to generate motion information. The encoding and decoding as above, further comprising: warping the reference frame onto the current frame using the motion information to generate a predictor for the current frame. A bitstream or signal containing one or more of the described syntax elements, or variations thereof. A bitstream or signal including a syntax carrying information generated according to any of the described embodiments. · Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described. A method, process, apparatus, instruction storage medium, data storage medium, or signal according to any of the described embodiments. Inserting syntax elements into the signaling that allow the decoder to determine the decoding information in a manner corresponding to that used by the encoder. Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal that includes one or more of the described syntax elements, or variations thereof. An electronic device, such as a television, a set-top box, a mobile phone, a tablet, etc., that performs the conversion method according to any of the described embodiments. An electronic device, such as a television, set-top box, mobile phone, tablet, etc., that makes the conversion method determinations and displays the resulting images (e.g., using a monitor, screen, etc. type of display) according to any of the described embodiments. A television, set-top box, mobile phone, tablet, or other electronic device that selects, bandlimits, or tunes (e.g., using a tuner) a channel to receive a signal containing the encoded image, and performs a conversion method according to any of the described embodiments. A TV, set-top box, mobile phone, tablet, or other electronic device that receives a signal containing encoded images over the air and performs a conversion process.

Claims

1. A method for generating a convolutional input for a conditional convolutional layer, the conditional convolutional layer including a series of fully connected layers, the input including a concatenated tensor of a current block and a reference block; performing a conditional convolution on the input based on data representative of a current condition to obtain an output, the current condition being based on a group of pictures (GOP) structure; encoding a motion flow using the output from the conditional convolution; generating a bitstream including the encoded motion flow; A method for providing the above.

2. 1. An apparatus comprising: receiving an input to a conditional convolutional layer, the conditional convolutional layer including a series of fully connected layers, the input including a concatenated tensor of a current block and a reference block; performing a conditional convolution on the input based on data representative of a current condition to obtain an output, the current condition being based on a group of pictures (GOP) structure; encoding a motion flow using the output from the conditional convolution; generating a bitstream including the encoded motion flow; Processor configured for An apparatus comprising:

3. The method of claim 2, further comprising the steps of: entropy decoding a bitstream including motion flow data for a current block of video data; performing a constrained deconvolution on the bitstream to generate a reconstructed residual, the constrained deconvolution comprising a series of fully connected layers; combining the reconstructed residual with a prediction of the current block to generate a decoded block. A method for providing the above.

4. 1. An apparatus comprising: entropy decoding a bitstream including motion flow data for a current block of video data; performing a conditional deconvolution on the bitstream to generate a reconstructed residual, the conditional deconvolution including a series of fully connected layers; 4. Combining the reconstructed residual with a prediction of the current block to generate a decoded block. Processor configured for An apparatus comprising:

5. The method of claim 1 , implemented using an autoencoder.

6. The method of claim 1 , 3 or 5 , wherein the motion flow is based on luminance.

7. The method of claim 1 or 5, wherein the current conditions include the number of frames separating two reference blocks and the frame rate of the captured video that includes the blocks.

8. A method according to any one of claims 1, 5 or 7, further comprising a step of warping the reference block onto the current block using motion information to generate a predictor.

9. 9. The method of claim 1, 5, 7 or 8, wherein performing the conditional convolution comprises applying a non-linear function.

10. 4. The method of claim 1 or 3, wherein performing the conditional convolution comprises applying a scaling function.

11. 11. The method of claim 1, 3 or 5-10, wherein the bitstream is entropy coded.

12. An antenna configured to receive a signal, the signal including a video block. a band limiter configured to limit the received signal to a frequency band that includes the video block; and 5. The apparatus of claim 4, further comprising at least one of a display configured to display an output representative of the video block.

13. The apparatus of claim 2 implemented using an autoencoder.

14. The apparatus of claim 2, 4, 12 or 13, wherein the motion flow is based on luminance.

15. An apparatus as described in claim 2 or 13, wherein the current conditions include the number of frames separating two reference blocks and the frame rate of a collected video including the blocks.

16. The apparatus of claim 2, 13 or 15, further comprising: warping the reference block onto the current block using motion information to generate a predictor.

17. The apparatus of claim 2, 13, 15 or 16, wherein the conditional convolution includes a nonlinear function.

18. The apparatus of claim 2, 13, 15 or 17, wherein the conditional convolution includes a scaling function.

19. An apparatus as described in any one of claims 2, 4, 12 to 18, wherein the bitstream is entropy coded.

20. A computer program comprising instructions which, when executed by a computer, cause the computer to carry out a method according to any one of claims 1, 3, 5 to 11.

Images