Method and apparatus for picture encoding and decoding

Abstract

A method for video decoding is disclosed. A first reconstructed version of an image block of an encoded video is accessed. The first reconstructed version of the image block is filtered by a first neural network to form a second reconstructed version of the image block to be used as a reference. The second reconstructed version of the image block is filtered by a second neural network to form a third reconstructed version of the image block to be displayed. The first and second neural networks are jointly trained.

Description

1. Technical Field

[0001] At least one embodiment of the present invention generally relates to a method and apparatus for picture encoding and decoding, and more particularly, to a method and apparatus for filtering using multiple neural networks in video encoding and decoding. 2. Background Technology

[0002] To achieve high compression efficiency, image and video coding schemes typically employ prediction and transforms to exploit spatial and temporal redundancy in video content. Typically, intra-frame or inter-frame prediction is used to exploit intra-frame or inter-frame correlations. The difference between the original image block and the predicted image block, typically represented as a prediction error, prediction residual, or prediction value, is then transformed, quantized, and entropy-coded. During encoding, the original image block may be partitioned / split into sub-blocks using quadtree partitioning. To reconstruct the video, the compressed data is decoded using the inverse processes of prediction, transform, quantization, and entropy coding. 3. Summary of the Invention

[0003] According to a general aspect of at least one embodiment, a method for video decoding is presented, comprising:

[0004] accessing a first reconstructed version of an image block of the encoded video; and

[0005] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0006] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0007] Wherein, the first neural network and the second neural network are trained jointly.

[0008] According to a general aspect of at least one embodiment, an apparatus for video decoding is presented that includes one or more processors configured to:

[0009] accessing a first reconstructed version of an image block of the encoded video; and

[0010] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block The second reconstructed version will be used as a reference,

[0011] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0012] Wherein, the first neural network and the second neural network are trained jointly.

[0013] According to a general aspect of at least one embodiment, a method for encoding video data is presented, comprising:

[0014] accessing a first reconstructed version of an image block of the encoded video; and

[0015] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0016] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0017] Wherein, the first neural network and the second neural network are trained jointly.

[0018] According to a general aspect of at least one embodiment, an apparatus for encoding video data is presented that includes one or more processors configured to:

[0019] accessing a first reconstructed version of an image block of the encoded video; and

[0020] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0021] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0022] Wherein, the first neural network and the second neural network are trained jointly.

[0023] According to another general aspect of at least one embodiment, a bitstream is formatted to include a signal generated according to the encoding method described above.

[0024] One or more embodiments of the present invention further provide a computer-readable storage medium having stored thereon instructions for encoding or decoding video data according to at least a portion of any of the above methods. One or more embodiments further provide a computer-readable storage medium having stored thereon a bitstream generated according to the above encoding method. One or more embodiments further provide a method and apparatus for transmitting or receiving a bitstream generated according to the above encoding method. One or more embodiments further provide a computer program product including instructions for performing at least a portion of any of the above methods. 4. Description of the Figures

[0025] - Figure 1 Block diagram showing an exemplary HEVC (High Efficiency Video Coding) video encoder

[0026] Figure 2 The four in-loop filters used in JEM 6.0 are shown;

[0027] - Figure 3 is a block diagram illustrating an exemplary HEVC video decoder;

[0028] - Figure 4 An exemplary CNN (Convolutional Neural Network) is shown;

[0029] - Figure 5A 、 5B 5C show the training process, encoding process and decoding process of using CNN as the in-loop filter respectively;

[0030] - Figure 6 An exemplary method for filtering a reconstructed image block according to an embodiment is shown;

[0031] - Figure 7A The training process using CNN as an in-loop filter and CNN as an out-of-loop filter is shown;

[0032] - Figure 7B Describes the discriminator-based loss function;

[0033] - Figure 7C and 7D The encoding process and the decoding process using CNN as an in-loop filter and using CNN as an out-loop filter according to the first embodiment are respectively shown;

[0034] - Figure 8A 、 8B 8C show the encoding process and the decoding process using CNN as an in-loop filter and using CNN as an out-loop filter, respectively, according to the second embodiment;

[0035] - Figure 9A 、 9B 9C show the encoding process and the decoding process using CNN as an in-loop filter and using CNN as an out-loop filter, respectively, according to the third embodiment; and

[0036] - Figure 10 A block diagram is shown of an exemplary system in which aspects of the exemplary embodiments may be implemented. 5. Specific implementation methods

[0037] In HEVC coding, pictures are partitioned into square-shaped CTUs with a configurable size, typically 64×64. In other video coding standards, the size can be 128×128 or 256×256. A CTU is the root of a quadtree that is partitioned into 4 equally sized square coding units (CUs), which are half the size of the parent block in width and height. A quadtree is a tree in which a parent node can be divided into four child nodes, each of which can become another parent node that is divided into four child nodes. In HEVC, a coding block (CB) is partitioned into one or more prediction blocks (PBs) and forms the root of a quadtree partitioned into transform blocks (TBs). Corresponding to coding blocks, prediction blocks, and transform blocks, a coding unit (CU) includes a tree structured set of prediction units (PUs) and transform units (TUs), where the PU includes prediction information for all color components and the TU includes a residual coding syntax structure for each color component. The sizes of the CB, PB, and TB for the luma component apply to the corresponding CU, PU, and TU.

[0038] In more recent coding systems, a CTU is the root of a coding tree that is partitioned into coding units (CUs). A coding tree is a tree in which a parent node (usually corresponding to a block) can be partitioned into child nodes (e.g., into 2, 3, or 4 child nodes), each of which can become a parent node for another partitioned child node. In addition to the quadtree partitioning mode, new partitioning modes (binary tree symmetric partitioning mode, binary tree asymmetric partitioning mode, and ternary tree partitioning mode) are defined, which increase the total number of possible partitioning modes. A coding tree has a unique root node, such as a CTU. The leaves of the coding tree are the terminal nodes of the tree. Each node of the coding tree represents a block that can be further partitioned into smaller blocks, also called sub-blocks. Once the partitioning of the CTU into CUs is determined, the CUs corresponding to the leaves of the coding tree are encoded. The partitioning of the CTU into CUs and the coding parameters for encoding each CU (corresponding to the leaves of the coding tree) can be determined on the encoder side through a rate-distortion optimization process.

[0039] In this application, the term "block" or "picture block" may be used to refer to any of a CTU, CU, PU, TU, CB, PB, and TB. Additionally, the term "block" or "picture block" may be used to refer to macroblocks, partitions, and sub-blocks as specified in H.264 / AVC or other video coding standards, and more generally to sample arrays of many sizes.

[0040] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0041] This application describes a number of aspects, including tools, features, embodiments, models, methods, etc. Many of these aspects are described as having specificity, and at least in order to illustrate individual characteristics, are usually described in a manner that may sound limited. However, this is for the purpose of describing clearly, and does not limit the application or scope of those aspects. In fact, all different aspects can be combined and interchanged to provide other aspects. In addition, these aspects can also be combined and interchanged with the aspects described in earlier documents.

[0042] The aspects described and contemplated in this application can be implemented in many different forms. Figure 1 、 2 and 10 provide some embodiments, but other embodiments are contemplated and are not intended to be construed as limiting the scope of the present invention. Figure 1 、 2 The discussion of 10 does not limit the breadth of implementation. At least one aspect generally relates to video encoding and decoding, and at least one other aspect generally relates to transmitting a generated or encoded bitstream. These and other aspects can be implemented as methods, apparatus, computer-readable storage media having stored thereon instructions for encoding or decoding video data according to any of the described methods, and / or computer-readable storage media having stored thereon a bitstream generated according to any of the described methods.

[0043] In this application, the terms "reconstruction" and "decoding" are used interchangeably, the terms "pixel" and "sample" are used interchangeably, and the terms "image," "picture," and "frame" are used interchangeably. Typically, but not necessarily, the term "reconstruction" is used on the encoder side, while "decoding" is used on the decoder side.

[0044] Various methods are described herein, and each method includes one or more steps or actions for implementing the described method. Unless a specific order of steps or actions is required for proper operation of the method, the order and / or use of specific steps and / or actions may be modified or combined.

[0045] The various methods and other aspects described in this application can be used to modify e.g. Figure 1 and 2 Modules of the video encoder 100 and decoder 200 are shown, such as the in-loop filtering module and the post-decoding process (165, 265, 285). In addition, aspects of the present invention are not limited to VVC or HEVC, and can be applied to, for example, other standards and recommendations, whether pre-existing or developed in the future, as well as extensions of any such standards and recommendations (including VVC and HEVC). Unless otherwise specified or technically excluded, the aspects described in this application can be used alone or in combination.

[0046] Various numerical values are used in this application, for example, the number of layers of a convolutional neural network, the number of convolutions per layer, etc. The specific values are for example purposes, and the described aspects are not limited to these specific values.

[0047] Figure 1 An encoder 100 is shown. Variations of the encoder 100 are contemplated, but for clarity, the encoder 100 is described below without describing all contemplated variations.

[0048] Before being encoded, the video sequence may undergo pre-encoding processing (101), for example, applying a color transform to the input color picture (e.g., conversion from RGB 4:4:4 to YCbCr 4:2:0), or performing a remapping of the input picture components in order to obtain a signal distribution that is more resilient to compression (e.g., using histogram equalization of one of the color components). Metadata may be associated with the pre-processing and appended to the bitstream.

[0049] In an encoder 100, a picture is encoded by encoder elements as described below. The picture to be encoded is divided (102) and processed in units such as CUs. Each unit is encoded using, for example, intra or inter mode. When the unit is encoded in intra mode, it performs intra prediction (160). In inter mode, motion estimation (175) and compensation (170) are performed. The encoder decides (105) whether to encode the unit using intra mode or inter mode, and indicates the intra / inter decision by, for example, a prediction mode flag. For example, a prediction residual is calculated by subtracting (110) the predicted block from the original image block.

[0050] The prediction residual is then transformed (125) and quantized (130). The quantized transform coefficients, along with motion vectors and other syntax elements, are entropy coded (145) to output a bitstream. The encoder can skip the transform and apply quantization directly to the untransformed residual signal. The encoder can bypass the transform and quantization, i.e., directly decode the residual without applying a transform or quantization process.

[0051] The encoder decodes the coded block to provide a reference for further prediction. The quantized transform coefficients are dequantized (140) and inverse transformed (150) to decode the prediction residual. The decoded prediction residual and the prediction block are combined (155) and the image block is reconstructed. An in-loop filter (165) is applied to the reconstructed picture to perform, for example, DBF (deblocking) / SAO (sample adaptive offset) filtering to reduce coding artifacts. More generally, for video compression, other filters can be used for in-loop filtering. For example, Figure 2As shown, for the current JEM 6.0 (Joint Exploration Model 6.0) developed by JVET (Joint Video Exploration Team), four filters are applied successively, namely, bilateral filter (BLF), deblocking filter (DBF), SAO and ALF (Adaptive In-Loop Filter). These different filters are generally based on: (1) sample analysis and pixel classification and (2) class correlation filtering. The filtered image is stored in a reference picture buffer (180).

[0052] For ease of representation, we refer to the encoder input image as S and the in-loop filter input as And the output of the in-loop filter is called It can also be called the initial reconstruction or initial reconstructed version of the image. Figure 2 As shown, the input to the in-loop filtering is the sum of the prediction samples and the decoded prediction residual. For some blocks, when the prediction residual is zero or does not exist (eg, in skip mode), the input to the in-loop filtering is directly the predicted samples.

[0053] In the current JEM, the reconstructed samples are processed before the deblocking filter. Apply a bilateral filter. BLF works by basing the filter weights not only on the distance to neighboring samples but also on their values. Each sample in the initial reconstructed image is replaced by a weighted average of itself and its neighbors. The weights are calculated based on the distance from the center sample and the difference in sample values. Because the filter is a small plus-shaped filter (i.e., the filter uses four neighboring samples), all distances are either 0 or 1.

[0054] The sample at (i, j) will be filtered using its neighboring samples. The weight ω(i, j, k, l) is the weight assigned to the neighboring sample (k, l) for filtering the current sample (i, j) and is defined as:

[0055]

[0056] Among them, I(i, j) and I(k, l) are the initial reconstruction The intensity values of samples (i, j) and (k, l) in , σ d is a spatial parameter, and σ r is the range parameter. The characteristics (or strength) of the bilateral filter are determined by the parameter σ d and σ r Control. In JEM 6.0, σ d Set according to the transform unit size and prediction mode, and σ r Set based on the QP for the current block.

[0057] Output filtered sample value I F(i, j) is calculated as:

[0058]

[0059] In the encoder and decoder, the proposed bilateral filter is applied to each CU or block of a maximum size of 16×16 if the CU is larger than 16×16. In JEM 6.0, the bilateral filter is performed within the RDO (Rate-Distortion Optimization) loop on the encoder side. Therefore, the filtered block can also be used to predict subsequent blocks (intra-frame prediction).

[0060] ALF is fundamentally based on Wiener filter design, which aims to design a linear filter (1D or 2D) to minimize L2 distortion, that is, to minimize the squared error between the filtered samples and the reference samples (usually the original samples). In JEM, an ALF with block-based filter adaptation is applied. For the luma component, one of 25 filters is selected for each 2×2 block based on the direction and activity of the signal.

[0061] Up to three circularly symmetric filter shapes are supported for the luma component. An index is signaled at the picture level to indicate the filter shape used for the luma component of a picture. For the chroma components in a picture, a 5x5 diamond filter is always used.

[0062] Block classification is applied to each 2x2 block, which is classified into one of 25 categories based on local signal analysis (gradient, directionality).For the two chroma components in a picture, no classification method is applied, i.e., a single set of ALF coefficients is applied to each chroma component.

[0063] The filtering process for the luma component can be controlled at the CU level. A flag is signaled to indicate whether ALF is applied to the luma component of a CU. For chroma components, whether ALF is applied is indicated only at the picture level. ALF filter parameters are signaled in the first CTU before the SAO parameters of the first CTU. Up to 25 sets of luma filter coefficients can be signaled. To reduce bit overhead, filter coefficients from different classifications can be merged. Furthermore, the ALF coefficients of a reference picture can be reused as the ALF coefficients of the current picture.

[0064] Figure 3 A block diagram of a video decoder 200 is shown. In the decoder 200, a bitstream is decoded by decoder elements as described below. The video decoder 200 generally performs the same Figure 1 The encoding pass described in

[0044] is a decoding pass that is the inverse of the encoding pass described in

[0045] . The encoder 100 also typically performs video decoding as part of encoding the video data.

[0065] In particular, the input to the decoder comprises a video bitstream, which may be generated by the video encoder 100. The bitstream is first entropy decoded (230) to obtain transform coefficients, motion vectors, and other decoding information. Picture segmentation information indicates how the picture is segmented. The decoder can therefore divide (235) the picture according to the decoded picture segmentation information. The transform coefficients are dequantized (240) and inverse transformed (250) to decode the prediction residual. The decoded prediction residual is combined (255) with the prediction block to reconstruct the image block. The prediction block can be obtained (270) from intra-frame prediction (260) or motion compensated prediction (i.e., inter-frame prediction) (275). An in-loop filter (265) is applied to the reconstructed image. The filtered image is stored in a reference picture buffer RPB (280), also known as a decoded picture buffer (DPB).

[0066] The decoded pictures may be further subjected to post-decoding processing (285), such as post-filtering for improved rendering of the picture on a display, inverse color conversion (e.g., conversion from YCbCr 4:2:0 to RGB 4:4:4), or performing an inverse remapping of the remapping process performed in the pre-encoding process (101). The post-decoding processing may use metadata derived in the pre-encoding process and signaled in the bitstream.

[0067] It is known to use deep learning to perform in-loop filtering. The field of deep learning involves the use of deep neural networks. A neural network contains neurons organized in groups called layers. In a neural network, there is an input layer, an output layer, and one or more hidden layers. A deep neural network has two or more hidden layers. Video compression can be considered to be linked to pattern recognition, as compression generally looks for repeating patterns in order to remove redundancy. Because artifact removal or artifact reduction in video compression can be considered to be recognizing and restoring the original image, a neural network can be used as a filter to reduce artifacts. Here, artifact reduction is also referred to as image restoration, and the neural network used to reduce artifacts may also be referred to as a restoration filter.

[0068] Figure 4 An exemplary M-layer convolutional neural network is shown, where no filtering from the video codec is performed. The initial reconstruction of is restored by adding the residual R calculated by CNN Mathematically, the network can be represented as:

[0069]

[0070] Where W i and B i are the weight and bias parameters for layer i, respectively, g() is the activation function (such as a sigmoid or rectified linear unit (ReLU) function), and * represents the convolution operation. Output from CNN It can be stored in a reference picture buffer (RPB) to be used as a predictor for encoding or decoding subsequent frames.

[0071] For example, by minimizing the loss function defined based on the error between the restored image and the original image, the K training samples including W i and B i , parameter set θ of i = {1, ..., M}, such as:

[0072]

[0073] Note that the loss function can also contain other terms in order to stabilize convergence or avoid overfitting. These regularization terms can be simply added to the error function.

[0074] Figure 5A A CNN trained on a large image database is shown, where the network attempts to recover the reconstructed image by the encoder by minimizing the error (i.e., the difference from the original image). Figure 5B The resulting CNN is shown to be used in the encoder to restore the reconstructed image. The restored image can then be displayed or used as a reference for predicting other frames in the sequence. Symmetrically, as Figure 5C The decoder shown receives the bitstream, reconstructs the image and restores the image using the same CNN.

[0075] The input to the first layer of a CNN is usually the Y component of the reconstructed image, i.e. an image of size WxHx1, where W and H are the width and height of the image. In order to also use other information as input, we consider the reconstructed image as one channel, and use additional channels to input other information. In the example, the first layer applies N convolutions of n×n×1 filters to each sample of the input image, for example n=3. Thus, N values are obtained for each sample. A bias can then be added to each value, followed by a nonlinear function such as ReLU. If the input value is negative, ReLU returns 0, and if the input value is positive, ReLU returns the input value. In the second layer, M convolutions of 3×3xN filters are applied, followed by the addition of bias and ReLU functions. Several such layers can be applied consecutively.

[0076] In-loop filters such as SAO or DBF are applied to the reconstructed samples of the video image with the goal of reducing coding artifacts mainly due to the quantization of the signal. The output of these filters is stored in the RPB (reference picture buffer) of the encoder (respectively, the decoder). In in-loop filtering, an objective metric (usually based on the L1 or L2 norm) is traditionally used to optimize the filtered image, which is used as a reference image for prediction.

[0077] Using more perceptual metrics, such as MS-SSI, to optimize these filters generally reduces the performance of compression, i.e., the performance of prediction, since the resulting image is used as a reference for prediction.

[0078] Since images are filtered to improve picture rendering on a display, out-of-loop filters (such as those running in a television set) are preferably optimized using perceptual metrics.

[0079] For example, such additional post-filters are applied in set-top boxes or TV sets to adapt the picture rendering to the display characteristics. Since these filtered images are not used as reference images for prediction, perceptual improvements can be performed. Using objective metrics, such as the L2 norm, usually limits the perceptual improvement.

[0080] These two types of filtering are optimized independently in current systems and result in suboptimal filtering and / or higher complexity.

[0081] In at least one embodiment, a multi-part CNN is trained by jointly optimizing the quality of the reconstructed image for later use as a reference in the codec and the visual quality of the image for display. The multi-part CNN comprises at least two CNNs, such as Figure 7A 、 8A and CNNa and CNNb in 9A.

[0082] Figure 6 An exemplary method for filtering a reconstructed image block according to an embodiment is shown. Figure 6 The method can be used for encoding method and decoding method.

[0083] The method begins at step S10. At S12, a first reconstructed version of the image block is accessed. At S14, the first reconstructed version of the image block is filtered by a first neural network to form a second reconstructed version of the image block for use as a reference. At S16, the second reconstructed version of the image block is filtered by a second neural network to form a third reconstructed version of the image block to be displayed. According to this embodiment, the first neural network and the second neural network are jointly trained. The method ends at S18.

[0084] Figure 7A The joint training of CNNa and CNNb on a large image database according to the first embodiment is shown, where CNNa is used as an in-loop filter and CNNb is used as an out-loop filter. and As input. CNNa input are the pixels (luminance and / or chrominance channels) of the reconstructed image. The deep neural network used, CNNa, is used to learn the pixels to be added to the reconstructed image. To generate the restored image The residual Ra of is then trained on a fully convolutional neural network (FCNN) which is optimized to be used as a reference for encoding other images of the video.

[0085] The deep neural network CNNb used is to learn to be added to To generate the restored image to be displayed The output of CNNa is the residual Rb of the fully convolutional neural network (FCNN). As the input of CNNb. The entire CNN has two parts: CNNa for in-loop filtering and CNNb for out-loop filtering. The two parts of the network are trained jointly. The first loss function La used to train the objective filter, namely CNNa, is usually the original image S and the image filtered by CNNa. The L1 or L2 norm between:

[0086] The second loss Lb for training the perceptual filter, CNNb, is typically designed using high-level functions. For example, a discriminator such as in a generative adversarial network (GAN) can be used. A generative adversarial network (GAN) is a class of artificial intelligence algorithms used for unsupervised machine learning that is implemented by a system of two neural networks. One network, the generative model G, captures the data distribution, and one network, the discriminative model D, estimates the probability that the sample comes from the training data rather than G. The training procedure for G is to maximize the probability that D makes an error. This framework corresponds to a minimax two-player game.

[0087] Figure 7B Describes the loss function based on the discriminator. The discriminator is used to determine the Lb loss.

[0088] Reconstructed image Filtered through the entire network, and then filtered by CNNa and CNNb to produce a filtered image

[0089] is fed into the discriminator to output the discriminator score where D() denotes the discriminator forward function, and G() denotes the generator forward function (here the output of the codec is sent as input to CNNa+CNNb).

[0090] The original image S is also input into the discriminator.

[0091] During discriminator training, the following function is maximized:

[0092]

[0093] where θd is the network parameter of the discriminator.

[0094] During generator training, the following loss is minimized:

[0095]

[0096] Where θg is the network parameter of the generator CNNa+CNNb.

[0097] Another approach consists in building a discriminator network that combines S and Both are taken as input and the network is made to discriminate between the original image and the filtered image.

[0098] The typical D() function is a classification function. The discriminator attempts to classify the original image as correct (score = 1) and the filtered image as incorrect (score = 0). A typical classifier uses a ResNet network, but more advanced networks, especially those based on multi-scale analysis, can be used as discriminators.

[0099] The final loss function of the entire network CNNa+CNNb during training can be expressed as:

[0100]

[0101] Among them, λ and γ are parameters designed to balance different terms, and R is a typical weight regularization function used to improve convergence, which is set here to the L2 norm of the weights of CNNa+CNNb.

[0102] Figure 7C The resulting CNNa is shown to be used in the encoder to recover the reconstructed image. Symmetrically, as Figure 7D The decoder shown receives a bit stream, reconstructs an image using CNNa and CNNb determined according to the first embodiment, and restores the image.

[0103] Figure 8A The second embodiment shows the joint training of CNNa and CNNb on a large image database, where CNNa is used as an in-loop filter and CNNb is used as an out-loop filter. The input of CNNb consists of the reconstructed image of the in-loop filter. and restore the image It allows the second network CNNb to exploit information about the restoration already performed by the first network CNNa. The two inputs of CNNb and are concatenated, i.e., for each pixel, an appropriate quantity with two values is obtained (from The value of and from The concatenation of the two inputs thus produces a tensor of size WxHx2. Thus, the first layer of CNNb applies N convolutions of nxnx2 filters to each sample, for example, n=3. The same as used in the first embodiment and Figure 7A The same loss functions La and Lb as described in . In particular, Figure 7B The discriminator can therefore be used to define the loss function Lb.

[0104] In a variant, CNNb does not use the image recovered by CNNa (the final output of CNNa), but instead uses an intermediate representation, e.g., a latent representation (typically, the output of the layer before the last layer, which has size WxHxN instead of WxHx1 for the final output). This allows CNNb to utilize the latent representation of CNNa to perform perceptual filtering.

[0105] Figure 8B The resulting CNNa is used in an encoder to recover the reconstructed image. The encoder first ( Figure 7C ) and the second ( Figure 8B ) is the same as in the embodiment. Symmetrically, as Figure 8C The decoder shown receives a bit stream, reconstructs an image using CNNa and CNNb determined according to the second embodiment, and restores the image.

[0106] Figure 9A It shows the joint training of CNNa, CNNa' and CNNb on a large image database according to the third embodiment.

[0107] The network consists of three parts: CNNa, CNNa' and CNNb. The output S0 of CNNa is used as input to CNNa' and CNNb. The purpose of the network CNNa' is to recover the image. The main idea is to have a common part (CNNa) used by both objective and perceptual filtering and a dedicated part (CNNa' for objective filtering and CNNb for perceptual filtering). The output S0 does not have to be an image, but can be a tensor. The tensor S0 can typically have dimensions WxHxN, where WxH is the image size and N is the number of convolutions on the last layer of CNNa. Figure 7A The same loss function used in the first embodiment is shown.

[0108] Figure 9B The resulting CNNa and CNNa' are shown to be used in the encoder to restore the reconstructed image. Symmetrically, as Figure 9C The decoder shown receives a bit stream, reconstructs an image using the CNNa, CNNa', and CNNb determined according to the second embodiment, and restores the image.

[0109] In one variant that can be applied to all three embodiments, the coefficients of the "out-of-loop filter", i.e., the network parameters of CNNb, are sent in the bitstream. In a second variant that can also be applied to all three embodiments, the coefficients are learned by training on a large dataset and are known to the post-filtering module.

[0110] These three embodiments make it possible to obtain optimal image restoration for display while maintaining high codec performance by combining jointly trained CNN-based filters. It also relies on sharing of CNNa and CNNb layers, thus reducing the computational requirements of CNNb. For the third embodiment, the sharing is even higher.

[0111] Figure 10 A block diagram showing an example of a system in which various aspects and embodiments are implemented is shown. System 1000 can be implemented as a device including the various components described below, and is configured to perform one or more aspects described herein. Examples of such devices include, but are not limited to, various electronic devices, such as personal computers, laptop computers, smart phones, tablet computers, digital multimedia set-top boxes, digital television receivers, personal video recording systems, connected home appliances and servers. The elements of system 1000 can be implemented individually or in combination in a single integrated circuit (IC), multiple ICs and / or discrete components. For example, in at least one embodiment, the processing and encoder / decoder elements of system 1000 are distributed over multiple ICs and / or discrete components. In various embodiments, system 1000 is coupled to one or more other systems or other electronic devices via, for example, a communication bus or by dedicated input and / or output ports. In various embodiments, system 1000 is configured to implement one or more aspects described herein.

[0112] The system 1000 includes at least one processor 1010 configured to execute instructions loaded therein for implementing various aspects described herein, for example. The processor 1010 may include embedded memory, input / output interfaces, and various other circuits known in the art. The system 1000 includes at least one memory 1020 (e.g., a volatile memory device and / or a non-volatile memory device). The system 1000 includes a storage device 1040, which may include non-volatile memory and / or volatile memory, including but not limited to electrically erasable programmable read-only memory (EEPROM), read-only memory (ROM), programmable read-only memory (PROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, magnetic disk drive, and / or optical disk drive. As non-limiting examples, the storage device 1040 may include an internal storage device, an attached storage device (including detachable and non-detachable storage devices), and / or a network-accessible storage device.

[0113] System 1000 includes an encoder / decoder module 1030, which is configured to process data to provide encoded video or decoded video, for example, and may include its own processor and memory. Encoder / decoder module 1030 represents a module (or modules) that may be included in a device to perform encoding and / or decoding functions. As is known, a device may include one or both encoding and decoding modules. In addition, encoder / decoder module 1030 may be implemented as a separate element of system 1000 or may be incorporated into processor 1010 as a combination of hardware and software as known to those skilled in the art.

[0114] Program code to be loaded onto the processor 1010 or the encoder / decoder 1030 to perform various aspects described in this document may be stored in the storage device 1040 and subsequently loaded onto the memory 1020 for execution by the processor 1010. According to various embodiments, one or more of the processor 1010, the memory 1020, the storage device 1040, and the encoder / decoder module 1030 may store one or more of various items during the execution of the processes described herein. These stored items may include, but are not limited to, input video, decoded video or portions of decoded video, bitstreams, matrices, variables, and intermediate or final results from the processing of equations, formulas, operations, and operational logic.

[0115] In some embodiments, memory within the processor 1010 and / or the encoder / decoder module 1030 is used to store instructions and provide working memory for processing required during encoding or decoding. However, in other embodiments, memory external to the processing device (e.g., the processing device may be the processor 1010 or the encoder / decoder module 1030) is used for one or more of these functions. The external memory may be memory 1020 and / or storage device 1040, such as dynamic volatile memory and / or non-volatile flash memory. In some embodiments, the external non-volatile flash memory is used to store, for example, the operating system of the television. In at least one embodiment, fast external dynamic volatile memory such as RAM is used as working memory for video encoding and decoding operations, such as working memory for MPEG-2 (MPEG refers to Moving Picture Experts Group, MPEG-2 is also known as ISO / IEC 13818, and 13818-1 is also known as H.222, and 13818-2 is also known as H.262), HEVC (HEVC refers to High Efficiency Video Coding, also known as H.265 and MPEG-H Part 2), or VVC (Versatile Video Coding, a new standard developed by JVET, Joint Video Experts Team).

[0116] As shown in block 1130, input to the elements of system 1000 may be provided through various input devices. Such input devices include, but are not limited to, (i) an RF section that receives a radio frequency (RF) signal transmitted over the air, for example, by a broadcaster, (ii) a component (COMP) input terminal (or a set of COMP input terminals), (iii) a universal serial bus (USB) input terminal, and / or (iv) a high-definition multimedia interface (HDMI) input terminal. Figure 10 Other examples not shown in include composite video.

[0117] In various embodiments, the input device of block 1130 has associated corresponding input processing elements known in the art. For example, the RF section may be associated with elements suitable for: (i) selecting a desired frequency (also known as selecting a signal, or band-limiting a signal to a frequency band), (ii) down-converting the selected signal, (iii) again band-limiting to a narrower frequency band to select a signal frequency band that may be referred to as a channel in some embodiments, (iv) demodulating the down-converted and band-limited signal, (v) performing error correction, and (vi) demultiplexing to select the desired data packet stream. The RF section of various embodiments includes one or more elements to perform these functions, such as a frequency selector, a signal selector, a band limiter, a channel selector, a filter, a down-converter, a demodulator, an error corrector, and a demultiplexer. The RF section may include a tuner that performs various of these functions, including, for example, down-converting a received signal to a lower frequency (e.g., an intermediate frequency or near-baseband frequency) or to baseband. In a set-top box embodiment, the RF part and its relevant input processing element receive the RF signal that sends by wired (for example, cable) medium, and by filtering, down-conversion and filtering to the frequency band of expectation again to perform frequency selection.Various embodiments rearrange the order of above-mentioned (and other) element, remove some in these elements, and / or add other element that performs similar or different functions.Adding element can be included in and inserts element between existing element, for example inserts amplifier and analog to digital converter.In various embodiments, the RF part comprises antenna.

[0118] In addition, the USB and / or HDMI terminals may include corresponding interface processors for connecting the system 1000 to other electronic devices via USB and / or HDMI connections. It should be understood that various aspects of input processing, such as Reed-Solomon error correction, may be implemented as needed, for example, within a separate input processing IC or within the processor 1010. Similarly, various aspects of USB or HDMI interface processing may be implemented as needed within a separate interface IC or within the processor 1010. The demodulated, error-corrected, and demultiplexed streams are provided to various processing elements, including, for example, the processor 1010 and the encoder / decoder 1030, which operate in conjunction with memory and storage elements to process the data streams as needed for presentation on an output device.

[0119] The various components of system 1000 can be disposed within an integrated housing. Within the integrated housing, the various components can interconnect and transmit data between them using a suitable connection arrangement 1140, such as an internal bus known in the art, including an inter-IC (I2C) bus, wiring, and a printed circuit board.

[0120] System 1000 includes a communication interface 1050 that enables communication with other devices via a communication channel 1060. Communication interface 1050 may include, but is not limited to, a transceiver configured to send and receive data through communication channel 1060. Communication interface 1050 may include, but is not limited to, a modem or a network card, and communication channel 1060 may be implemented, for example, within a wired and / or wireless medium.

[0121] In various embodiments, a wireless network, such as a Wi-Fi network, such as IEEE 802.11 (IEEE refers to the Institute of Electrical and Electronics Engineers), is used to stream data or otherwise provide it to the system 1000. The Wi-Fi signals of these embodiments are received through a communication channel 1060 and a communication interface 1050 suitable for Wi-Fi communication. The communication channel 1060 of these embodiments is typically connected to an access point or router, which provides access to external networks including the Internet to allow streaming applications and other over-the-top communications. Other embodiments use a set-top box that transmits data via an HDMI connection of an input block 1130 to provide streaming data to the system 1000. Still other embodiments use an RF connection of an input block 1130 to provide streaming data to the system 1000. As described above, various embodiments provide data in a non-streaming manner. In addition, various embodiments use a wireless network other than Wi-Fi, such as a cellular network or a Bluetooth network.

[0122] System 1000 can provide output signals to various output devices, including a display 1100, speakers 1110, and other peripheral devices 1120. The display 1100 of various embodiments includes, for example, one or more of a touch screen display, an organic light emitting diode (OLED) display, a curved display, and / or a foldable display. Display 1100 can be used for a television, a tablet computer, a laptop computer, a cellular phone (mobile phone), or other devices. Display 1100 can also be integrated with other components (e.g., as in a smartphone) or standalone (e.g., an external monitor for a laptop computer). In various examples of various embodiments, other peripheral devices 1120 include one or more of a standalone digital video disk (or digital versatile disk) (DVR, two items), a disk player, a stereo system, and / or a lighting system. Various embodiments use one or more peripheral devices 1120 that provide functions based on the output of system 1000. For example, a disk player performs the function of playing the output of system 1000.

[0123] In various embodiments, control signals are transmitted between the system 1000 and the display 1100, speaker 1110, or other peripheral devices 1120 using signaling such as AV Link, Consumer Electronics Control (CEC), or other communication protocols that enable device-to-device control with or without user intervention. Output devices can be communicatively coupled to the system 1000 via dedicated connections through respective interfaces 1070, 1080, and 1090. Alternatively, output devices can be connected to the system 1000 via communication interface 1050 using communication channel 1060. The display 1100 and speaker 1110 can be integrated into a single unit in an electronic device (e.g., a television) along with other components of the system 1000. In various embodiments, the display interface 1070 includes a display driver, such as a timing controller (Tcon) chip.

[0124] For example, if the RF portion of input 1130 is part of a separate set-top box, the display 1100 and speaker 1110 may alternatively be separate from one or more of the other components. In various embodiments where the display 1100 and speaker 1110 are external components, the output signal may be provided via a dedicated output connection, including, for example, an HDMI port, a USB port, or a COMP output.

[0125] These embodiments may be implemented by computer software implemented by the processor 1010 or a combination of hardware and software. As a non-limiting example, embodiments may be implemented by one or more integrated circuits. The memory 1020 may be of any type suitable for the technical environment and may be implemented using any appropriate data storage technology, such as, as non-limiting examples, optical memory devices, magnetic memory devices, semiconductor-based memory devices, fixed memory, and removable memory. The processor 1010 may be of any type suitable for the technical environment and may include, as non-limiting examples, one or more of a microprocessor, a general-purpose computer, a special-purpose computer, and a processor based on a multi-core architecture.

[0126] Various implementations involve decoding. As used herein, "decoding" may include, for example, all or part of the processing performed on a received coded sequence to produce a final output suitable for display. In various embodiments, such processes include one or more of the processes typically performed by a decoder, such as entropy decoding, inverse quantization, inverse transform, and differential decoding. In various embodiments, such processes also or alternatively include processes performed by the decoder of the various implementations described herein, such as filtering an image block using at least two jointly trained neural networks.

[0127] As a further example, in one embodiment, "decoding" refers only to entropy decoding, in another embodiment, "decoding" refers only to differential decoding, and in another embodiment, "decoding" refers to a combination of entropy decoding and differential decoding. Whether the phrase "decoding process" is intended to refer specifically to a subset of operations or generally to a broader decoding process will be clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0128] Various implementations relate to encoding. In a manner similar to the discussion above regarding "decoding", "encoding" as used in this application can include, for example, all or part of a process performed on an input video sequence to produce an encoded bitstream. In various embodiments, such processes include one or more processes typically performed by an encoder, such as segmentation, differential encoding, transforms, quantization, and entropy encoding. In various embodiments, such processes also or alternatively include processes performed by the encoder of the various implementations described in this application, such as filtering an image block using at least two neural networks that are jointly trained.

[0129] As a further example, in one embodiment, "encoding" refers only to entropy encoding, in another embodiment, "encoding" refers only to differential encoding, and in another embodiment, "encoding" refers to a combination of differential encoding and entropy encoding. Whether the phrase "encoding process" is intended to refer specifically to a subset of operations or generally to a broader encoding process will become clear based on the context of the specific description and is believed to be well understood by those skilled in the art.

[0130] Note that the syntax elements used herein, such as coefficients of a neural network, are descriptive terms and therefore do not preclude the use of other syntax element names.

[0131] When a figure is presented as a flow chart, it should be understood that it also provides a block diagram of the corresponding apparatus. Similarly, when a figure is presented as a block diagram, it should be understood that it also provides a flow chart of the corresponding method / process.

[0132] Various embodiments relate to rate-distortion optimization. In particular, during the encoding process, a balance or trade-off between rate and distortion is often considered, often given a constraint on computational complexity. Rate-distortion optimization is typically formulated as minimizing a rate-distortion function, which is a weighted sum of rate and distortion. There are different approaches to solving the rate-distortion optimization problem. For example, these approaches can be based on extensive testing of all coding options, including all considered modes or coding parameter values, with a complete evaluation of their coding costs and associated distortion in the reconstructed signal after encoding and decoding. Faster approaches can also be used to save coding complexity, particularly by computing approximate distortion based on predictions or prediction residuals rather than the reconstructed signal. A hybrid of these two approaches can also be used, for example by using approximate distortion for only some possible coding options and full distortion for others. Other approaches evaluate only a subset of possible coding options. More generally, many approaches employ any of a variety of techniques to perform optimization, but optimization does not necessarily require a complete evaluation of both coding costs and associated distortion.

[0133] The implementations and aspects described herein can be implemented in, for example, a method or process, a device, a software program, a data stream, or a signal. Even if discussed only in the context of a single form of implementation (e.g., discussed only as a method), the implementation of the features discussed can also be implemented in other forms (e.g., a device or program). For example, an apparatus can be implemented with appropriate hardware, software, and firmware. The method can be implemented in, for example, a processor, which generally refers to a processing device, including, for example, a computer, a microprocessor, an integrated circuit, or a programmable logic device. The processor also includes communication equipment, such as a computer, a cellular phone, a portable / personal digital assistant ("PDA"), and other equipment that facilitates information communication between end users.

[0134] Reference to "one embodiment" or "an embodiment" or "one implementation" or "an embodiment" and other variations thereof means that a particular feature, structure, characteristic, etc. described in connection with the embodiment is included in at least one embodiment. Thus, the appearances of the phrases "in one embodiment" or "in an embodiment" or "in one implementation" or "in an implementation" and any other variations thereof in various places throughout this application are not necessarily all referring to the same embodiment.

[0135] Additionally, this application may refer to “determining” various information. Determining information may include, for example, one or more of estimating information, calculating information, predicting information, or retrieving information from a memory.

[0136] Furthermore, the present application may refer to "accessing" various information. Accessing information may include, for example, one or more of receiving information, retrieving information (e.g., from a memory), storing information, moving information, copying information, calculating information, determining information, predicting information, or estimating information.

[0137] Additionally, this application may refer to "receiving" various information. Like "accessing," receiving is intended to be a broad term. Receiving information can include, for example, one or more of accessing information or retrieving information (e.g., from a memory device). Furthermore, during operations such as storing information, processing information, sending information, moving information, copying information, erasing information, calculating information, determining information, predicting information, or estimating information, "receiving" is often involved in one way or another.

[0138] It should be understood that, for example, in the case of "A / B," "A and / or B," and "at least one of A and B," use of any of the following " / ," "and / or," and "at least one of" is intended to encompass selection of only the first-listed option (A), or only the second-listed option (B), or both options (A and B). As a further example, in the case of "A, B, and / or C" and "at least one of A, B, and C," such wording is intended to include selection of only the first-listed option (A), or only the second-listed option (B), or only the third-listed option (C), or only the first and second-listed options (A and B), or only the first and third-listed options (A and C), or only the second and third-listed options (B and C), or all three options (A, B, and C). This can be extended to multiple items listed, as will be apparent to one of ordinary skill in this and related arts.

[0139] Furthermore, as used herein, the word "signal" specifically refers to indicating something to a corresponding decoder. For example, in some embodiments, an encoder signals a specific one of the coefficients of a neural network. Thus, in one embodiment, the same parameters are used on both the encoder and decoder sides. Thus, for example, an encoder can send (explicit signaling) specific parameters to a decoder so that the decoder can use the same specific parameters. Conversely, if the decoder already has the specific parameters along with other parameters, signaling can be used without sending them (implicit signaling) to simply allow the decoder to know and select the specific parameters. By avoiding the transmission of any actual functionality, bit savings are achieved in various embodiments. It should be understood that signaling can be implemented in various ways. For example, in various embodiments, one or more syntax elements, flags, etc. are used to signal information to a corresponding decoder. Although the foregoing relates to the verb form of the word "signal," the word "signal" may also be used herein as a noun.

[0140] As will be apparent to one of ordinary skill in the art, implementations can generate various signals formatted to carry information that can be stored or transmitted, for example. The information can include, for example, instructions for performing a method, or data generated by one of the described implementations. For example, a signal can be formatted to carry a bitstream of the described embodiments. Such a signal can be formatted as, for example, an electromagnetic wave (e.g., using the radio frequency portion of the spectrum) or a baseband signal. Formatting can include, for example, encoding a data stream and modulating a carrier wave with the encoded data stream. The information carried by the signal can be, for example, analog or digital information. As is known, the signal can be transmitted over a variety of different wired or wireless links. The signal can be stored on a processor-readable medium.

[0141] We have described a number of embodiments. Features of these embodiments may be provided individually or in any combination across the various claim categories and types. In addition, embodiments may include one or more of the following features, devices, or aspects, individually or in any combination across the various claim categories and types:

[0142] Filtering the reconstructed image patch using at least two neural networks, wherein the two neural networks are jointly trained;

[0143] • A bitstream or signal comprising one or more of the described syntax elements, or variations thereof.

[0144] • A bitstream or signal comprising syntax conveying information generated according to any of the described embodiments.

[0145] • Inserting syntax elements in the signaling that enable the decoder to adapt in-loop and / or out-of-loop filtering in a manner corresponding to that used by the encoder.

[0146] • Creating and / or transmitting and / or receiving and / or decoding a bitstream or signal comprising one or more of the described syntax elements or variations thereof.

[0147] • Creating and / or transmitting and / or receiving and / or decoding according to any of the embodiments described.

[0148] • A method, process, apparatus, medium storing instructions, medium storing data, or signal according to any described embodiment.

[0149] • A TV, set-top box, cell phone, tablet computer, or other electronic device that performs in-loop and out-of-loop filtering according to any of the described embodiments.

[0150] • A TV, set-top box, cell phone, tablet, or other electronic device that performs the in-loop and out-of-loop filtering processes according to any of the described embodiments and displays (e.g., using a monitor, screen, or other type of display) the resulting image.

[0151] A TV, set-top box, cellular phone, tablet, or other electronic device that selects (e.g., using a tuner) a channel to receive a signal including an encoded image and performs in-loop and out-of-loop filtering according to any of the described embodiments.

[0152] • A TV, set-top box, cellular phone, tablet, or other electronic device that receives a signal including an encoded image over the air (e.g., using an antenna) and performs in-loop and out-of-loop filtering according to any of the described embodiments.

[0153] According to a general aspect of at least one embodiment, a method for video decoding is presented, comprising:

[0154] accessing a first reconstructed version of an image block of the encoded video; and

[0155] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0156] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0157] Wherein, the first neural network and the second neural network are trained jointly.

[0158] According to a general aspect of at least one embodiment, an apparatus for video decoding is presented that includes one or more processors configured to:

[0159] accessing a first reconstructed version of an image block of the encoded video; and

[0160] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0161] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0162] Wherein, the first neural network and the second neural network are trained jointly.

[0163] In one embodiment, the first reconstructed version of the image patch is used as additional input to train the second neural network.

[0164] In one embodiment, the first reconstructed version of the image block is filtered by a first neural network to form a second reconstructed version of the image block to be used as a reference:

[0165] - filtering the first reconstructed version of the image block through an initial neural network to form an initial reconstructed version of the image block, the initial reconstructed version to be used as input to the first neural network and the second neural network;

[0166] Therein, the initial neural network is trained to optimize objective and perceptual filtering.

[0167] In one embodiment, an intermediate representation of the output of the first neural network is used as an input to the second neural network to train the second neural network.

[0168] According to a general aspect of at least one embodiment, a method for encoding video data is presented, comprising:

[0169] accessing a first reconstructed version of an image block of the encoded video; and

[0170] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0171] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0172] Wherein, the first neural network and the second neural network are trained jointly.

[0173] According to a general aspect of at least one embodiment, an apparatus for encoding video data is presented that includes one or more processors configured to:

[0174] accessing a first reconstructed version of an image block of the encoded video; and

[0175] filtering the first reconstructed version of the image block through a first neural network to form a second reconstructed version of the image block, the second reconstructed version to be used as a reference,

[0176] filtering the second reconstructed version of the image block through a second neural network to form a third reconstructed version of the image block to be displayed,

[0177] Wherein, the first neural network and the second neural network are trained jointly.

[0178] In one embodiment, the first reconstructed version of the image patch is used as additional input to train the second neural network.

[0179] In one embodiment, filtering the first reconstructed version of the image block by a first neural network to form a second reconstructed version of the image block to be used as a reference comprises:

[0180] - filtering the first reconstructed version of the image block through an initial neural network to form an initial reconstructed version of the image block, the initial reconstructed version to be used as input to the first neural network and the second neural network;

[0181] Therein, the initial neural network is trained to optimize objective and perceptual filtering.

[0182] In one embodiment, the first neural network is trained using a first loss function between the image patch and a second reconstructed version of the image patch.

[0183] In one embodiment, the first loss function is an L1 or L2 norm.

[0184] In one embodiment, a second neural network is trained using a second loss function between the image patch and a third reconstructed version of the image patch.

[0185] In one embodiment, a generative adversarial network is used to determine the second loss function.

[0186] In one embodiment, an intermediate representation of the output of the first neural network is used as an input to the second neural network to train the second neural network.

Claims

1. A method for video decoding, comprising: Access the first reconstructed version of the image block of the coded video ( ); as well as filtering the first reconstructed version of the image block by a first neural network (CNNa) to form a second reconstructed version of the image block , The result of concatenating the first reconstructed version of the image block and the version based on the second reconstructed version of the image block is filtered by a second neural network (CNNb) to form a third reconstructed version of the image block ( ).

2. The method according to claim 1, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

3. The method according to claim 1, wherein The version based on the second reconstructed version of the image block is the second reconstructed version of the image block, or an intermediate representation of an output of the first neural network, the output corresponding to an output of a layer before the last layer of the first neural network.

4. A method for video encoding, comprising: Accessing a first reconstructed version of an image block of a picture of the coded video ( ); as well as filtering the first reconstructed version of the image block by a first neural network (CNNa) to form a second reconstructed version of the image block , The first neural network and the second neural network (CNNb) are jointly trained using a joint loss function, and the second neural network is used to filter the result of concatenating the first reconstructed version of the image block and a version based on the second reconstructed version of the image block to generate a third reconstructed version of the image block ( ).

5. The method according to claim 4, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

6. The method according to claim 4, wherein: The joint loss function includes a first term representing a first loss function between the image block and the second reconstructed version of the image block.

7. The method according to claim 4, wherein: The joint loss function includes a second term representing a second loss function between the image block and the third reconstructed version of the image block.

8. The method according to claim 4, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

9. An apparatus for video decoding, comprising one or more processors configured to: Access the first reconstructed version of the image block of the coded video ( );as well as filtering the first reconstructed version of the image block by a first neural network (CNNa) to form a second reconstructed version of the image block , The result of concatenating the first reconstructed version of the image block and the version based on the second reconstructed version of the image block is filtered by a second neural network (CNNb) to form a third reconstructed version of the image block ( ).

10. The device according to claim 9, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

11. The device according to claim 9, wherein The version based on the second reconstructed version of the image block is the second reconstructed version of the image block, or an intermediate representation of an output of the first neural network, the output corresponding to an output of a layer before the last layer of the first neural network.

12. An apparatus for video encoding, comprising one or more processors configured to: Accessing a first reconstructed version of an image block of a picture of the coded video ( );as well as filtering the first reconstructed version of the image block by a first neural network (CNNa) to form a second reconstructed version of the image block , in, The first neural network and the second neural network (CNNb) are jointly trained using a joint loss function, and the second neural network is used to filter the result of concatenating the first reconstructed version of the image block and a version based on the second reconstructed version of the image block to generate a third reconstructed version of the image block ( ).

13. The device according to claim 12, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

14. The device according to claim 12, wherein The joint loss function includes a first term representing a first loss function between the image block and the second reconstructed version of the image block.

15. The device according to claim 12, wherein The joint loss function includes a second term representing a second loss function between the image block and the third reconstructed version of the image block.

16. The device according to claim 12, wherein The first reconstructed version of the image block ( ) is used as an additional input for training the second neural network.

17. A computer program product comprising program code instructions for implementing the method according to claim 1.

18. An information storage device storing program code instructions for implementing the method according to claim 1.

19. A computer program product comprising program code instructions for implementing the method according to claim 4.

20. An information storage device storing program code instructions for implementing the method according to claim 4.

Images