Reference image resampling method for video coding and decoding combined with loop filtering and super-resolution

Abstract

The present invention discloses a video encoding and decoding reference image resampling method combining loop filtering and super-resolution, comprising the following steps: Step 1: for a given original frame, first using a coding resolution decision module to determine whether the current frame needs to be downsampled; Step 2: performing an encoding operation according to a standard video encoding process; then performing a decoding operation on the encoded and compressed video data; Step 3: if downsampling is used during encoding, directly processing the low-resolution reconstructed frame through a loop filtering and super-resolution combined neural network module to obtain a low-resolution loop filtering output and a high-resolution frame, i.e., a super-resolution output; if downsampling is not used during encoding, only the loop filtering and super-resolution combined neural network module is used to obtain the loop filtering output. The present invention reduces the redundancy between the filtering and super-resolution network models in the image resampling method, reduces the network computational load, and improves the subjective video quality and coding efficiency.

Description

Technical Field

[0001] The present invention belongs to the technical field of image resampling, and in particular relates to a video encoding and decoding reference image resampling method combining loop filtering and super-resolution. Background Art

[0002] 4K and 8K video, with their rich detail, bring unprecedented realistic visual experiences to multimedia services. However, this also leads to an explosive growth in video data volumes, which undoubtedly places higher demands on the efficiency of video codecs. To address this challenge, the Joint Video Exploration Team (JVET) recently launched a new video coding standard—Versatile Video Coding (VVC). Compared to the previous generation High Efficiency Video Coding (HEVC) standard, VVC achieves significant bitrate savings of over 50%.

[0003] In order to flexibly adjust the spatial resolution in video sequences, VVC introduces the coding tool Reference Picture Resampling (RPR). For example, in real-time communications such as video conferencing, due to unstable network conditions, flexible adjustment of video resolution ensures smooth video transmission. When using Reference Picture Resampling (RPR) as a coding tool, the video can be encoded at the original resolution or at a lower resolution after downsampling. First, it is determined whether the current frame needs to be downsampled; then, the frame is encoded and loop filtered in sequence through Luma Mapped Chroma Scaling (LMCS), Deblocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF); if downsampling has been performed before, the filtered result is upsampled back to its original resolution. It can be seen that super-resolution processing and loop filtering will affect the performance of Reference Picture Resampling coding.

[0004] Thanks to the rapid development of deep learning, deep learning-based methods have achieved state-of-the-art results in many areas of image and video processing. Subsequently, deep learning-based loop filtering for video coding has also been widely studied, using convolutional neural networks to eliminate distortion and noise introduced by video coding. In RPR, traditional interpolation filters have long been used to perform image upsampling tasks. However, with the rapid advancement of super-resolution (SR) technology, neural networks have gradually emerged as a new type of upsampling solution. Given the stringent requirements for models and computational complexity in practical applications, most SR methods currently used in RPR are based on convolutional neural networks (CNNs). There are also some super-resolution networks based on a combination of CNNs and Transformers.

[0005] Research on CNN-based super-resolution and loop filtering has improved coding performance. In order to achieve better performance, an intuitive solution is to directly integrate the two, but this is not the best choice.

[0006] An existing image resampling method based on a super-resolution neural network (Publication No.: CN114025168A) downsamples the current frame to reduce the image resolution before encoding. The downsampled image is then encoded, and the super-resolution network is used at the decoder to upsample the image to the original resolution. However, this method does not involve loop filtering, and does not combine super-resolution and loop filtering.

[0007] Existing neural network-based image resampling methods only consider using super-resolution networks to upsample images, without considering the joint design of loop filtering and super-resolution. Separately designing loop filtering and super-resolution will lead to redundancy between network models, large computational complexity, and longer time consumption. Summary of the Invention

[0008] In order to overcome the shortcomings of the above-mentioned prior art, the purpose of the present invention is to provide a video codec reference image resampling method for combined loop filtering and super-resolution. This method uses image resampling to change the encoding resolution during encoding, and utilizes one network to simultaneously complete both loop filtering and super-resolution tasks or perform the loop filtering task separately, thereby reducing the redundancy between the two network models of filtering and super-resolution in the image resampling method, reducing the computational complexity of the network, and improving the subjective quality of the video and encoding efficiency.

[0009] In order to achieve the above object, the technical solution adopted by the present invention is:

[0010] A video codec reference image resampling method for joint loop filtering and super-resolution, comprising a joint neural network module based on loop filtering and super-resolution for simultaneously processing both loop filtering and super-resolution tasks; a coding resolution decision module for determining whether to downsample the current image before encoding;

[0011] The following steps are included:

[0012] Step 1: For a given original frame, the encoding resolution decision module is first used to determine whether downsampling is required. If so, it is downsampled to obtain a low-resolution image. Otherwise, it goes directly to step 2.

[0013] Step 2: Encode the image obtained in step 1 according to a standard video coding process (such as VVC, etc.); then decode the encoded and compressed video data;

[0014] Step 3: The decoded video frame has many discontinuous blocks that need to be removed using a loop filter. If downsampling is performed in step 1, the low-resolution reconstructed frame is directly processed through the loop filtering and super-resolution joint neural network module to obtain the low-resolution frame output by the loop filtering and the high-resolution frame output by the super-resolution.

[0015] If downsampling is not used in step 1, the loop filtering and super-resolution joint neural network module is used to obtain the loop filtering output.

[0016] In the strategy of the encoding resolution decision module, step 1 first downsamples the width and height of the current frame with a scale factor SF=2 before encoding, and the downsampled resolution is smaller than the source resolution. Then, the downsampled image is upsampled using a traditional interpolation filter to reconstruct a resampled image of its original resolution. Then, the peak signal-to-noise ratio (PSNR) between the original frame and the resampled image of the luminance Y and chrominance UV components is calculated, and the PSNR is used. y (ori,dwn) and PSNR uv (ori,dwn) represents the PSNR of Y and UV respectively.

[0017] The specific process of the strategy of the encoding resolution decision module in step 1 is as follows:

[0018] The input is the original frame. The data of the original frame is input into the flowchart of the encoding resolution decision strategy to determine whether the current frame needs to be downsampled. In video encoding, several consecutive frames are divided into a picture group. The variable QP of the encoding resolution decision strategy is the initial QP of the first frame of the current group of pictures (GOP). The current GOP needs to be encoded at a downsampled resolution only in three cases, including the following three cases:

[0019] 1) The initial QP is greater than a threshold thres qp ,Currently the video is encoded at a low bit rate;

[0020] 2) Initial QP is small and PSNR is high y (ori,dwn) is greater than the threshold g(QP);

[0021] 3) The initial QP value is small, when PSNR y (ori,dwn) is less than g(QP) but greater than f(QP) and PSNR uv (ori,dwn)>=thresPSNR uv When , the frames of this GOP will be downsampled;

[0022] In the strategy, f(QP) and g(QP) are functions with QP as the variable, which can be linear functions or nonlinear functions. For example, f(QP) can be defined as a linear function

[0023] f(QP)=Pre y1 -(QP-BaseQP)*0.5. (1)

[0024] g(QP) is defined as

[0025] g(QP)=Pre y2 -(QP-BaseQP)*0.5. (2)

[0026] Pre y1 and Pre y2 is a predefined value for brightness.

[0027] The encoding in step 2 is followed by decoding to obtain a reconstructed image, the reconstructed image including luminance and chrominance components, and the two components of the reconstructed image are processed using two networks respectively.

[0028] In step 3, the joint neural network module has network models for luminance and chrominance, namely, a luminance component network and a chrominance component network; each network structure includes three modules connected in series: a multi-input branch module, a shared backbone module, and a multi-output branch module;

[0029] The data obtained after decoding in step 2 is used as the network input of the brightness component network structure;

[0030] The decoded data includes the unfiltered luminance component and the reconstructed image Rec y , the brightness component prediction image Pred generated during the encoding process y , QP map, slice type and marking information flag dwn ;

[0031] When the flag value flag dwn =1, it means that downsampling is enabled for the current frame, and the reconstructed image needs to be upsampled to obtain a high-resolution reconstructed frame;

[0032] The flag value flag dwn =0, it means that the current frame is encoded at the original resolution and no upsampling operation is involved in the encoding process;

[0033] The loop filtering and super-resolution joint neural network will first start from Rec y and Pred y The key features are extracted from the image, and then these features are spliced ​​together with the QP map and slice type input information in the channel dimension, and the residual features are further extracted to complete loop filtering and super-resolution processing.

[0034] The luminance component network structure consists of three main modules: multiple input branches, shared backbone and multiple output branches;

[0035] 1) Multi-input branch: This module uses two branches, one for loop filtering and the other for super-resolution. The network structure of the two branches is not limited. The purpose of setting up two branches is to extract shallow features for loop filtering and super-resolution respectively;

[0036] Provides an instance of a multi-input branch with flags dwn = 0 is designed for loop filtering, and a convolutional layer and a PRelu layer can be used to keep the network lightweight, marked flagg dwn =1 branch is designed for super-resolution. The network structure of the branch can use 6 series-connected residual blocks (ResBlocker1s) and a short connection with a convolutional layer;

[0037] 2) Shared Backbone: The shared structure extracts common features of loop filtering and SR. The shared backbone network structure is not limited. A convolutional layer with a stride of 2, 24 residual blocks (ResBlocker1s) to extract deep features, and a sub-pixel convolutional layer (PixelShuffle) can be used in series as the shared backbone. Other network structures such as Transformer can also be used as components of the shared backbone.

[0038] 3) Multiple output branches: Multiple output branches are introduced to handle loop filtering and upsampling respectively, and are marked with flags. dwn = 0 is used for loop filtering. The network structure of the two branches is not limited. The purpose of setting two branches is to extract features for loop filtering and super-resolution separately.

[0039] Provide an instance of a multi-output branch, marked with flag dwn = 0 branch is used for loop filtering, which contains two residual blocks (ResBlocker2s) to extract specific information related to loop filtering from the shared features; then, the learned residual is directly added to the input reconstructed image (unfiltered reconstructed image Rec y ) to obtain the final loop filter output;

[0040] Similarly, flag dwn =1 branch is used for super-resolution. The branch structure uses two residual blocks (ResBlocker2s) and then uses a sub-pixel convolution layer to upsample the learned residual features to the original resolution. Finally, the residual is added to the high-resolution image RPR sampled using the traditional interpolation filter. y middle.

[0041] The network structure of the chrominance component is similar to that of the luminance component, both of which adopt multiple input branches, a shared backbone and multiple output branches. In terms of input, a low-resolution luminance reconstructed image is introduced as input in the network of the chrominance component.

[0042] Beneficial effects of the present invention:

[0043] The present invention is better than the BD-rate ( -Delta rate) is saved by 8%, which is because the present invention uses a joint neural network to simultaneously complete loop filtering and super-resolution. This joint processing brings a relatively high performance gain.

[0044] The computational complexity of the network of the present invention is smaller than that of using two networks to separately process loop filtering and super-resolution. This is because the present invention uses one network model to obtain loop filtering output and super-resolution output at the same time, avoiding the use of two network models to separately process loop filtering and super-resolution tasks. Using one network model to process both tasks reduces redundancy between models. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flow chart of the video encoding and decoding reference image resampling method combining loop filtering and super-resolution according to the present invention.

[0046] Figure 2 It is a schematic diagram of the brightness network structure of the loop filtering and super-resolution joint neural network of the present invention.

[0047] Figure 3 It is a schematic diagram of the chroma network structure of the loop filtering and super-resolution joint neural network of the present invention.

[0048] Figure 4 This is a schematic diagram of the coding resolution decision proposed by the present invention. DETAILED DESCRIPTION

[0049] The present invention will be described in further detail below with reference to the accompanying drawings.

[0050] like Figure 1 As shown, the present invention proposes a video coding and decoding reference image resampling method combining loop filtering and super-resolution;

[0051] The framework of this method includes a loop filtering and super-resolution joint neural network module and a coding resolution decision module.

[0052] The specific steps include:

[0053] Step 1: For a given original frame, the encoding resolution decision module is first used to determine whether the current frame needs to be downsampled. If downsampling is determined to be necessary, the current frame will be downsampled to a resolution smaller than the original resolution.

[0054] Step 2: If downsampling is enabled, perform encoding according to the standard video encoding process; then perform decoding on the encoded compressed video data;

[0055] Step 3: The decoded video frame has many discontinuous blocks that need to be removed using a loop filter. If downsampling is used during encoding, the low-resolution reconstructed frame is directly processed through the loop filtering and super-resolution joint neural network module to obtain a low-resolution loop filtering output and a high-resolution frame, namely the super-resolution output. If downsampling is not used during encoding, only the loop filtering and super-resolution joint neural network module is used to obtain the loop filtering output.

[0056] For frames encoded at the original resolution, the reconstructed frame is the filtered output obtained by the joint neural network.

[0057] In-loop filtering and super-resolution joint neural network module: A joint neural network based on in-loop filtering and super-resolution. The present invention takes into account that when encoding with the original resolution, only the in-loop filtering function needs to be enabled; when encoding with the downsampled low resolution, both the in-loop filtering and super-resolution functions need to be enabled. The network structure of the joint neural network is as follows: Figure 2 The model has multiple input branches to accommodate diverse needs. To further reduce computational complexity, a shared backbone network was constructed. This backbone extracts key features from the image, providing the foundation for subsequent loop filtering and super-resolution tasks. By sharing the backbone network, redundancy introduced by independently processing the two tasks is avoided, reducing the model's computational complexity. The output of the joint neural network is a structure with multiple output branches, resulting in both super-resolution and loop filtering outputs.

[0058] go through Figure 1 The unfiltered reconstructed image obtained by the encoding process is in YUV format and contains luminance components (Y) and chrominance components (UV). The luminance and chrominance components are processed using a joint neural network module. Luminance and chrominance can be processed simultaneously using a joint neural network, or they can be processed separately using two separate networks. We provide an example of separate processing: the luminance component is processed by a luminance network model, and the chrominance component is processed by another chrominance network model; each network structure contains three modules in series: a multi-input branch module, a shared backbone module, and a multi-output branch module.

[0059] Figure 2The network structure of the brightness component contains three main modules - multiple input branches, shared backbone and multiple output branches. Figure 2 The network input is Figure 1 The decoded data includes the unfiltered luminance component and the reconstructed image Rec y , the brightness component prediction image Pred generated during the encoding process y , QP map, slice type and marking information flag dwn ;

[0060] When the flag value flag dwn =1, it means that downsampling is enabled for the current frame, and the reconstructed image needs to be upsampled to obtain a high-resolution reconstructed frame;

[0061] The flag value flag dwn =0, it means that the current frame is encoded at the original resolution and no upsampling operation is involved in the encoding process;

[0062] The loop filtering and super-resolution joint neural network will first start from Rec y and Pred y The key features are extracted from the image, and then these features are concatenated with the QP map and the slice type input information in the channel dimension as follows: Figure 2 As shown in the concatenate icon, the residual features are further extracted to complete loop filtering and super-resolution processing.

[0063] 1) Multi-input branch: This module uses two branches, one for loop filtering and the other for super-resolution. The network structure of the two branches is not limited. The purpose of setting up two branches is to extract shallow features for loop filtering and super-resolution respectively;

[0064] Provides an instance of a multi-input branch with flags dwn = 0 is designed for loop filtering, which simply uses one convolutional layer and one PRelu layer to keep the network lightweight. Considering that SR is more complex than loop filtering and requires more powerful feature extraction, the flag is marked. dwn =1 branch is designed for super-resolution, introducing 6 series-connected residual blocks (ResBlocker1s) and a short connection with a convolutional layer such as Figure 2 The specific network structure in the branch is not limited to convolutional neural networks.

[0065] 2) Shared Backbone: The purpose of the shared structure is to extract common features of loop filtering and SR. The shared backbone network structure is not limited to a specific one. A convolutional layer with a stride of 2, 24 residual blocks (ResBlocker1s) to extract deep features, and a sub-pixel convolutional layer (PixelShuffle) can be used in series as the shared backbone. Other network structures such as Transformer can also be used as components of the shared backbone.

[0066] 3) Multiple output branches: Multiple output branches are introduced to handle loop filtering and upsampling respectively, and are marked with flags. dwn = 0 is used for loop filtering, which is used to extract specific information related to loop filtering from the shared features; then, the learned residual is directly added to the input reconstructed image (the unfiltered reconstructed image Rec y ) to obtain the final loop filter output;

[0067] Flag dwn =1 branch is used to learn the residual features and upsample the learned features to the original resolution. Finally, the residual is added to the high-resolution image RPR upsampled using the traditional interpolation filter. y middle.

[0068] The structure of the network model of the chrominance component is basically the same as that of the luminance component, with text descriptions for the differences.

[0069] Figure 3 The network structure of the chrominance component is Figure 2 The networks for the luma component are similar, both employing a shared backbone and multiple output branches. However, there are some key differences in their inputs. Since chroma components are correlated with luma components, this cross-component correlation is highly beneficial for super-resolution processing of chroma components. Therefore, in addition to conventional inputs such as prediction values ​​and QP maps, the chroma network also incorporates a low-resolution luma reconstructed image as input.

[0070] Coding resolution decision module: The coding resolution decision takes into account luma and chroma components, as well as the quantization parameter (QP). The decision prioritizes the quality of the luma component. This is because the luma Y component has a more significant impact on visual quality and is therefore given greater weight in the decision process. This difference gives the decision a greater advantage in improving visual quality.

[0071] In this strategy, the width and height of the current frame are first downsampled with a scale factor SF=2 before encoding, and then the downsampled image is upsampled using a traditional interpolation filter to reconstruct a resampled image of its original resolution. Subsequently, the peak signal-to-noise ratio (PSNR) between the original frame and the resampled image of the luminance Y and chrominance UV components is calculated. y (ori,dwn) and PSNR uv (ori,dwn) represents the PSNR of Y and UV respectively.

[0072] Figure 4 This is a flowchart of the coding resolution decision strategy. The input of the flowchart is the original frame. The data of the original frame is input into the flowchart of the coding resolution decision strategy to determine whether the current frame needs to be downsampled. The QP is the initial QP of the first frame of the current group of pictures (GOP). There are only three cases in which the current GOP needs to be encoded at a downsampled resolution of SF=2.0, including the following three cases:

[0073] 1) The initial QP is greater than a threshold thres qp , which means the current video is encoded at a low bitrate.

[0074] 2) The initial QP is small and the PSNR y (ori,dwn) is greater than the threshold g(QP). In this case, it is inferred that downsampling coding has limited impact on the quality of the image Y component. Downsampling coding can further reduce coding complexity while maintaining a certain visual quality.

[0075] 3) The initial QP value is small. The quality of the Y component obtained after downsampling encoding is not very good, but not too bad, and the quality of the UV component obtained after downsampling encoding is relatively good. This situation means that when PSNR y (ori,dwn) is less than g(QP) but greater than f(QP) and PSNR uv (ori,dwn)>=thresPSNR uv , the frames of this GOP will be downsampled.

[0076] In the strategy, f(QP) and g(QP) are functions with QP as the variable, which can be linear functions or nonlinear functions. Provide an example of a linear function f(QP) defined as

[0077] f(QP)=Pre y1 -(QP-BaseQP)*0.5. (1)

[0078] g(QP) is defined as

[0079] g(QP)=Pre y2-(QP-BaseQP)*0.5. (2)

[0080] Pre y1 and Pre y2 is a predefined value for brightness.

[0081] The purpose of encoding resolution decision is to select images suitable for downsampling encoding and then restoring them to their original resolution using super-resolution during decoding. Some images lose significant information after downsampling, making them difficult to restore using a super-resolution network. The effect of encoding resolution decision is to select appropriate frames for downsampling encoding, resulting in approximately 4% higher BD-rate performance for the luminance component compared to images without encoding resolution decision.

[0082] The key points and protection points of the video codec reference image resampling method for joint loop filtering and super-resolution are:

[0083] The present invention provides an image resampling framework that combines loop filtering and super-resolution. The framework includes a joint neural network module based on loop filtering and super-resolution for simultaneously processing loop filtering tasks and super-resolution tasks; the framework also includes a coding resolution decision module for determining whether the current image should be downsampled before being encoded.

[0084] The present invention provides an example of a joint neural network based on loop filtering and super-resolution, which can simultaneously process loop filtering and super-resolution tasks or only process loop filtering tasks according to the encoding resolution. This kind of neural network that dynamically adjusts the output according to the encoding function needs has not been used in previous image resampling methods.

[0085] For the application of image resampling, structural changes and adjustments have been made to the loop filtering and super-resolution joint neural network, which has multiple input branches, a shared trunk and multiple output branches to cope with both source resolution encoding and reduced resolution encoding in image resampling.

[0086] An example of coding resolution decision taking into account luma, chroma and quantization parameters is presented.

[0087] The present invention provides a reference image resampling method for video encoding and decoding that combines loop filtering and super-resolution, a method that has not been previously studied. The joint loop filtering and super-resolution network module in the present invention can be replaced with other network structures (such as generative adversarial networks (GANs) and transformers). The coding resolution decision module in the present invention can also use other methods (such as rate-distortion optimization) to determine the appropriate coding resolution.

Claims

1. A video codec reference image resampling method for joint loop filtering and super-resolution, characterized in that: The following steps are included: Step 1: For a given original frame, the encoding resolution decision module is first used to determine whether downsampling is required. If so, it is downsampled to obtain a low-resolution image. Otherwise, it goes directly to step 2. Step 2: Encode the image obtained in step 1 according to the standard process of video encoding; then decode the encoded compressed video data; Step 3: If downsampling is performed in step 1, the low-resolution reconstructed frame is directly processed through the loop filtering and super-resolution joint neural network module to obtain the low-resolution frame output by the loop filtering and the high-resolution frame output by the super-resolution; If downsampling is not used in step 1, the loop filtering output is obtained by using the loop filtering and super-resolution joint neural network module; In step 3, the luminance component network structure and the chrominance component network structure of the joint neural network module include three modules connected in series: a multi-input branch module, a shared backbone module, and a multi-output branch module; The data obtained after decoding in step 2 is used as the network input of the brightness component network structure; The decoded data includes the unfiltered luminance component and the reconstructed image Rec y , the brightness component prediction image Pred generated during the encoding process y , QP picture type and marking information flag dwn ; When the flag value flag dwn =1, it means that downsampling is enabled for the current frame, and the reconstructed image needs to be upsampled to obtain a high-resolution reconstructed frame; The flag value flag dwn =0, it means that the current frame is encoded at the original resolution and no upsampling operation is involved in the encoding process; The loop filtering and super-resolution joint neural network will first start from Rec y and Pred y The key features are extracted from the image, and then these features are spliced ​​with the QP picture type input information in the channel dimension, and the residual features are further extracted to complete the loop filtering and super-resolution processing.

2. The method for joint loop filtering and super-resolution video coding and decoding reference image resampling according to claim 1, characterized in that: In the strategy of the encoding resolution decision module, step 1 first downsamples the width and height of the current frame before encoding, and the downsampled image is smaller than the source resolution. Then, the downsampled image is upsampled by an interpolation filter to reconstruct a resampled image of its original resolution. Then, the peak signal-to-noise ratio (PSNR) between the original frame and the resampled image of the luminance Y and chrominance UV components is calculated. The PSNR is used. y (ori,dwn) and PSNR uv (ori,dwn) represents the PSNR of Y and UV respectively.

3. The method for joint loop filtering and super-resolution video coding and decoding reference image resampling according to claim 2, characterized in that: The specific process of the strategy of the encoding resolution decision module in step 1 is as follows: The input is the original frame. In video encoding, several consecutive frames are divided into a group of pictures. The variable QP of the encoding resolution decision strategy is the initial QP of the first frame of the current group of pictures (GOP). There are only three cases in which the current GOP needs to be encoded at a downsampled resolution, including the following three cases: 1) The initial QP is greater than a threshold thres qp ,Currently the video is encoded at a low bit rate; 2) The initial QP is less than or equal to the threshold thres qp , and PSNR y (ori,dwn) is greater than the threshold g(QP); 3) The initial QP is less than or equal to the threshold thres qp , when PSNR y (ori,dwn) is less than g(QP) but greater than f(QP) and PSNR uv (ori,dwn)>=thresPSNR uv When , the frames of this GOP will be downsampled; In the strategy, f(QP) and g(QP) are functions with QP as the variable, which can be linear functions or nonlinear functions.

4. The method for joint loop filtering and super-resolution video coding and decoding reference image resampling according to claim 3, characterized in that: The encoding in step 2 is followed by decoding to obtain a reconstructed image, the reconstructed image including luminance and chrominance components, which are processed using a joint neural network module.

5. The method for joint loop filtering and super-resolution video coding and decoding reference image resampling according to claim 1, characterized in that: The luminance component network structure includes three main modules: multi-input branches, shared backbones, and multi-output branches. The three main modules are specifically implemented using convolutional neural networks or Transformers, providing an implementation method of a joint neural network based on convolutional neural networks. 1) Multiple input branches: Two branches are used in this module, one for loop filtering and the other for super-resolution; With flag dwn = 0 is designed for loop filtering, marked flag dwn =1 branch is designed for super-resolution. The network structure of the two branches is not limited. Convolutional neural network or Transformer is used, and two branches are set to extract shallow features for loop filtering and super-resolution respectively; 2) Shared backbone: The shared structure is used to extract common features of loop filtering and SR. The shared backbone network structure is not limited. It uses a convolutional layer with a stride of 2, 24 residual blocks (ResBlocker1s) to extract deep features, and a sub-pixel convolutional layer (PixelShuffle) in series as the shared backbone, or uses Transformer as a component of the shared backbone. 3) Multiple output branches: Multiple output branches are introduced to handle loop filtering and upsampling respectively, and are marked with flags. dwn = 0 is used for loop filtering, which is used to extract specific information related to loop filtering from the shared features; then, the learned residual is directly added to the input unfiltered reconstructed image Rec y To obtain the final loop filter output; Flag dwn =1 branch is used to learn the residual features and upsample the learned features to the original resolution. Finally, the residual is added to the high-resolution image RPR upsampled using the traditional interpolation filter. y middle.

6. The method for joint loop filtering and super-resolution video coding and decoding reference image resampling according to claim 5, characterized in that: The network structure of the chrominance component and the network structure of the luminance component both adopt multiple input branches, a shared backbone and multiple output branches. In terms of input, a low-resolution luminance reconstructed image is introduced as input in the network of the chrominance component.

Images