Video decoding method and device

Abstract

This application provides a video decoding method and apparatus, relating to the field of video encoding and decoding technology. It includes: acquiring encoded data of a target video frame; decoding the encoded data of the target video frame to obtain a reconstructed video frame; acquiring a filtering strategy, wherein the filtering strategy is a binary vector with elements of a target number, the target number being the number of filtering modules in an initial loop filter network; adjusting the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain a target loop filter network; and performing loop filtering on the reconstructed video frame through the target loop filter network to obtain an enhanced video frame corresponding to the reconstructed video frame. Some embodiments of this application are used to achieve loop filtering with higher versatility and flexibility, thereby adapting to the complexity requirements of loop filters in different application scenarios.

Description

Technical Field

[0001] Some embodiments of this application relate to the field of video encoding and decoding technology. More specifically, they relate to a video decoding method and apparatus. Background Technology

[0002] During video encoding, video frames undergo operations such as prediction, transformation, and quantization. These operations introduce various distortions, such as blockiness and ringing. Loop filters can filter the reconstructed image, improve image quality, reduce blockiness, reduce ringing, and enhance the overall quality of the video. Therefore, loop filters are widely used in video encoders and video decoders.

[0003] With the development of deep neural network technology, many deep neural network filtering techniques have been applied to loop filtering tasks in video coding. Currently, loop filtering tasks typically use fixed-structure neural networks to process all video frames. However, different video frames exhibit different levels of degradation under different quantization conditions, requiring the use of different loop filters. Furthermore, different application scenarios have varying requirements for the performance and complexity of loop filters. Therefore, using fixed-structure deep neural network models for loop filtering has limitations in terms of versatility and flexibility. Summary of the Invention

[0004] This application provides an exemplary embodiment of a video decoding method, a video decoding apparatus, and a loop filter that enables greater versatility and flexibility, thereby adapting to the complexity requirements of loop filters in different application scenarios.

[0005] The technical solutions provided by some embodiments of this application are as follows:

[0006] Firstly, some embodiments of this application provide a video decoding method, including:

[0007] Obtain the encoded data of the target video frame;

[0008] Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame;

[0009] Obtain a filtering strategy, wherein the filtering strategy is a binary vector whose elements are the target number, and the target number is the number of filtering modules in the initial loop filtering network;

[0010] The number of filtering modules in the initial loop filter network is adjusted based on the filtering strategy to obtain the target loop filter network.

[0011] The reconstructed video frame is subjected to loop filtering by the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

[0012] Secondly, some embodiments of this application provide a video decoding apparatus, including:

[0013] The acquisition unit is used to acquire the encoded data of the target video frame;

[0014] A decoding unit is used to decode the encoded data of the target video frame and obtain the reconstructed video frame of the target video frame;

[0015] The processing unit obtains a filtering strategy, wherein the filtering strategy is a binary vector whose elements are a target number, and the target number is the number of filtering modules in the initial loop filtering network;

[0016] An adjustment unit is used to adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network.

[0017] The filtering unit is used to perform loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

[0018] Thirdly, some embodiments of this application provide an electronic device, including: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method described in the first aspect when executing the computer program.

[0019] Fourthly, some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video decoding method described in the first aspect.

[0020] Fifthly, some embodiments of this application provide a chip including a processor and a memory, the memory being used to store programs or instructions executable on the processor, and the processor being used to execute the programs or instructions to cause the video decoding method described in the first aspect to be executed.

[0021] Sixthly, some embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the video decoding method described in the first aspect.

[0022] As can be seen from the above technical solutions, the video decoding method provided in some embodiments of this application, after obtaining the encoded data of the target video frame, first decodes the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame, then obtains a filtering strategy, and adjusts the number of filtering modules of the initial loop filter network based on the filtering strategy to obtain a target loop filter network, and then performs loop filtering on the reconstructed video frame through the target loop filter network to obtain the enhanced video frame corresponding to the reconstructed video frame. Since the video decoding method provided in the embodiments of this application can obtain a filtering strategy and adjust the number of filtering modules of the initial loop filter network based on the filtering strategy to obtain a target loop filter network, and then perform loop filtering on the reconstructed video frame through the target loop filter network, the video decoding method provided in the embodiments of this application can avoid using a fixed loop filter to filter the reconstructed video frame, achieving loop filtering with higher versatility and flexibility, and can adapt to the complexity requirements of loop filters in different application scenarios. Attached Figure Description

[0023] To more clearly illustrate the implementation methods in some embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0024] Figure 1 The diagram shows a schematic representation of the structure of a video decoding system provided in some embodiments;

[0025] Figure 2 One of the flowcharts of a video decoding method provided in some embodiments is shown;

[0026] Figure 3 This diagram illustrates one of the structural schematics of a policy generation network in some embodiments;

[0027] Figure 4 Schematic diagrams of the upsampling module in some embodiments are shown;

[0028] Figure 5 The diagram shows a schematic representation of the motion compensation unit provided in some embodiments;

[0029] Figure 6 Schematic diagrams of upsampling modules provided in some embodiments are shown;

[0030] Figure 7 This is the second flowchart of the video decoding method provided in some embodiments;

[0031] Figure 8A second schematic diagram of the policy generation network in some embodiments is shown;

[0032] Figure 9 The third flowchart of the video decoding method provided in some embodiments is shown;

[0033] Figure 10 The third schematic diagram of the policy generation network in some embodiments is shown;

[0034] Figure 11 The fourth step of the video decoding method provided in some embodiments is shown;

[0035] Figure 12 The fourth schematic diagram of the policy generation network in some embodiments is shown;

[0036] Figure 13 Schematic diagrams of the residual block structure in some embodiments are shown;

[0037] Figure 14 Fifth of several schematic diagrams illustrating the structure of the policy generation network in various embodiments is shown;

[0038] Figure 15 The fifth step of a video decoding method provided in some embodiments is shown;

[0039] Figure 16 This diagram illustrates one of the policy regression modules provided in some embodiments;

[0040] Figure 17 A flowchart of the steps of a video decoding method provided in some embodiments is shown as the sixth one;

[0041] Figure 18 This is a second schematic diagram of a strategy regression module provided in some embodiments;

[0042] Figure 19 The seventh step of a video decoding method provided in some embodiments is shown;

[0043] Figure 20 A schematic diagram of one of the loop filter networks provided in some embodiments is shown;

[0044] Figure 21 The eighth step of a video decoding method provided in some embodiments is shown;

[0045] Figure 22 A second schematic diagram of a loop filter network provided in some embodiments is shown;

[0046] Figure 23 A flowchart of the steps of a video decoding method provided in some embodiments is shown as number nine;

[0047] Figure 24 A third schematic diagram of a loop filter network provided in some embodiments is shown;

[0048] Figure 25 The tenth flowchart illustrates the steps of a video decoding method provided in some embodiments;

[0049] Figure 26 A fourth schematic diagram of a loop filter network provided in some embodiments is shown;

[0050] Figure 27 The following are schematic diagrams illustrating the structure of a video decoding apparatus in some embodiments of this application;

[0051] Figure 28 A schematic diagram of the hardware structure of an electronic device in some embodiments of this application is shown. Detailed Implementation

[0052] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below. In order to make the purpose and implementation of this application clearer, the exemplary embodiments of this application will be clearly and completely described below with reference to the accompanying drawings of the exemplary embodiments of this application. Obviously, the exemplary embodiments described are only some embodiments of this application, and not all embodiments.

[0053] This application is not intended to limit the implementation of the invention. Unless otherwise stated, these terms should be understood in their ordinary and common sense.

[0054] The terms “comprising” and “having”, and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device that includes a range of components is not necessarily limited to all of the components that are clearly listed, but may include other components that are not clearly listed or that are inherent to such product or device.

[0055] The use of phrases such as "some implementations" or "some embodiments" in the specification indicates that the described implementations or embodiments may include specific features, structures, or characteristics, but not every embodiment may necessarily include that specific feature, structure, or characteristic. Furthermore, such phrases do not necessarily refer to the same implementation. Additionally, when describing a specific feature, structure, or characteristic in connection with an embodiment, it is considered that implementing that feature, structure, or characteristic in connection with other implementations (whether explicitly described herein or not) is within the knowledge of those skilled in the art.

[0056] The following describes the video decoding system in some embodiments of this application.

[0057] Reference Figure 1As shown, in some embodiments, the video decoding system includes an encoder 11, used to perform lossy encoding on the original video frame X to obtain the encoded data of the original video frame X. For example, the encoder 11 can use video coding standards such as High-Efficiency Video Coding (HEVC / H.265) and Versatile Video Coding (VCC) to perform lossy encoding on the original video frame X. During the lossy encoding process of the encoder 11 on the original video frame X, the encoder 11 will transmit the data to the reconstructed video frame. Compression noise is introduced during encoding, resulting in a degrade in video frame quality. The reconstructed video frames are compressed by the encoder. It can be represented as:

[0058]

[0059] Here, Enc() represents the lossy encoding operation of encoder 11, and q represents the quantization parameter (QP) configured with the encoder.

[0060] Furthermore, as shown in formula (1) above, the video frame content itself and the quantization parameter are two factors affecting the degradation level of the video frame. During the encoding process, the quantization parameter directly determines the quality of the reconstructed video frame. A larger quantization parameter will greatly compress the value space of the discrete signal and guide the encoder to quantize more transform coefficients to 0, resulting in severe loss of high-frequency details in the reconstructed video frame. Compared with simple or flat image content, the image degradation phenomenon after quantization of complex image content with richer high-frequency details may be more severe. Considering both image quality and complexity, the filtering intensity of the loop filtering operation performed on images with different degrees of degradation should also be different.

[0061] Reference Figure 1 As shown, in some embodiments, the video decoding system includes a loop filter 12. The loop filter 12 consists of a policy generation network 1201 and a loop filtering network 1202. The policy generation network 1201 generates content-aware filtering policies for the reconstructed video frames, and the loop filtering network 1202 generates enhanced video frames corresponding to the reconstructed video frames under the guidance of the filtering policies. The input to the policy generation network 1201 may include two factors affecting the image degradation level: the reconstructed video frames... And the quantization parameter q. The output of the policy generation network 1201 is the filtering policy D. In order to improve the efficiency of loop filtering, different intensities of filtering need to be performed for reconstructed video frames with different degradation levels. In the neural network-based loop filtering, the depth of the network model directly affects the filtering intensity and inference time. Therefore, the role of filtering policy D is to determine an appropriate inference path for the loop filtering network 1202, that is, to guide the loop filtering network 1202 to skip specific redundant filtering modules during the inference process, thereby effectively reducing computational complexity while maintaining performance. Under the guidance of filtering policy D, the loop filtering network 1202 reconstructs video frames The quantization parameter q and boundary intensity information S are used as inputs to generate reconstructed video frames. Corresponding enhanced video frames Furthermore, for simple image patches compressed with smaller quantization parameters, only low-intensity filtering is required. In this case, the policy generation network 1201 should generate sparser inference paths, guiding the loop filtering network 1202 to skip most filtering modules to significantly reduce inference time. However, for complex image patches compressed with larger quantization parameters, high-intensity filtering may be required. In this case, the policy generation network 1201 should retain most of the filtering modules of the filtering network to ensure the quality of the enhanced reconstructed image.

[0062] In practical applications, different demands may exist for codecs in terms of complexity and performance tradeoffs. For example, in latency-sensitive scenarios such as real-time streaming, the computational complexity of encoding tools needs to be strictly limited; while in video-on-demand applications with ample bandwidth, high quality is of greater concern. To address the issue of poor generalization and versatility of single neural network models across different application scenarios, a user-defined balancing factor λ can be introduced into the input of the policy network. By adjusting the size of the balancing factor λ, the policy generation network 1201 can generate more suitable inference policies for various complexity and quality requirements, and the customizable balancing factor λ also enables the loop filter to support flexible complexity adjustment.

[0063] Based on the above, the process of policy generation network 1201 generating filtering policy D and loop filtering network 1202 generating enhanced video frames are described. The process can be expressed as equations (2) and (3) respectively:

[0064]

[0065] Where P() represents the policy generation network 1201, and F() represents the loop filter network.

[0066] This application also provides a video decoding method, referring to... Figure 2 As shown, the video decoding method includes the following steps S21 to S25:

[0067] S21. Obtain the encoded data of the target video frame.

[0068] In some embodiments, the video decoding method provided in this application is a video encoding method, wherein obtaining the encoded data of the target video frame includes: encoding the target video frame to obtain the encoded data of the target video frame.

[0069] In other embodiments, the video decoding method provided in this application is a video decoding method, wherein obtaining the encoded data of the target video frame includes: receiving the encoded data of the target video frame sent by a media resource server.

[0070] S22. Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame.

[0071] During video encoding, the target video frame may also serve as a reference frame for other subsequent video frames. Therefore, after obtaining the encoded data of the target video frame, it is necessary to further decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame.

[0072] S23. Obtain the filtering strategy.

[0073] The filtering strategy is a binary vector whose elements are the target number, where the target number is the number of filtering modules in the initial loop filtering network.

[0074] In this embodiment, a binary vector refers to a vector in which each element is either a first preset value or a second preset value. For example, a binary vector is a vector in which each element is either 0 or 1.

[0075] Since the filtering strategy is a binary vector with elements representing the target number, and the target number is the number of filtering modules in the initial loop filter network, the number of elements in the filtering strategy is the same as the number of filtering modules in the initial loop filter network, both being the target number.

[0076] S24. Adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network.

[0077] In some embodiments, adjusting the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain a target loop filter network includes: if the value of the i-th element of the filtering strategy is a first preset value, then enabling the i-th filtering module of the initial loop filter network; if the value of the i-th element of the filtering strategy is a second preset value, then bypassing the i-th filtering module of the initial loop filter network; wherein i is a positive integer less than or equal to the target number.

[0078] For example, refer to Figure 3 As shown, Figure 3 Taking an initial loop filter network 31 containing 10 filter modules as an example, if the filtering strategy is 1010100110, then after adjusting the number of filter modules in the initial loop filter network 31 based on the filtering strategy, the first, third, fifth, eighth, and ninth filter modules of the initial loop filter network 31 are enabled, while the other filter modules are bypassed. Therefore, the target loop filter network 32 contains only 5 filter modules.

[0079] S25. Perform loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

[0080] The video decoding method provided in some embodiments of this application, after obtaining the encoded data of the target video frame, first decodes the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame. Then, it obtains a filtering strategy and adjusts the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain a target loop filter network. The reconstructed video frame is then loop-filtered using the target loop filter network to obtain the enhanced video frame corresponding to the reconstructed video frame. Since the video decoding method provided in this application can obtain a filtering strategy and adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain a target loop filter network, and then loop-filter the reconstructed video frame using the target loop filter network, the video decoding method provided in this application can avoid using a fixed loop filter to filter the reconstructed video frame, achieving loop filtering with higher versatility and flexibility, and can adapt to the complexity requirements of loop filters in different application scenarios.

[0081] As an extension and refinement of the above embodiments, this application provides another video decoding method, referring to... Figure 4 As shown, the video decoding method includes the following steps S401 to S410:

[0082] S401. Obtain the encoded data of the target video frame.

[0083] S402. Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame.

[0084] S403. Upsample the chroma components of the reconstructed video frame to obtain the reconstructed upsampled image corresponding to the reconstructed video frame.

[0085] In this case, the chromaticity component and the luminance component of the reconstructed upsampled image have the same size.

[0086] In some embodiments, upsampling the chroma components of the reconstructed video frame includes: upsampling the chroma components of the reconstructed video frame using a nearest neighbor interpolation method.

[0087] In some embodiments, upsampling the chroma components of the reconstructed video frame to obtain a reconstructed upsampled image corresponding to the reconstructed video frame includes converting the reconstructed video frame from a YUV420 format image to a YUV444 format image.

[0088] Typically, reconstructed video frames are in YUV420 format, where Y represents luminance, i.e., the grayscale information of the image; U and V represent chrominance, used to describe the hue and saturation of colors. In the YUV420 format, every four Y components (luminance components) share one U component and one V component. Therefore, in both the horizontal and vertical directions, the sampling rate of the chrominance components (U and V) is half that of the luminance (Y) sampling rate. In contrast, the YUV444 format image has a 1:1:1 sampling ratio for the Y, U, and V components.

[0089] In some embodiments, obtaining the filtering strategy includes: obtaining the filtering strategy based on a policy generation network, referring to... Figure 5 As shown, the policy generation network includes: an upsampling module 51, used for upsampling the reconstructed video frames. chromaticity components Upsampling is performed to obtain the reconstructed video frames. Corresponding reconstruction upsampling Figure X ′.

[0090] Reference Figure 6 As shown, in some embodiments, the upsampling module 51 includes: an upsampling layer 511 and a stitching layer 512, wherein the upsampling layer 511 is used to perform upsampling on the reconstructed video frames respectively. chromaticity components Upsampling is performed, and the stitching layer 512 is used to stitch together the upsampled chromaticity components. and reconstructing video frames luminance component And output the reconstructed video frames. Corresponding reconstruction upsampling Figure X ′.

[0091] S404. Perform a convolution operation on the reconstructed upsampled image to obtain the reconstruction matrix.

[0092] Reference Figure 5As shown, the policy generation network further includes a convolutional layer 52 for upsampling the reconstruction. Figure X Perform convolution operations to obtain the reconstruction matrix M. X′ .

[0093] In some embodiments, the convolutional layer 52 has a kernel size of 3*3, 3 input channels, and 64 output channels.

[0094] S405. Generate a quantization parameter map and a balance factor map based on the quantization parameters corresponding to the target video frame and the pre-configured balance factor, respectively.

[0095] The resolution of the quantization parameter map and the resolution of the balance factor map are both the same as the resolution of the reconstructed video frame.

[0096] In some embodiments, generating a quantization parameter map and a balance factor map based on the quantization parameters corresponding to the target video frame and a pre-configured balance factor respectively includes: filling the quantization parameters and balance factor with the pixel values ​​of each pixel of the reconstructed video frame respectively, thereby obtaining the quantization parameter map and the balance factor map.

[0097] S406. Perform convolution operations on the quantization parameter map and the balance factor map respectively to obtain the quantization parameter matrix and the balance factor matrix.

[0098] Reference Figure 5 As shown, the policy generation network further includes a convolutional layer 53 and a convolutional layer 54. The convolutional layer 53 is used to perform a convolution operation on the quantization parameter map Q to obtain the quantization parameter matrix M. Q Convolutional layer 54 is used to perform a convolution operation on the balance factor matrix graph Λ to obtain the balance factor matrix M. Λ .

[0099] In some embodiments, the kernel size of both convolutional layer 53 and convolutional layer 54 is 3*3, the number of input channels is 1, and the number of output channels is 16.

[0100] S407. Concatenate the reconstruction matrix, the quantization parameter matrix, and the balance factor matrix along the channel dimension to obtain the initial strategy features.

[0101] Reference Figure 5 As shown, the policy generation network further includes: a stitching module 55, used to stitch the reconstruction matrix M along the channel dimension. X′ The quantization parameter matrix M Q and the balance factor matrix M Λ To obtain the initial policy features F u .

[0102] S408. Obtain the filtering strategy based on the initial strategy features.

[0103] Reference Figure 5 As shown, the policy generation network further includes: a policy generation module 56, used for initializing policy features F. u Processing is performed to obtain the filtering strategy D.

[0104] S409. Adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network.

[0105] S410. Obtain initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary strength map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame.

[0106] In some embodiments, before obtaining initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame, the method further includes: obtaining the predicted image of the target video frame and the boundary intensity map of the reconstructed video frame.

[0107] In some embodiments, obtaining a predicted image of the target video frame includes: obtaining the predicted blocks of each coded block of the target video frame, and concatenating the predicted blocks of each coded block of the target video frame to obtain a predicted image of the target video frame.

[0108] A boundary intensity map is a tool used in image processing and computer vision to represent the boundary intensity at different locations in an image. It is typically a two-dimensional matrix of the same size as the input image, where the value of each element represents the boundary intensity at the corresponding location in the image.

[0109] S411. The initial filtering features are processed by the target loop filtering network to obtain refined filtering features.

[0110] That is, the initial filtering features are input into the target loop filter network, and the refined filtering features are received from the target loop filter network.

[0111] S412. Obtain the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features.

[0112] As an extension and refinement of the above embodiments, this application provides another video decoding method, referring to... Figure 7 As shown, the video decoding method includes the following steps S701 to S712:

[0113] S701. Obtain the encoded data of the target video frame.

[0114] S702. Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame.

[0115] S703. Upsample the chroma components of the reconstructed video frame to obtain the reconstructed upsampled image corresponding to the reconstructed video frame.

[0116] In this case, the chromaticity component and the luminance component of the reconstructed upsampled image have the same size.

[0117] In some embodiments, obtaining the filtering strategy includes: obtaining the filtering strategy based on a policy generation network, referring to... Figure 8 As shown, the policy generation network includes: an upsampling module 81, used for upsampling the reconstructed video frames. chromaticity components Upsampling is performed to obtain the reconstructed video frames. Corresponding reconstruction upsampling Figure X ′.

[0118] S704. Perform a convolution operation on the reconstructed upsampled image to obtain the reconstruction matrix.

[0119] Reference Figure 8 As shown, the policy generation network further includes a convolutional layer 82 for upsampling the reconstruction. Figure X Perform convolution operations to obtain the reconstruction matrix M. X′ .

[0120] In some embodiments, the kernel size of the convolutional layer 82 is 3*3, the number of input channels is 3, and the number of output channels is 64.

[0121] S705. Generate a quantization parameter map based on the quantization parameters corresponding to the target video frame.

[0122] The resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame.

[0123] In some embodiments, generating a quantization parameter map based on the quantization parameters corresponding to the target video frame includes: filling the quantization parameters with the pixel values ​​of each pixel of the reconstructed video frame, thereby obtaining the quantization parameter map.

[0124] S706. Perform a convolution operation on the quantization parameter map to obtain the quantization parameter matrix.

[0125] Reference Figure 8 As shown, the policy generation network further includes a convolutional layer 83. The convolutional layer 83 is used to perform a convolution operation on the quantization parameter map Q to obtain the quantization parameter matrix M. Q .

[0126] In some embodiments, the convolutional layer 83 has a kernel size of 3*3, 1 input channel, and 16 output channels.

[0127] S707. Concatenate the reconstruction matrix and the quantization parameter matrix along the channel dimension to obtain the initial strategy features.

[0128] Reference Figure 8 As shown, the policy generation network further includes: a stitching module 85, used to stitch the reconstruction matrix M along the channel dimension. X′ and the quantization parameter matrix M Q To obtain the initial policy features F u .

[0129] S708. Obtain the filtering strategy based on the initial strategy features.

[0130] Reference Figure 8 As shown, the policy generation network further includes: a policy generation module 85, used for initializing policy features F. u Processing is performed to obtain the filtering strategy D.

[0131] S709. Adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network.

[0132] S710. Obtain initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame.

[0133] S711. The initial filtering features are processed by the target loop filtering network to obtain refined filtering features.

[0134] S712. Obtain the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features.

[0135] As an extension and refinement of the above embodiments, this application provides another video decoding method, referring to... Figure 9 As shown, the video decoding method includes the following steps S901 to S908:

[0136] S901. Obtain the encoded data of the target video frame.

[0137] S902. Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame.

[0138] S903. Generate a balance factor diagram based on pre-configured balance factors.

[0139] The resolution of the balance factor map is the same as the resolution of the reconstructed video frame.

[0140] S904. Perform a convolution operation on the balance factor graph to obtain the balance factor matrix.

[0141] In some embodiments, obtaining the filtering strategy includes: obtaining the filtering strategy based on a policy generation network, referring to... Figure 10 As shown, the policy generation network includes a convolutional layer 101. The convolutional layer 101 is used to perform a convolution operation on the balance factor matrix graph Λ to obtain the balance factor matrix M. A .

[0142] In some embodiments, the convolutional kernel size of the convolutional layer 101 is 3*3, the number of input channels is 1, and the number of output channels is 16.

[0143] S905. The balance factor matrix is ​​determined as the initial strategy feature.

[0144] That is, directly using the balance factor matrix M Λ As the initial strategy feature F u Proceed to the next steps.

[0145] S906. Obtain the filtering strategy based on the initial strategy features.

[0146] Reference Figure 10 As shown, the policy generation network further includes: a policy generation module 102, used for initializing policy features F. u Processing is performed to obtain the filtering strategy D.

[0147] S907. Adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network.

[0148] S908. Obtain initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary strength map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame.

[0149] In some embodiments, before obtaining initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame, the method further includes: obtaining the predicted image of the target video frame and the boundary intensity map of the reconstructed video frame.

[0150] In some embodiments, obtaining a predicted image of the target video frame includes: obtaining the predicted blocks of each coded block of the target video frame, and concatenating the predicted blocks of each coded block of the target video frame to obtain a predicted image of the target video frame.

[0151] A boundary intensity map is a tool used in image processing and computer vision to represent the boundary intensity at different locations in an image. It is typically a two-dimensional matrix of the same size as the input image, where the value of each element represents the boundary intensity at the corresponding location in the image.

[0152] S909. The initial filtering features are processed by the target loop filtering network to obtain refined filtering features.

[0153] That is, the initial filtering features are input into the target loop filter network, and the refined filtering features are received from the target loop filter network.

[0154] S910. Obtain the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features.

[0155] In some embodiments, refer to Figure 11 As shown, the implementation method of obtaining the filtering strategy based on the initial strategy features in any of the above embodiments includes the following steps:

[0156] S111. The initial policy features are processed by a residual network to obtain refined policy features.

[0157] In some embodiments, refer to Figure 12 As shown, in Figure 5 Based on the policy generation network shown, the policy generation module includes: a residual network 120, which processes the initial policy features F. u The process is performed to obtain the refinement strategy feature F. The residual network 120 includes multiple cascaded residual blocks 121. Figure 12 The residual network 120 is illustrated as an example, which includes three cascaded residual blocks 121.

[0158] In some embodiments, refer to Figure 13As shown, the residual block 121 includes: a first convolutional layer 131, a second convolutional layer 132, a third convolutional layer 133, a splicing layer 134, a fourth convolutional layer 135, a fifth convolutional layer 136, a sixth convolutional layer 137, and an adder 138. The first convolutional layer 131 and the second convolutional layer 132 are respectively used to perform convolution operations on the input In of the residual block, and the third convolutional layer 133 is used to perform a convolution operation on the output of the second convolutional layer 132. Layer 134 is used to concatenate the output of the first convolutional layer 131 and the output of the third convolutional layer 133. The fourth convolutional layer 135, the fifth convolutional layer 136, and the sixth convolutional layer 137 are connected in series. The input of the fourth convolutional layer 135 is the output of the concatenation layer 134. The input of the adder 138 is the output of the sixth convolutional layer 137 and the input In of the residual block. The output of the adder 138 is the output Out of the residual block.

[0159] S112. Generate the filtering strategy based on the features of the extraction strategy.

[0160] In some embodiments, refer to Figure 12 As shown, the policy generation module also includes a policy regression module (PRM) 122, which is used to process the extracted policy features F and output the filtering policy D.

[0161] In other embodiments, the video decoding method further includes: performing a first convolution operation and a second convolution operation on the initial policy features sequentially before processing the initial policy features through the residual network;

[0162] The first convolution operation is used to reduce the number of channels in the initial policy feature, and the second convolution operation is used to reduce the size of the initial policy feature.

[0163] That is, refer to Figure 14 As shown, in Figure 12 Based on the policy generation module shown, the policy generation module further includes a convolutional layer 141 and a convolutional layer 142. The convolutional layer 141 is used to perform a convolution operation on the initial policy features to reduce the number of channels of the initial policy features. The convolutional layer 142 is used to perform a convolution operation on the initial policy features output by the convolutional layer 141 after reducing the number of channels to further reduce the size of the initial policy features.

[0164] In some embodiments, the kernel size of convolutional layer 141 is 1*1, the input channel data is 96, and the output channel data is 16; the kernel size of convolutional layer 142 is 3*3, the input channel data is 16, and the output channel data is 16.

[0165] Before processing the initial policy features through the residual network, performing a first convolution operation and a second convolution operation on the initial policy features sequentially can reduce the number of channels and the feature size of the initial policy features, thereby reducing the computational load in the process of obtaining the filtering policy.

[0166] In some embodiments, refer to Figure 15 As shown, the implementation of generating the filtering strategy based on the extraction strategy features in any of the above embodiments includes the following steps:

[0167] S151. The features of the extraction strategy are processed through a fully connected module to obtain a probability vector.

[0168] The fully connected module includes multiple fully connected layers connected in series.

[0169] A fully connected layer is a basic layer type in neural networks. In a fully connected layer, each neuron is connected to all neurons in the previous layer. It takes the outputs of all neurons in the previous layer as input, processes them through weighted summation and activation functions, and then outputs them to the neurons in the next layer.

[0170] For example, refer to Figure 16 As shown, the strategy regression module 122 includes a fully connected module 161, which includes multiple serially connected fully connected layers 1610. Figure 16 The example shown is a fully connected module 161 comprising two fully connected layers 1610. The fully connected module 161 is used to process the extraction strategy features F to obtain a probability vector P.

[0171] S152. The probability vector is processed through an activation function layer to obtain a decision vector.

[0172] Activation function layers are a fundamental type of layer in neural networks. Located between neurons (or neural layers), they perform nonlinear transformations on the neuron's output. During the forward propagation of a neural network, after the input data undergoes a linear transformation (such as weighted summation), the activation function layer processes the result, introducing nonlinear factors. Without activation function layers, regardless of the number of layers in a neural network, its overall expressive power remains linear. Activation functions enable neural networks to fit various complex nonlinear functions, thereby enhancing their expressive and learning capabilities.

[0173] In some embodiments, the activation function of the activation function layer is the Sigmoid activation function.

[0174] Reference Figure 16As shown, the policy regression module 122 includes an activation function layer 162, which processes the probability vector P to make a decision vector P′.

[0175] S153. The decision vector is processed through an indicator function layer to obtain the filtering strategy.

[0176] The indicator function layer, also known as the indicator function layer, is a special type of function layer that is mainly used to label specific conditions in neural networks or other mathematical models.

[0177] In some implementations, the decision vector is processed by an indicator function layer, including: if a component in the decision vector is greater than 0.5, the output is 1; otherwise, the input is 0.

[0178] Reference Figure 16 As shown, the policy regression module 122 includes: an indicator function layer 163, used to process the decision vector P′ to obtain the filtering policy D.

[0179] In some embodiments, refer to Figure 17 As shown, the implementation of generating the filtering strategy based on the extraction strategy features in any of the above embodiments includes the following steps:

[0180] S171. The extraction strategy features are processed through the first max-pooling layer, the second max-pooling layer, and the third max-pooling layer respectively to obtain the first pooling matrix, the second pooling matrix, and the third pooling matrix.

[0181] The step sizes of the first max pooling layer, the second max pooling layer, and the third max pooling layer are all different.

[0182] Max pooling is a fundamental layer structure in neural networks. It performs downsampling by sliding a window across the input feature map and taking the maximum value within each window. Its purpose is to reduce the dimensionality of the data while preserving the most salient features, which helps improve the computational efficiency of the model, reduce overfitting, and make the model more robust to small changes in the input.

[0183] For example, refer to Figure 18 As shown, the policy regression module 122 includes a first max pooling layer 181, a second max pooling layer 182, and a third max pooling layer 183. The first max pooling layer 181, the second max pooling layer 182, and the third max pooling layer 183 are respectively used to extract policy features F for processing to obtain a first pooling matrix M. MP1 The second pooling matrix M MP2 and the third pooling matrix MMP3 .

[0184] S172. Stretch the first pooling matrix, the second pooling matrix, and the third pooling matrix into one-dimensional vectors respectively to obtain the first vector, the second vector, and the third vector.

[0185] For example, refer to Figure 18 As shown, the policy regression module 122 further includes: a first stretching module 184, a second stretching module 185, and a third stretching module 186, which are respectively used to stretch the first pooling matrix M. MP1 The second pooling matrix M MP2 and the third pooling matrix M MP3 Stretch to obtain the first vector V1, the second vector V2, and the third vector V3.

[0186] S173. Concatenate the first vector, the second vector, and the third vector to obtain a concatenated vector.

[0187] For example, refer to Figure 18 As shown, the strategy regression module 122 also includes a splicing module 187, used to splice the first vector V1, the second vector V2, and the third vector V3, and output the spliced ​​vector V. C .

[0188] S174. The concatenated vector is processed by a fully connected module to obtain a probability vector.

[0189] The fully connected module includes multiple serially connected fully connected layers.

[0190] For example, refer to Figure 18 As shown, the strategy regression module 122 includes a fully connected module 188, which includes multiple fully connected layers 1880 connected in series. Figure 16 The example shown is a fully connected module 188 comprising two fully connected layers 1880. The fully connected module 188 is used to process the splicing vector V. C The process is performed to obtain the probability vector P.

[0191] S175. The probability vector is processed through an activation function layer to obtain a decision vector.

[0192] In some embodiments, the activation function of the activation function layer is the Sigmoid activation function.

[0193] Reference Figure 18 As shown, the policy regression module 122 includes an activation function layer 189, which processes the probability vector P to make a decision vector P′.

[0194] S176. The decision vector is processed through an indicator function layer to obtain the filtering strategy.

[0195] In some implementations, the decision vector is processed by an indicator function layer, including: if a component in the decision vector is greater than 0.5, the output is 1; otherwise, the input is 0.

[0196] Reference Figure 18 As shown, the policy regression module 122 includes: an indicator function layer 1810, used to process the decision vector P′ to obtain the filtering policy D.

[0197] In some embodiments, refer to Figure 19 As shown, in any of the above embodiments, performing loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame includes:

[0198] S191. Generate a quantization parameter map based on the quantization parameters corresponding to the target video frame.

[0199] The resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame.

[0200] S192. Upsample the chroma components of the reconstructed video frame, the chroma components of the predicted video frame, and the chroma components of the boundary intensity map respectively to obtain the reconstructed upsampled map, the predicted upsampled map, and the boundary intensity upsampled map.

[0201] In some embodiments, refer to Figure 20 As shown, the loop filtering network includes: a first upsampling module 2001, a second upsampling module 2002, and a third upsampling module 2003. The first upsampling module 2001, the second upsampling module 2002, and the third upsampling module 2003 are respectively used to process the reconstructed video frames. chromaticity components The predicted video frame chromaticity components and the boundary strength map chromaticity components Perform upsampling to obtain reconstructed upsampling. Figure X ′, Predicted upsampled map P′ and Boundary strength upsampled map BP′.

[0202] The structures of the first upsampling module 2001, the second upsampling module 2002, and the third upsampling module 2003 can refer to the structure of the upsampling model shown in Figure 6. To avoid redundancy, they will not be described again here.

[0203] S193. Perform convolution operations on the reconstructed upsampled map, the predicted upsampled map, the boundary intensity upsampled map, and the quantization parameter map respectively to obtain the reconstruction matrix, the prediction matrix, the boundary intensity matrix, and the quantization parameter matrix.

[0204] In some embodiments, refer to Figure 20 As shown, the loop filter network further includes convolutional layers 2004, 2005, 2006, and 2007, which are respectively used for upsampling the reconstruction. Figure X From the predicted upsampled image P′, the boundary intensity upsampled image BP′, and the quantization parameter image Q, the reconstruction matrix M is obtained. X′ Prediction matrix M P′ Boundary strength matrix M BP′ and the quantization parameter matrix M Q .

[0205] S194. The reconstruction matrix, the prediction matrix, the boundary strength matrix, and the quantization parameter matrix are concatenated from the channel dimension to obtain the initial filtering features.

[0206] In some embodiments, refer to Figure 20 As shown, the loop filter network further includes a splicing module 2008, which is used to splice the reconstruction matrix M from the channel dimension. X′ Prediction matrix M P′ Boundary strength matrix M BP′ and the quantization parameter matrix M Q Thus, the initial filter feature Z is output. u .

[0207] S195. The initial filtering features are processed by the target loop filtering network to obtain refined filtering features.

[0208] In some embodiments, refer to Figure 20 As shown, the loop filter network further includes: an initial loop filter network 2009, and the number of filtering modules in the initial loop filter network 2009 is first adjusted by a filtering strategy to obtain a target loop filter network 2010, and the initial filtering feature Z is filtered by the target loop filter network 2010. u Processing is performed to obtain the refined filtered feature Z.

[0209] In some embodiments, the filtering module is a residual block. The structure of the residual block can be referred to... Figure 13 As shown, to avoid redundancy, the explanation will not be repeated here.

[0210] S196. Obtain the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features.

[0211] In some embodiments, refer to Figure 20 As shown, the loop filtering network further includes an output module 2011 for processing the refined filtering feature Z to output the reconstructed video frame. Corresponding enhanced video frames

[0212] In some embodiments, refer to Figure 21 As shown, in Figure 19 Based on the illustrated embodiment, before step S195 (processing the initial filtering features through the target loop filtering network to obtain refined filtering features), the video decoding method provided in this application embodiment further includes:

[0213] S211. Perform the third and fourth convolution operations on the initial filtered features in sequence.

[0214] The third convolution operation is used to reduce the number of channels in the initial filtered feature, and the fourth convolution operation is used to reduce the size of the initial filtered feature.

[0215] In some embodiments, refer to Figure 22 As shown, the loop filtering network further includes a convolutional layer 221 and a convolutional layer 222. The convolutional layer 221 performs a third convolution operation on the initial filtered features to reduce the number of channels in the initial filtered features, and the convolutional layer 222 performs a fourth convolution operation on the output features of the convolutional layer 221 to reduce the size of the initial filtered features.

[0216] Before processing the initial filtered features through the loop filtering network, performing a third convolution operation and a fourth convolution operation on the initial filtered features sequentially can reduce the number of channels and the feature size of the initial filtered features, thereby reducing the computational load in the process of obtaining enhanced video frames.

[0217] In some embodiments, refer to Figure 23 As shown, obtaining the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features includes:

[0218] S231. Perform a convolution operation on the refined filtered features to obtain the first enhanced features.

[0219] In some embodiments, refer to Figure 24 As shown, in Figure 23 Based on the illustrated embodiment, the output module 2011 of the loop filter network includes: a convolutional layer 241, used to perform a convolution operation on the refined filter feature Z to obtain the first enhanced feature Z.e .

[0220] S232. Perform a convolution operation on the first enhancement feature to obtain the enhanced video frame.

[0221] In some embodiments, refer to Figure 24 As shown, the output module 2011 of the loop filter network further includes a convolutional layer 242, used for processing the first enhanced feature Z. e Perform convolution operations to obtain the reconstructed video frames. Corresponding enhanced video frames

[0222] In some embodiments, refer to Figure 25 As shown, obtaining the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features includes:

[0223] S251. Perform a convolution operation on the refined filtered features to obtain the first enhanced features.

[0224] In some embodiments, refer to Figure 26 As shown, in Figure 23 Based on the illustrated embodiment, the output module 2011 of the loop filter network includes: a convolutional layer 261, used to perform a convolution operation on the refined filter feature Z to obtain the first enhanced feature Z. e .

[0225] S252. Perform a convolution operation on the first enhanced feature to obtain a first residual feature, a second residual feature, a third residual feature, a fourth residual feature, a fifth residual feature, and a sixth residual feature.

[0226] In some embodiments, refer to Figure 26 As shown, the output module 2011 of the loop filter network further includes a convolutional layer 262, used for processing the first enhanced feature Z. e Perform convolution operations to obtain the first residual feature R1, the second residual feature R2, the third residual feature R3, the fourth residual feature R4, the fifth residual feature R5, and the sixth residual feature R6.

[0227] S253. Fuse the first residual feature and the U component of the reconstructed video frame to obtain the enhanced U component.

[0228] In some embodiments, refer to Figure 26 As shown, the output module 2011 of the loop filter network further includes an adder 263, used to fuse the first residual feature R1 and the U component of the reconstructed video frame. To obtain enhanced U component

[0229] S254. Fuse the second residual feature and the V component of the reconstructed video frame to obtain the enhanced V component.

[0230] In some embodiments, refer to Figure 26 As shown, the output module 2011 of the loop filter network further includes an adder 264, used to fuse the second residual feature R2 and the V component of the reconstructed video frame. To obtain enhanced V component

[0231] S255. Perform pixel shuffling operation on the third residual feature, the fourth residual feature, the fifth residual feature and the sixth residual feature to obtain the brightness residual feature.

[0232] In some embodiments, refer to Figure 26 As shown, the output module 2011 of the loop filter network further includes a pixel shuffle unit 265, used to perform pixel shuffle operations on the third residual feature R3, the fourth residual feature R4, the fifth residual feature R5, and the sixth residual feature R6 to obtain the luminance residual feature R. Y .

[0233] S256. Fuse the luminance residual features and the luminance components of the reconstructed video frame to obtain an enhanced luminance component.

[0234] In some embodiments, refer to Figure 26 As shown, the output module 2011 of the loop filter network further includes an adder 266, used to fuse the luminance residual feature R. Y and the luminance component of the reconstructed video frame To obtain enhanced luminance components

[0235] S257. Combine the enhanced U component, the enhanced V component, and the enhanced luminance component to obtain the enhanced video frame.

[0236] In some embodiments, the video decoding method provided in this application further includes: training a policy generation network and a loop filter network. The implementation of training the policy generation network and the loop filter network may include the following steps a to c:

[0237] Step a: Pre-train the loop filter network.

[0238] Before applying the policy network to perform residual block skipping on the filtering network, it should be ensured that the filtering network already has the ability to enhance images and remove compression artifacts. This is more conducive to the rapid convergence of the network joint training. Therefore, in this embodiment, the filtering network is pre-trained first.

[0239] In some embodiments, pre-training the loop filter network includes the following steps a1 to a3:

[0240] Step a1: Input the sample data into the loop filter network and obtain the predicted enhanced image corresponding to the sample data output by the loop filter network.

[0241] Step a2: Calculate the loss value based on the label data corresponding to the sample data and the predicted enhanced image corresponding to the sample data output by the loop filter network.

[0242] Step a3: Adjust the network parameters of the loop filter network according to the loss value.

[0243] Step b: Jointly train the policy generation network and the loop filter network.

[0244] By jointly training the loop filter network and the policy generation network, the loop filter gradually learns the ability to adapt to samples and skip network layers with adjustable complexity. For each image patch sample, the loop filter comprehensively considers both computational complexity and enhancement quality to determine the optimal inference path; therefore, the training loss function should be a combination of these two aspects. In the loop filter, the policy network adopts a lightweight structure design, and its computational cost is almost negligible compared to the filter network. Furthermore, statistically, the multiply-accumulate operations in the backbone module of the filter network account for approximately 90% of the entire network. Therefore, this chapter uses the number of residual blocks executed in the backbone module (i.e., the sum of all elements in the filter policy) to describe the computational complexity of the loop filter.

[0245] Furthermore, the filtering strategy determines whether the filtering network executes each filtering module during inference. However, the operation of mapping continuous probabilities to discrete binary decisions during model training is non-differentiable. To address this issue, embodiments of this application use Gumbel noise and the sigmoid function to approximate the probability vector as a binary vector.

[0246] Based on the same inventive concept, as an implementation of the above method, this application also provides a video decoding device. This embodiment corresponds to the aforementioned method embodiment. For ease of reading, this embodiment will not repeat the details of the aforementioned method embodiment one by one, but it should be clear that the video decoding device in this embodiment can implement all the contents of the aforementioned method embodiment.

[0247] Reference Figure 27 As shown, some embodiments of this application provide a video decoding apparatus, which may include:

[0248] Acquisition unit 271 is used to acquire the encoded data of the target video frame;

[0249] Decoding unit 272 is used to decode the encoded data of the target video frame and obtain the reconstructed video frame of the target video frame;

[0250] Processing unit 273 obtains a filtering strategy, wherein the filtering strategy is a binary vector whose elements are a target number, and the target number is the number of filtering modules in the initial loop filtering network;

[0251] Adjustment unit 274 is used to adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network;

[0252] The filtering unit 275 is used to perform loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

[0253] In some embodiments, the processing unit 273 is specifically configured to: upsample the chroma components of the reconstructed video frame to obtain a reconstructed upsampled image corresponding to the reconstructed video frame, wherein the chroma and luminance components of the reconstructed upsampled image have the same size; perform a convolution operation on the reconstructed upsampled image to obtain a reconstruction matrix; generate a quantization parameter map and a balance factor map based on the quantization parameters corresponding to the target video frame and a pre-configured balance factor, respectively, wherein the resolution of the quantization parameter map and the resolution of the balance factor map are the same as the resolution of the reconstructed video frame; perform a convolution operation on the quantization parameter map and the balance factor map, respectively, to obtain a quantization parameter matrix and a balance factor matrix; concatenate the reconstruction matrix, the quantization parameter matrix, and the balance factor matrix in the channel dimension to obtain initial strategy features; and obtain the filtering strategy based on the initial strategy features.

[0254] In some embodiments, the processing unit 273 is specifically configured to: upsample the chroma components of the reconstructed video frame to obtain a reconstructed upsampled image, wherein the chroma and luminance components of the reconstructed upsampled image have the same size; perform a convolution operation on the reconstructed upsampled image to obtain a reconstruction matrix; generate a quantization parameter map based on the quantization parameters corresponding to the target video frame, wherein the resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame; perform a convolution operation on the quantization parameter map to obtain a quantization parameter matrix; concatenate the reconstruction matrix and the quantization parameter matrix in the channel dimension to obtain initial policy features; and obtain the filtering policy based on the initial policy features.

[0255] In some embodiments, the processing unit 273 is specifically configured to: generate a balance factor map based on pre-configured balance factors, wherein the resolution of the balance factor map is the same as the resolution of the reconstructed video frame; perform a convolution operation on the balance factor map to obtain a balance factor matrix; determine the balance factor matrix as an initial strategy feature; and obtain the filtering strategy based on the initial strategy feature.

[0256] In some embodiments, the processing unit 273 is specifically used to process the initial policy features through a residual network to obtain refined policy features. The residual network includes multiple concatenated residual blocks, each residual block including: a first convolutional layer, a second convolutional layer, a third convolutional layer, a concatenation layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and an adder. The first and second convolutional layers are respectively used to perform convolution operations on the input of the residual block. The third convolutional layer is used to perform a convolution operation on the output of the second convolutional layer. The concatenation layer is used to concatenate the output of the first convolutional layer and the output of the third convolutional layer. The fourth, fifth, and sixth convolutional layers are sequentially concatenated. The input of the fourth convolutional layer is the output of the concatenation layer. The input of the adder is the output of the sixth convolutional layer and the input of the residual block. The output of the adder is the output of the residual block. The filtering policy is generated based on the refined policy features.

[0257] In some embodiments, the processing unit 273 is further configured to perform a first convolution operation and a second convolution operation on the initial policy features sequentially before processing the initial policy features through the residual network;

[0258] The first convolution operation is used to reduce the number of channels in the initial policy feature, and the second convolution operation is used to reduce the size of the initial policy feature.

[0259] In some embodiments, the processing unit 273 is specifically used to process the extracted strategy features through a fully connected module to obtain a probability vector; the fully connected module includes multiple concatenated fully connected layers; the probability vector is processed through an activation function layer to obtain a decision vector; and the decision vector is processed through an indicator function layer to obtain the filtering strategy.

[0260] In some embodiments, the processing unit 273 is specifically configured to: process the extraction strategy features through a first max pooling layer, a second max pooling layer, and a third max pooling layer respectively to obtain a first pooling matrix, a second pooling matrix, and a third pooling matrix; the step sizes of the first max pooling layer, the second max pooling layer, and the third max pooling layer are all different; stretch the first pooling matrix, the second pooling matrix, and the third pooling matrix into one-dimensional vectors respectively to obtain a first vector, a second vector, and a third vector; concatenate the first vector, the second vector, and the third vector to obtain a concatenated vector; process the concatenated vector through a fully connected module to obtain a probability vector; the fully connected module includes multiple concatenated fully connected layers; process the probability vector through an activation function layer to obtain a decision vector; and process the decision vector through an indicator function layer to obtain the filtering strategy.

[0261] In some embodiments, the filtering unit 275 is specifically configured to: obtain initial filtering features based on the reconstructed video frame, the predicted image of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame; process the initial filtering features through the target loop filtering network to obtain refined filtering features; and obtain the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features.

[0262] In some embodiments, the filtering unit 275 is specifically configured to: generate a quantization parameter map based on the quantization parameters corresponding to the target video frame, wherein the resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame; upsample the chroma components of the reconstructed video frame, the chroma components of the predicted video frame, and the chroma components of the boundary intensity map, respectively, to obtain a reconstructed upsampled map, a predicted upsampled map, and a boundary intensity upsampled map; perform convolution operations on the reconstructed upsampled map, the predicted upsampled map, the boundary intensity upsampled map, and the quantization parameter map, respectively, to obtain a reconstruction matrix, a prediction matrix, a boundary intensity matrix, and a quantization parameter matrix; and concatenate the reconstruction matrix, the prediction matrix, the boundary intensity matrix, and the quantization parameter matrix from the channel dimension to obtain the initial filtering features.

[0263] In some embodiments, the filtering unit 275 is further configured to perform a third convolution operation and a fourth convolution operation on the initial filtering features sequentially before processing the initial filtering features through the target loop filtering network;

[0264] The third convolution operation is used to reduce the number of channels in the initial filtered feature, and the fourth convolution operation is used to reduce the size of the initial filtered feature.

[0265] In some embodiments, the filtering unit 275 is specifically configured to perform a convolution operation on the refined filtering features to obtain a first enhanced feature; and perform a convolution operation on the first enhanced feature to obtain the enhanced video frame.

[0266] In some embodiments, the filtering unit 275 is specifically configured to: perform a convolution operation on the refined filtered features to obtain a first enhanced feature; perform a convolution operation on the first enhanced feature to obtain a first residual feature, a second residual feature, a third residual feature, a fourth residual feature, a fifth residual feature, and a sixth residual feature; fuse the first residual feature and the U component of the reconstructed video frame to obtain an enhanced U component; fuse the second residual feature and the V component of the reconstructed video frame to obtain an enhanced V component; perform a pixel shuffling operation on the third residual feature, the fourth residual feature, the fifth residual feature, and the sixth residual feature to obtain a luminance residual feature; fuse the luminance residual feature and the luminance component of the reconstructed video frame to obtain an enhanced luminance component; and combine the enhanced U component, the enhanced V component, and the enhanced luminance component to obtain the enhanced video frame.

[0267] In some embodiments, the filtering module is a residual block;

[0268] The residual block includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a concatenation layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and an adder. The first and second convolutional layers are used to perform convolution operations on the input of the residual block, the third convolutional layer is used to perform convolution operations on the output of the second convolutional layer, the concatenation layer is used to concatenate the output of the first convolutional layer and the output of the third convolutional layer, the fourth, fifth, and sixth convolutional layers are connected in series, the input of the fourth convolutional layer is the output of the concatenation layer, the input of the adder is the output of the sixth convolutional layer and the input of the residual block, and the output of the adder is the output of the residual block.

[0269] In some embodiments, the adjustment unit 274 is specifically used to: enable the i-th filtering module of the initial loop filter network if the value of the i-th element of the filtering strategy is a first preset value, and bypass the i-th filtering module of the initial loop filter network if the value of the i-th element of the filtering strategy is a second preset value; wherein i is a positive integer less than or equal to the target number.

[0270] In some embodiments, the acquisition unit 271 is specifically used to encode the target video frame to obtain the encoded data of the target video frame.

[0271] In some embodiments, the acquisition unit 271 is specifically used to receive the encoded data of the target video frame sent by the media resource server.

[0272] The video decoding apparatus provided in the above embodiments can execute the video decoding method provided in the above embodiments, and the implementation principle and the achieved technical effect are the same. To avoid redundancy, it will not be described in detail again.

[0273] Some embodiments of this application provide an electronic device that may include: a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the video decoding method described in any of the above embodiments or the video decoding method described in any of the above embodiments.

[0274] Figure 28 An exemplary configuration block diagram of an electronic device 2800 from some of the above embodiments is shown. For example... Figure 28 As shown, the display device 2800 may include at least one of the following: a tuner 281, a communicator 282, a detector 283, an external device interface 284, a processor 285, a display 286, an audio output interface 287, a memory 288, a power supply 289, and a user interface.

[0275] In some embodiments, processor 285 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0276] The display 286 includes a display screen assembly for presenting images, a driving assembly for driving image display, a component for receiving image signals from the processor output, and a user interface for displaying video content, image content, menu control interface, and user control UI.

[0277] The display 286 can be a liquid crystal display, an organic light-emitting diode (OLED) display, or a projection display, etc.

[0278] The communicator 282 is a component used to communicate with external devices or servers according to various communication protocol types. For example, the communicator may include at least one of the following: a Wi-Fi module, a Bluetooth module, a wired Ethernet module, other network communication protocol chips or near-field communication protocol chips, and an infrared receiver. The electronic device 2800 can use the communicator 282 to send and receive control signals and data signals with the control device or server.

[0279] The user interface can be used to receive control signals input by the user through a control device (such as an infrared remote control) or by touch or gesture.

[0280] Detector 283 can be used to acquire signals from the external environment or to interact with the external environment. For example, detector 283 may include a light receiver, a sensor for acquiring ambient light intensity; or, detector 283 may include an image acquisition device, such as a camera, for acquiring external environmental scenes, user attributes, or user interaction gestures; or, detector 283 may include a sound acquisition device, such as a microphone, for receiving external sounds.

[0281] The external device interface 284 may include, but is not limited to, one or more of the following: High-Definition Multimedia Interface (HDMI), analog or high-definition component input interface (component), Composite Video Broadcast Signal (CVBS), Universal Serial Bus (USB), RGB (Red, Green, Blue) port, etc. It may also be a composite input / output interface formed by multiple of the above interfaces.

[0282] The tuner / demodulator 281 receives broadcast television signals via wired or wireless means, and demodulates audio and video signals, such as EPG data signals, from multiple wireless or wired broadcast television signals. In some embodiments, the processor 285 and the tuner / demodulator 281 may be located in different separate devices, that is, the tuner / demodulator 281 may also be located in an external device of the main device where the processor 285 is located, such as an external set-top box.

[0283] Processor 285 controls the operation of the display device and responds to user operations through various software control programs stored in memory. Processor 285 controls the overall operation of the electronic device. For example, in response to receiving a user command to select a UI object to display on display 286, processor 285 can execute operations related to the object selected by the user command.

[0284] In some embodiments, processor 285 includes at least one of: a central processing unit (CPU), a video processor, an audio processor, a graphics processing unit (GPU), RAM (random access memory), ROM (read-only memory), a first to an nth interface for input / output, a communication bus, etc.

[0285] Some embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a computing device, causes the computing device to implement the video decoding method described in any of the above embodiments.

[0286] Some embodiments of this application provide a chip including a processor and a memory. The memory is used to store programs or instructions that can run on the processor, and the processor is used to execute the programs or instructions to cause the video decoding method described in any of the above embodiments to be executed.

[0287] Some embodiments of this application provide a computer program product that, when run on a computer, enables the computer to implement the video decoding method described in any of the above embodiments.

[0288] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

[0289] For ease of explanation, the above description has been provided in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Various modifications and variations can be obtained based on the above teachings. The selection and description of the above embodiments are for the purpose of better explaining the principles and practical applications, thereby enabling those skilled in the art to better utilize the described embodiments and various different variations of embodiments suitable for specific use considerations.

Claims

1. A video decoding method, characterized in that, include: Obtain the encoded data of the target video frame; Decode the encoded data of the target video frame to obtain the reconstructed video frame of the target video frame; Obtain a filtering strategy, wherein the filtering strategy is a binary vector whose elements are the target number, and the target number is the number of filtering modules in the initial loop filtering network; The number of filtering modules in the initial loop filter network is adjusted based on the filtering strategy to obtain the target loop filter network. The reconstructed video frame is subjected to loop filtering by the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

2. The method according to claim 1, characterized in that, The acquisition filtering strategy includes: The chroma components of the reconstructed video frame are upsampled to obtain a reconstructed upsampled image corresponding to the reconstructed video frame. The chroma components and luminance components of the reconstructed upsampled image have the same size. Perform a convolution operation on the reconstructed upsampled image to obtain the reconstruction matrix; A quantization parameter map and a balance factor map are generated based on the quantization parameters corresponding to the target video frame and a pre-configured balance factor, respectively. The resolution of the quantization parameter map and the resolution of the balance factor map are the same as the resolution of the reconstructed video frame. Convolution operations are performed on the quantization parameter map and the balance factor map respectively to obtain the quantization parameter matrix and the balance factor matrix; The reconstruction matrix, the quantization parameter matrix, and the balance factor matrix are concatenated along the channel dimension to obtain the initial strategy features; The filtering strategy is obtained based on the initial strategy features.

3. The method according to claim 1, characterized in that, The acquisition filtering strategy includes: The chroma components of the reconstructed video frame are upsampled to obtain a reconstructed upsampled image, wherein the chroma components and luminance components of the reconstructed upsampled image have the same size. Perform a convolution operation on the reconstructed upsampled image to obtain the reconstruction matrix; A quantization parameter map is generated based on the quantization parameters corresponding to the target video frame, and the resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame. Perform a convolution operation on the quantization parameter map to obtain the quantization parameter matrix; The reconstruction matrix and the quantization parameter matrix are concatenated along the channel dimension to obtain the initial strategy features; The filtering strategy is obtained based on the initial strategy features.

4. The method according to claim 1, characterized in that, The acquisition filtering strategy includes: A balance factor map is generated based on a pre-configured balance factor, and the resolution of the balance factor map is the same as the resolution of the reconstructed video frame. Perform a convolution operation on the balance factor graph to obtain the balance factor matrix; The balance factor matrix is ​​determined as the initial strategy features; The filtering strategy is obtained based on the initial strategy features.

5. The method according to any one of claims 2-4, characterized in that, The step of obtaining the filtering strategy based on the initial strategy features includes: The initial policy features are processed by a residual network to obtain refined policy features. The residual network includes multiple concatenated residual blocks. Each residual block includes a first convolutional layer, a second convolutional layer, a third convolutional layer, a concatenation layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and an adder. The first and second convolutional layers are used to perform convolution operations on the input of the residual block, respectively. The third convolutional layer is used to perform a convolution operation on the output of the second convolutional layer. The concatenation layer is used to concatenate the output of the first convolutional layer and the output of the third convolutional layer. The fourth, fifth, and sixth convolutional layers are concatenated in sequence. The input of the fourth convolutional layer is the output of the concatenation layer. The input of the adder is the output of the sixth convolutional layer and the input of the residual block. The output of the adder is the output of the residual block. The filtering strategy is generated based on the features of the extraction strategy.

6. The method according to claim 5, characterized in that, Before processing the initial policy features through the residual network, the method further includes: The initial policy features are then subjected to a first convolution operation and a second convolution operation in sequence. The first convolution operation is used to reduce the number of channels in the initial policy feature, and the second convolution operation is used to reduce the size of the initial policy feature.

7. The method according to claim 5, characterized in that, The step of generating the filtering strategy based on the extraction strategy features includes: The extraction strategy features are processed by a fully connected module to obtain a probability vector; the fully connected module includes multiple serially connected layers. The probability vector is processed by an activation function layer to produce a decision vector; The decision vector is processed by an instruction function layer to obtain the filtering strategy.

8. The method according to claim 5, characterized in that, The step of generating the filtering strategy based on the extraction strategy features includes: The extraction strategy features are processed through a first max pooling layer, a second max pooling layer, and a third max pooling layer, respectively, to obtain a first pooling matrix, a second pooling matrix, and a third pooling matrix; the step sizes of the first max pooling layer, the second max pooling layer, and the third max pooling layer are all different. The first pooling matrix, the second pooling matrix, and the third pooling matrix are stretched into one-dimensional vectors respectively to obtain a first vector, a second vector, and a third vector; Concatenate the first vector, the second vector, and the third vector to obtain a concatenated vector; The concatenated vector is processed by a fully connected module to obtain a probability vector; the fully connected module includes multiple concatenated fully connected layers. The probability vector is processed by an activation function layer to produce a decision vector; The decision vector is processed by an instruction function layer to obtain the filtering strategy.

9. The method according to claim 1, characterized in that, The step of performing loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame includes: Initial filtering features are obtained based on the reconstructed video frame, the predicted image of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame. The initial filtering features are processed by the target loop filtering network to obtain refined filtering features; The enhanced video frame corresponding to the reconstructed video frame is obtained based on the refined filtering features.

10. The method according to claim 9, characterized in that, The step of obtaining initial filtering features based on the reconstructed video frame, the predicted video frame of the target video frame, the boundary intensity map of the reconstructed video frame, and the quantization parameters corresponding to the target video frame includes: A quantization parameter map is generated based on the quantization parameters corresponding to the target video frame, and the resolution of the quantization parameter map is the same as the resolution of the reconstructed video frame. The chroma components of the reconstructed video frame, the chroma components of the predicted video frame, and the chroma components of the boundary intensity map are upsampled respectively to obtain the reconstructed upsampled map, the predicted upsampled map, and the boundary intensity upsampled map. Convolution operations are performed on the reconstructed upsampled map, the predicted upsampled map, the boundary intensity upsampled map, and the quantization parameter map respectively to obtain the reconstruction matrix, the prediction matrix, the boundary intensity matrix, and the quantization parameter matrix. The initial filtering features are obtained by concatenating the reconstruction matrix, the prediction matrix, the boundary strength matrix, and the quantization parameter matrix along the channel dimension.

11. The method according to claim 9, characterized in that, Before processing the initial filtered features through the target loop filter network, the method further includes: The initial filtered features are then subjected to a third and a fourth convolution operation in sequence. The third convolution operation is used to reduce the number of channels in the initial filtered feature, and the fourth convolution operation is used to reduce the size of the initial filtered feature.

12. The method according to claim 9, characterized in that, The step of obtaining the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features includes: The refined filtered features are convolved to obtain the first enhanced features; Perform a convolution operation on the first enhancement feature to obtain the enhanced video frame.

13. The method according to claim 9, characterized in that, The step of obtaining the enhanced video frame corresponding to the reconstructed video frame based on the refined filtering features includes: The refined filtered features are convolved to obtain the first enhanced features; The first enhanced feature is convolved to obtain the first residual feature, the second residual feature, the third residual feature, the fourth residual feature, the fifth residual feature, and the sixth residual feature; The first residual feature and the U component of the reconstructed video frame are fused to obtain the enhanced U component; The second residual feature and the V component of the reconstructed video frame are fused to obtain the enhanced V component; A pixel shuffling operation is performed on the third residual feature, the fourth residual feature, the fifth residual feature, and the sixth residual feature to obtain the brightness residual feature; The luminance residual features and the luminance components of the reconstructed video frame are fused to obtain the enhanced luminance components; The enhanced U component, the enhanced V component, and the enhanced luminance component are combined to obtain the enhanced video frame.

14. The method according to claim 1, characterized in that, The filtering module is a residual block; The residual block includes: a first convolutional layer, a second convolutional layer, a third convolutional layer, a concatenation layer, a fourth convolutional layer, a fifth convolutional layer, a sixth convolutional layer, and an adder. The first and second convolutional layers are used to perform convolution operations on the input of the residual block, the third convolutional layer is used to perform convolution operations on the output of the second convolutional layer, the concatenation layer is used to concatenate the output of the first convolutional layer and the output of the third convolutional layer, the fourth, fifth, and sixth convolutional layers are connected in series, the input of the fourth convolutional layer is the output of the concatenation layer, the input of the adder is the output of the sixth convolutional layer and the input of the residual block, and the output of the adder is the output of the residual block.

15. The method according to claim 1, characterized in that, The step of adjusting the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network includes: If the value of the i-th element of the filtering strategy is a first preset value, then the i-th filtering module of the initial loop filtering network is enabled; if the value of the i-th element of the filtering strategy is a second preset value, then the i-th filtering module of the initial loop filtering network is bypassed; where i is a positive integer less than or equal to the target number.

16. The method according to claim 1, characterized in that, The acquisition of encoded data for the target video frame includes: The target video frame is encoded to obtain the encoded data of the target video frame.

17. The method according to claim 1, characterized in that, The acquisition of encoded data for the target video frame includes: Receive the encoded data of the target video frame sent by the media resource server.

18. A video decoding device, characterized in that, include: The acquisition unit is used to acquire the encoded data of the target video frame; A decoding unit is used to decode the encoded data of the target video frame and obtain the reconstructed video frame of the target video frame; The processing unit obtains a filtering strategy, wherein the filtering strategy is a binary vector whose elements are a target number, and the target number is the number of filtering modules in the initial loop filtering network; An adjustment unit is used to adjust the number of filtering modules in the initial loop filter network based on the filtering strategy to obtain the target loop filter network. The filtering unit is used to perform loop filtering on the reconstructed video frame through the target loop filtering network to obtain the enhanced video frame corresponding to the reconstructed video frame.

19. An electronic device, characterized in that, include: A memory and a processor, wherein the memory stores a computer program, and the processor is configured to implement the video decoding method as described in any one of claims 1-17 when executing the computer program.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a computing device, causes the computing device to implement the video decoding method according to any one of claims 1-17.

21. A chip, characterized in that, The chip includes a processor and a memory, the memory being used to store programs or instructions that can run on the processor, and the processor being used to execute the programs or instructions to cause the video decoding method of any one of claims 1-17 to be executed.

22. A computer program product, when run on a computer, causes the computer to implement the video decoding method as described in any one of claims 1-17.

Images