Filtering method, encoder, decoder, and storage medium

Abstract

The embodiment of the invention discloses a filtering method, an encoder, a decoder and a storage medium, and the method comprises the steps: obtaining a frame-level use identification bit through parsing a code stream; when the frame-level use identification bit represents that neural network filtering is started for the current frame, obtaining a frame-level switch identification bit and a frame-level quantization parameter adjustment identification bit; when the frame-level switch identifier represents that each block in the current frame is subjected to neural network filtering, and the frame-level quantization parameter adjustment identifier represents that the quantization parameter is adjusted for the current frame, obtaining an adjusted frame-level quantization parameter; and filtering the current block of the current frame based on the adjusted frame-level quantization parameter and a neural network filtering model to obtain first residual information of the current block.

Description

[0001] This application is a divisional application of the patent application filed on April 13, 2022, with application number 202280094789.4 and application name “Filtering method, encoder, decoder and storage medium”. Technical Field

[0002] The embodiments of the present application relate to the field of image processing technology, and in particular to a filtering method, an encoder, a decoder, and a storage medium. Background Art

[0003] In video coding and decoding systems, most video coding uses a block-based hybrid coding framework. Each frame in the video is divided into several coding tree units (CTUs), and a coding tree unit can be further divided into several rectangular coding units (CUs), which can be rectangular blocks or square blocks. Since adjacent CUs use different coding parameters, such as different transformation processes, different quantization parameters (QPs), different prediction methods, different reference image frames, etc., and the error size and distribution characteristics introduced by each CU are independent of each other, the discontinuity of the boundaries of adjacent CUs produces a block effect, which affects the subjective and objective quality of the reconstructed image and even affects the prediction accuracy of subsequent coding and decoding.

[0004] In this way, in the encoding and decoding process, the loop filter is used to improve the subjective and objective quality of the reconstructed image. The coding performance of the loop filtering method based on the neural network is the most outstanding. In the related art, on the one hand, the neural network filter model is switched at the coding tree unit level. Different neural network filter models are trained according to different sequence-level quantization parameter values ​​(BaseQP). The encoding end tries these different neural network filter models, and the neural network filter model with the lowest rate-distortion cost is used as the optimal network model of the current coding tree unit. Through the use flag bit and network model index information at the coding tree unit level, the decoding end can use the same network model as the encoding end for filtering. On the other hand, for different test conditions and quantization parameters, only a simplified low-complexity neural network filter model can be used for loop filtering. When using the low-complexity neural network filter model for filtering, the quantization parameter information is added as an additional input, that is, the quantization parameter information is used as the input of the network to improve the generalization ability of the neural network filter model, so as to achieve the effect of having good coding performance without switching the neural network filter model.

[0005] However, when the neural network filter model is switched at the coding tree unit level for filtering, since each coding tree unit corresponds to a neural network filter model, the hardware implementation complexity is high and the cost is large. When the low-complexity neural network filter model is used for filtering, the selection during filtering is not flexible enough due to the influence of quantization parameters, and the selection during encoding and decoding is still relatively small, which cannot achieve a good encoding and decoding effect. Summary of the invention

[0006] The embodiments of the present application provide a filtering method, an encoder, a decoder, and a storage medium, which can make the selection of input parameters for filtering more flexible without increasing the complexity, thereby improving the encoding and decoding efficiency.

[0007] The technical solution of the embodiment of the present application can be implemented as follows:

[0008] In a first aspect, an embodiment of the present application provides a filtering method, which is applied to a decoder, and the method includes:

[0009] Parse the bitstream and obtain the frame-level usage flag based on the neural network filtering model;

[0010] When the frame-level usage flag is used to indicate usage, a frame-level switch flag and a frame-level quantization parameter adjustment flag are obtained; the frame-level switch flag is used to determine whether all blocks in the current frame are filtered;

[0011] When the frame-level switch flag indicates that it is turned on and the frame-level quantization parameter adjustment flag indicates that it is used, obtaining an adjusted frame-level quantization parameter;

[0012] Based on the adjusted frame-level quantization parameter and the neural network filtering model, a current block of the current frame is filtered to obtain first residual information of the current block.

[0013] In a second aspect, an embodiment of the present application provides a filtering method, which is applied to an encoder, and the method includes:

[0014] Get the sequence-level permission flag;

[0015] When the sequence level allows the use of a flag to indicate permission, obtaining an original value of a current block in a current frame, a reconstructed value of the current block, and a frame-level quantization parameter;

[0016] Performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a first reconstruction value;

[0017] Estimating a rate-distortion cost based on the first reconstructed value and the original value of the current block to obtain a rate-distortion cost of the current block, and traversing the current frame to determine a first rate-distortion cost of the current frame;

[0018] Based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and a reconstructed value of a current block in the current frame, performing at least one filtering estimation on the current frame to determine at least one second rate-distortion cost of the current frame;

[0019] A frame-level quantization parameter adjustment flag is determined based on the first rate-distortion cost and the at least one second rate-distortion cost.

[0020] In a third aspect, an embodiment of the present application provides a decoder, the decoder comprising:

[0021] The parsing part is configured to parse the bitstream and obtain the frame-level usage identification bit based on the neural network filtering model;

[0022] The first determination part is configured to obtain a frame level switch flag and a frame level quantization parameter adjustment flag when the frame level usage flag indicates usage; the frame level switch flag is used to determine whether all blocks in the current frame are filtered;

[0023] The first adjustment part is configured to obtain an adjusted frame-level quantization parameter when the frame-level switch flag indicates that it is turned on and the frame-level quantization parameter adjustment flag indicates that it is used;

[0024] The first filtering part is configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block.

[0025] In a fourth aspect, an embodiment of the present application provides an encoder, the encoder comprising:

[0026] The second determining part is configured to obtain a sequence level permission flag; and when the sequence level permission flag indicates permission, obtain an original value of a current block in a current frame, a reconstructed value of the current block, and a frame level quantization parameter;

[0027] A second filtering part is configured to perform filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a first reconstruction value;

[0028] The second determining part is further configured to perform rate-distortion cost estimation based on the first reconstructed value and the original value of the current block to obtain the rate-distortion cost of the current block, and traverse the current frame to determine a first rate-distortion cost of the current frame;

[0029] The second filtering part is further configured to perform at least one filtering estimation on the current frame based on the neural network filtering model, at least one frame-level quantization bias parameter, the frame-level quantization parameter, and a reconstruction value of a current block in the current frame to determine at least one second rate-distortion cost of the current frame;

[0030] The second determining part is further configured to determine a frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost.

[0031] In a fifth aspect, an embodiment of the present application further provides a decoder, the decoder comprising:

[0032] A first memory configured to store a computer program executable on a first processor;

[0033] The first processor is configured to execute the method described in the first aspect when running the computer program.

[0034] In a sixth aspect, an embodiment of the present application further provides an encoder, the encoder comprising:

[0035] a second memory configured to store a computer program executable on a second processor;

[0036] The second processor is configured to execute the method described in the second aspect when running the computer program.

[0037] In the seventh aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a first processor, it implements the method as described in the first aspect, or when it is executed by a second processor, it implements the method as described in the second aspect.

[0038] The embodiment of the present application provides a filtering method, an encoder, a decoder and a storage medium, which obtains a frame-level usage flag based on a neural network filtering model by parsing a bit stream; when the frame-level usage flag indicates usage, obtains a frame-level switch flag and a frame-level quantization parameter adjustment flag; the frame-level switch flag is used to determine whether each block in the current frame is filtered; when the frame-level switch flag indicates that it is turned on and the frame-level quantization parameter adjustment flag indicates that it is used, obtain the adjusted frame-level quantization parameter; based on the adjusted frame-level quantization parameter and the neural network filtering model, filter the current block of the current frame to obtain the first residual information of the current block. In this way, it is possible to determine whether the quantization parameter of the input neural network filtering model needs to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameter (input parameter), thereby improving decoding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figures 1A-1C The distribution diagrams of each component in different color formats provided by the embodiments of the present application are exemplary;

[0040] Figure 2 A schematic diagram of the division of an exemplary coding unit provided in an embodiment of the present application;

[0041] Figure 3A A structural diagram of an exemplary neural network filtering model provided in an embodiment of the present application;

[0042] Figure 3B The structure of the exemplary neural network filtering model provided in the embodiment of the present application Figure 2 ;

[0043] Figure 4 Figure 3 of the structure of an exemplary neural network filtering model provided in an embodiment of the present application;

[0044] Figure 5 An exemplary video coding system structure diagram provided for an embodiment of the present application;

[0045] Figure 6 An exemplary video decoding system structure diagram provided in an embodiment of the present application;

[0046] Figure 7 A flowchart of a filtering method provided in an embodiment of the present application;

[0047] Figure 8 A flowchart of another filtering method provided in an embodiment of the present application;

[0048] Fig. 9 A schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0049] Fig.10 A schematic diagram of the hardware structure of a decoder provided in an embodiment of the present application;

[0050] Fig.11 A schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0051] Fig.12 A schematic diagram of the hardware structure of an encoder provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In the embodiments of the present application, the digital video compression technology mainly compresses the huge digital video data for transmission and storage. With the surge in Internet videos and people's increasing requirements for video clarity, although the existing digital video compression standards can save a lot of video data, there is still a need to pursue better digital video compression technology to reduce the bandwidth and flow pressure of digital video transmission.

[0053] In the process of digital video encoding, the encoder reads unequal pixels from the original video sequence of different color formats, including brightness components and chrominance components, that is, the encoder reads a black and white or color image. Then it is divided into blocks and handed over to the encoder for encoding. The encoder is usually a hybrid frame coding mode, which generally includes intra-frame and inter-frame prediction, transformation and quantization, inverse transformation and inverse quantization, loop filtering and entropy coding. Intra-frame prediction only refers to the information of the same frame image, predicts the pixel information in the current block of the division, and is used to eliminate spatial redundancy; inter-frame prediction can refer to the image information of different frames, and use motion estimation to search for the motion vector information of the current block that best matches the division, which is used to eliminate temporal redundancy; transformation and quantization convert the predicted image block to the frequency domain, redistribute the energy, and combine quantization to remove information that is not sensitive to the human eye, which is used to eliminate visual redundancy; entropy coding can eliminate character redundancy based on the current context model and the probability information of the binary code stream; loop filtering mainly processes the pixels after inverse transformation and inverse quantization to compensate for the distortion information and provide a better reference for subsequent encoding pixels.

[0054] At present, the scenario in which filtering processing can be performed can be a reference software test platform HPM based on AVS or a VVC reference software test platform (VVC TEST MODEL, VTM) based on Versatile Video Coding (VVC), which is not limited in the embodiments of the present application.

[0055] In a video image, a first video component, a second video component and a third video component are generally used to represent a current block (Coding Block, CB); wherein the three image components are a brightness component, a blue chrominance component and a red chrominance component, respectively. The brightness component is usually represented by the symbol Y, the blue chrominance component is usually represented by the symbol Cb or U, and the red chrominance component is usually represented by the symbol Cr or V; thus, the video image can be represented in the YCbCr format or in the YUV format.

[0056] Usually, digital video compression technology works on image data in the YCbCr (YUV) format using a color encoding method. The YUV ratio is generally 4:2:0, 4:2:2, or 4:4:4. Y represents brightness (Luma), Cb (U) represents blue chromaticity, Cr (V) represents red chromaticity, and U and V represent chromaticity (Chroma) for describing color and saturation. Figures 1A to 1C The distribution diagram of each component in different color formats is shown, where white is the Y component and dark gray is the UV component. Figure 1A As shown, in the color format, 4:2:0 means that every 4 pixels have 4 brightness components and 2 chrominance components (YYYYCbCr), such as Figure 1B As shown, 4:2:2 means that every 4 pixels have 4 brightness components and 4 chrominance components (YYYYCbCrCbCr), and Figure 1C As shown, 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0057] At present, the common video coding standards are based on a block-based hybrid coding framework. Each image frame in the video image is divided into a square largest coding unit (LCU) of the same size (such as 128×128, 64×64, etc.), and each largest coding unit can be divided into rectangular coding units (CU) according to rules; and the coding unit may be divided into smaller prediction units (PU). Specifically, the hybrid coding framework may include prediction, transform, quantization, entropy coding, in-loop filter and other modules; among which, the prediction module may include intra-frame prediction and inter-frame prediction, and inter-frame prediction may include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video image, the use of intra-frame prediction in video coding and decoding technology can eliminate the spatial redundancy between adjacent pixels. Inter-frame prediction can refer to image information of different frames and use motion estimation to search for the motion vector information that best matches the current partition block to eliminate temporal redundancy; the transformation converts the predicted image block to the frequency domain and redistributes the energy. Combined with quantization, it can remove information that the human eye is not sensitive to, thereby eliminating visual redundancy; entropy coding can eliminate character redundancy based on the current context model and the probability information of the binary code stream.

[0058] It should be noted that during the video encoding process, the encoder first reads the image information and divides the image into several coding tree units (CTUs), and a coding tree unit can be further divided into several coding units (CUs). These coding units can be rectangular blocks or square blocks. For specific relationships, please refer to Figure 2 shown.

[0059] In the intra-frame prediction process, the current coding unit cannot refer to the information of different frames, and can only use the adjacent coding units of the same frame as reference information for prediction. That is, according to the current coding order from left to right and from top to bottom, the current coding unit can refer to the upper left coding unit, the upper coding unit and the left coding unit as reference information to predict the current coding unit, and the current coding unit is used as the reference information of the next coding unit, so as to predict the entire image. If the input digital video is in color format, that is, the current mainstream digital video encoder input source is in YUV 4:2:0 format, that is, every 4 pixels of the image are composed of 4 Y components and 2 UV components, the encoder will encode the Y component and UV component separately, and the encoding tools and technologies used are slightly different. At the same time, the decoding end will also decode according to different formats.

[0060] For the intra-frame prediction part in digital video encoding and decoding, the current block is predicted mainly by referring to the image information of the adjacent blocks of the current frame. The residual information is obtained by calculating the residual between the predicted block and the original image block, and then the residual information is transmitted to the decoding end through the transformation and quantization process. After the decoding end receives and parses the bitstream, it obtains the residual information through the steps of inverse transformation and inverse quantization, and the predicted image block predicted by the decoding end is superimposed with the residual information to obtain the reconstructed image block.

[0061] At present, the common video coding standards (such as H.266 / VVC) all adopt a block-based hybrid coding framework. Each frame in the video is divided into square maximum coding units (LCUlargest coding unit) of the same size (such as 128x128, 64x64, etc.). Each maximum coding unit can be divided into rectangular coding units (CU) according to rules. Coding units may also be divided into prediction units (PU), transform units (TU), etc. The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filter. The prediction module includes intra prediction and inter prediction. Inter prediction includes motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a frame of a video, the intra prediction method is used in video coding and decoding technology to eliminate the spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, the inter-frame prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving the coding efficiency.

[0062] The basic process of video codec is as follows. At the encoding end, a frame of image is divided into blocks, and intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block. The original image block of the current block is subtracted from the prediction block to obtain the residual block. The residual block is transformed and quantized to obtain the quantization coefficient matrix. The quantization coefficient matrix is ​​entropy encoded and output to the bitstream. At the decoding end, intra-frame prediction or inter-frame prediction is used for the current block to generate the prediction block of the current block. On the other hand, the bitstream is parsed to obtain the quantization coefficient matrix. The quantization coefficient matrix is ​​inversely quantized and inversely transformed to obtain the residual block. The prediction block and the residual block are added to obtain the reconstructed block. The reconstructed blocks constitute the reconstructed image, and the reconstructed image is loop-filtered based on the image or block to obtain the decoded image. The encoding end also needs to perform similar operations as the decoding end to obtain the decoded image. The decoded image can be used as a reference frame for inter-frame prediction for subsequent frames. The block division information determined by the encoding end, the mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., if necessary, need to be output to the bitstream. The decoding end determines the same block division information as the encoding end by parsing and analyzing the existing information, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., so as to ensure that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end. The decoded image obtained by the encoding end is also usually called a reconstructed image. The current block can be divided into prediction units during prediction, and the current block can be divided into transformation units during transformation. The division of prediction units and transformation units can be different. The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The current block can be the current coding unit (CU) or the current prediction unit (PU), etc.

[0063] JVET, the international video coding standards development organization, has established two exploratory experimental groups, one is an exploratory experiment based on neural network coding, and the other is an exploratory experiment beyond VVC, and has also established several corresponding expert discussion groups.

[0064] The above-mentioned exploratory experimental group that surpasses VVC aims to explore higher coding efficiency based on the latest codec standard H.266 / VVC with strict performance and complexity requirements. The coding method studied by this group is closer to VVC and can be called a traditional coding method. At present, the algorithm reference model performance of this exploratory experiment has surpassed the latest VVC reference model VTM by about 15%.

[0065] The method studied by the first exploratory experimental group is an intelligent coding method based on neural networks. Deep learning and neural networks are hot topics in all walks of life, especially in the field of computer vision, where methods based on deep learning often have overwhelming advantages. Experts from the JVET standard organization brought neural networks into the field of video encoding and decoding. With the powerful learning ability of neural networks, coding tools based on neural networks often have very high coding efficiency. In the early stages of the VVC standard formulation, many companies set their sights on coding tools based on deep learning, and proposed methods including intra-frame prediction methods based on neural networks, inter-frame prediction methods based on neural networks, and loop filtering methods based on neural networks. Among them, the coding performance of the loop filtering method based on neural networks is the most outstanding. After many meetings and explorations, the coding performance can reach more than 8%. At present, the coding performance of the loop filtering scheme based on neural networks studied by the first exploratory experimental group of the JVET conference was as high as 12%, reaching a level that can almost contribute to half a generation of coding performance.

[0066] The embodiment of the present application is improved on the basis of the current exploratory experiment of the JVET conference, and a loop filtering enhancement scheme based on a neural network is proposed. The following will first briefly introduce the loop filtering scheme based on a neural network in the current JVET conference, and then introduce the improvement method of the embodiment of the present application in detail.

[0067] At present, the exploration of loop filtering solutions based on neural networks at the JVET conference is mainly concentrated in two forms. The first is a solution with multiple models that can be switched within the frame; the second is a solution with non-switchable models within the frame. However, no matter which solution is used, the architecture of the neural network does not change much, and the tool is in the loop filtering of the traditional hybrid coding framework. Therefore, the basic processing unit of the two solutions is the coding tree unit, that is, the maximum coding unit size.

[0068] The biggest difference between the first scheme of multi-model intra-frame switchable and the second scheme of non-switchable models is that when encoding and decoding the current frame, the first scheme can switch the neural network model at will, while the second scheme cannot switch the neural network model. Taking the first scheme as an example, when encoding a frame of image, each coding tree unit has multiple optional candidate neural network models. The encoder selects which neural network model to use for the current coding tree unit for the best filtering effect, and then writes the neural network model index into the bitstream. That is, in this scheme, if the coding tree unit needs to be filtered, it is necessary to first transmit a coding tree unit-level usage flag, and then transmit the neural network model index. If filtering is not required, only a coding tree unit-level usage flag needs to be transmitted; after parsing the index value, the decoder loads the neural network model corresponding to the index into the current coding tree unit to filter the current coding tree unit.

[0069] Taking the second scheme as an example, when encoding a frame of image, the available neural network model for each coding tree unit in the current frame is fixed, and each coding tree unit uses the same neural network model, that is, the second scheme does not have a model selection process at the encoding end; the decoding end parses and obtains the usage flag of whether the current coding tree unit uses the neural network-based loop filter. If the usage flag is true, the pre-set model (the same as the encoding end) is used to filter the coding tree unit. If the usage flag is false, no additional operation is performed.

[0070] The first multi-model intra-frame switchable scheme has strong flexibility at the coding tree unit level, and can adjust the model according to local details, that is, local optimization to achieve a better global effect. Usually this scheme has more neural network models, and different neural network models are trained under different quantization parameters for the JVET general test conditions. At the same time, different coding frame types may also require different neural network models to achieve better results. Taking filter1 of the JVET-Y0080 scheme as an example, the filter uses up to 22 neural network models to cover different coding frame types and different quantization parameters, and model switching is performed at the coding tree unit level. This filter can provide up to 10% more coding performance based on the existing VVC.

[0071] The second type of intra-frame non-switchable model solution, we take JVET-Y0078 as an example. Although this solution has two neural network models overall, the model is not switched within the frame. This solution judges at the encoding end. If the current encoding frame type is an I frame, the neural network model corresponding to the I frame is imported, and only this model is used in the current frame; if the current encoding frame type is a B frame, the neural network model corresponding to the B frame is imported, and similarly only the neural network model corresponding to the B frame is used in this frame. This solution can provide 8.65% encoding performance based on the existing VVC. Although it is slightly lower than solution one, the overall performance is a coding efficiency that is almost impossible to achieve compared to traditional encoding tools.

[0072] Solution 1 has higher flexibility and higher encoding performance, but it has a fatal flaw in hardware implementation. At a recent JVET meeting, it was discussed that hardware experts are concerned about the code for intra-frame model switching. Switching the model at the coding tree unit level means that in the worst case, the decoder needs to reload the neural network model every time it processes a coding tree unit. Not to mention the complexity of hardware implementation, it is an additional burden on existing high-performance GPUs. At the same time, the existence of multiple models also means that a large number of parameters need to be stored, which is also a huge overhead burden on current hardware implementation.

[0073] In contrast, the second scheme, this neural network loop filter further explores the powerful generalization ability of deep learning, taking various information as input instead of simply taking the reconstructed sample as the input of the model. More information provides more help for the learning of the neural network, making the model's generalization ability better reflected and removing many unnecessary redundant parameters. The scheme was continuously updated until the last meeting, and a simplified low-complexity neural network model was developed for different test conditions and quantitative parameters. Compared with the first scheme, this saves the consumption of constantly reloading the model and the need to open up a larger storage space for a large number of parameters.

[0074] The above is a simple comparison of the advantages and disadvantages of the two solutions. Next, we will mainly introduce the architecture of the neural network solution itself.

[0075] The model architecture of Solution 1 takes JVET-Y0080 as an example. The simple network structure is as follows Figure 3B shown.

[0076] It can be seen that the main body of the network is composed of multiple ResBlocks, and the structure of ResBlocks is Figure 3A A single ResBlocks consists of multiple convolutional layers connected to CBAM layers. CBAM (Convolutional Blocks Attention Module) is an attention mechanism module that is mainly responsible for further extraction of detail features. In addition, ResBlocks also has a direct skip connection structure between input and output. There is also a skip connection on the entire large network framework, which connects the input reconstructed YUV information with the shuffled output.

[0077] The input of the network mainly includes reconstructed YUV (rec), predicted YUV (prde) and YUV with partition information (par). All inputs are concatenated after simple convolution and activation operations and then sent to the main body of the network. It is worth noting that the processing of YUV with partition information may be different in I frames and B frames. I frames need to input YUV with partition information, while B frames do not need to input YUV with partition information.

[0078] In summary, for any JVET parameter point of each I frame and B frame, Scheme 1 has a corresponding neural network parameter model. At the same time, because the three components of YUV are mainly composed of two channels, brightness and chrominance, they are different in color components.

[0079] The model architecture of Solution 2 takes JVET-Y0078 as an example. The simple network structure is as follows Figure 4 shown.

[0080] It can be seen that the main structure of the network is basically the same as that of Scheme 1. The difference is that Scheme 2 adds quantization parameter information as an additional input compared to Scheme 1. Scheme 1 loads different neural network parameter models according to different quantization parameter information to achieve more flexible processing and more efficient encoding effects, while Scheme 2 uses quantization parameter information as the input of the network to improve the generalization ability of the neural network, so that the model can adapt to different quantization parameter conditions and provide good filtering performance.

[0081] like Figure 4 As shown in the figure, there are two quantization parameters entering the network as input, one is BaseQP and the other is SliceQP. BaseQP here indicates the sequence-level quantization parameter set by the encoder when encoding the video sequence, that is, the quantization parameter point required by the JVET pass test, and is also the parameter used to select the neural network model in Scheme 1. SliceQP is the quantization parameter of the current frame. The quantization parameter of the current frame can be different from the sequence level. This is because in the video encoding process, the quantization conditions of the B frame are different from those of the I frame, and the quantization parameters are different at different time domain levels. Therefore, SliceQP is generally different from BaseQP in the B frame. Therefore, in the design of the JVET-Y0078 scheme, the input of the I frame neural network model only requires SliceQP, while the neural network model of the B frame requires BaseQP and SliceQP as input at the same time.

[0082] There is another difference between Scheme 2 and Scheme 1. The output of the Scheme 1 model generally does not require additional processing. That is, if the output of the model is residual information, it is superimposed with the reconstructed sample of the current coding tree unit as the output of the neural network-based loop filter tool; if the output of the model is a complete reconstructed sample, the model output is the output of the neural network-based loop filter tool. The output of Scheme 2 generally needs to be scaled. Taking the residual information output by the model as an example, the model infers and outputs the residual information of the current coding tree unit. The residual information is scaled and then superimposed with the reconstructed sample information of the current coding tree unit. This scaling factor is obtained by the encoding end and needs to be written into the code stream to the decoding end.

[0083] It is precisely because the quantitative parameters are used as input for additional information that the reduction in the number of models can be achieved and has become the most popular solution at the current JVET conference.

[0084] In addition, the general neural network-based loop filtering solution may not be exactly the same as the above two solutions. The specific details of the solution may be different, but the main ideas are basically the same. For example, the different details of Solution 2 can be reflected in the design of the neural network architecture, such as the convolution size of ResBlocks, the number of convolution layers, and whether the attention module is included, etc., and can also be reflected in the input of the neural network. The input can even have more additional information, such as the boundary strength value of the deblocking filter.

[0085] Solution 1 can switch the neural network model at the coding tree unit level, and these different neural network models are trained according to different BaseQPs. By trying these different neural network models on the encoder side, the network model with the lowest rate-distortion cost is the optimal network model for the current coding tree unit. Through the use of the flag bit and network model index information at the coding tree unit level, the decoder side can use the same network model as the encoder side for filtering. Solution 2 uses the method of inputting quantization parameters to achieve good coding performance without switching models, which preliminarily solves the concerns about hardware implementation. However, in general, the performance of Solution 2 is still not as good as Solution 1. The main defect is the switching of BaseQP. Solution 2 has no flexibility, and the encoder side has few choices, resulting in suboptimal performance.

[0086] The present application provides a video encoding system. Figure 5The video coding system 10 is a schematic diagram of the composition structure of the embodiment of the present application. The video coding system 10 includes: a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, a coding unit 109 and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement DBF filtering / SAO filtering / ALF filtering, and the coding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (Context-based Adaptive Binary Arithmetic Coding, CABAC). For the input original video signal, the coding tree unit (Coding Tree A video coding block can be obtained by dividing the video coding block into a video coding unit (CTU), and then the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transform and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain, and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block; specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding of the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is a process of generating a motion vector, and the motion vector can estimate the motion of the video coding block, and then the motion compensation unit 104 calculates the motion vector based on the motion vector determined by the motion estimation unit 105. The intra prediction unit 103 is used to perform motion compensation; after determining the intra prediction mode, the intra prediction unit 103 is also used to provide the selected intra prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated motion vector data to the encoding unit 109; in addition, the inverse transform and inverse quantization unit 106 is used to reconstruct the video coding block, reconstruct the residual block in the pixel domain, and the reconstructed residual block is removed by the filter control analysis unit 107 and the filtering unit 108. The block effect artifacts are then removed, and the reconstructed residual block is then added to a predictive block in the frame of the decoded image buffer unit 110 to generate a reconstructed video coding block; the encoding unit 109 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra prediction mode and output the code stream of the video signal; and the decoded image buffer unit 110 is used to store the reconstructed video coding block for prediction reference.As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .

[0087] The embodiment of the present application provides a video decoding system. Figure 6 The video decoding system 20 is a schematic diagram of the composition structure of the embodiment of the present application. The video decoding system 20 includes: a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra-frame prediction unit 203, a motion compensation unit 204, a filtering unit 205 and a decoded image cache unit 206, etc., wherein the decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement DBF filtering / SAO filtering / ALF filtering. The input video signal is Figure 3A After the encoding process, the code stream of the video signal is output; the code stream is input into the video decoding system 20, and first passes through the decoding unit 201 to obtain the decoded transform coefficient; the transform coefficient is processed by the inverse transform and inverse quantization unit 202 to generate a residual block in the pixel domain; the intra-frame prediction unit 203 can be used to generate prediction data of the current video decoding block based on the determined intra-frame prediction mode and the data of the previously decoded block from the current frame or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses the prediction information The decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 202 and the corresponding predictive block generated by the intra-frame prediction unit 203 or the motion compensation unit 204; the decoded video signal is passed through the filtering unit 205 to remove the block effect artifacts, which can improve the video quality; the decoded video block is then stored in the decoded image cache unit 206, which stores the reference image used for subsequent intra-frame prediction or motion compensation, and is also used for the output of the video signal, that is, the restored original video signal is obtained.

[0088] It should be noted that the filtering method provided in the embodiment of the present application can be applied in the following aspects: Figure 5 The filter unit 108 shown in FIG. 1 (indicated by a black bold box) can also be used in Figure 6 The filtering unit 205 shown is represented by a black bold box. That is to say, the filtering method in the embodiment of the present application can be applied to a video encoding system (referred to as "encoder"), a video decoding system (referred to as "decoder"), and even a video encoding system and a video decoding system at the same time, but no limitation is made here.

[0089] The embodiment of the present application can be implemented on the basis of the above-mentioned solution of not switching models within a frame. The main idea is to use the variability of the input specification to provide more possibilities for the encoder. The input specification of the neural network filter model includes quantization parameters, and the quantization parameters include sequence-level quantization parameter values ​​(BaseQP) or frame-level quantization parameter values ​​(SliceQP). The BaseQP and SliceQP used as input are adjusted so that the codec end has more options to try, thereby improving the codec efficiency.

[0090] The present application embodiment provides a filtering method, which is applied to a decoder, such as Figure 7 As shown, the method may include:

[0091] S101, parsing the bitstream to obtain a frame-level usage identification bit based on a neural network filtering model;

[0092] In the embodiment of the present application, at the decoding end, the decoder uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block, and at the same time, the decoder parses the code stream to obtain a quantization coefficient matrix, dequantizes and inversely transforms the quantization coefficient matrix to obtain a residual block, adds the prediction block and the residual block to obtain a reconstructed block, and the reconstructed block constitutes a reconstructed image. The decoder performs loop filtering on the reconstructed image based on the image or block to obtain a decoded image.

[0093] It should be noted that, since the original image can be divided into CTU (coding tree unit), or CTU can be divided into CU; therefore, the filtering method of the embodiment of the present application can not only be applied to CU-level loop filtering (the block division information at this time is CU division information), but also can be applied to CTU-level loop filtering (the block division information at this time is CTU division information), and the embodiment of the present application does not make specific limitations.

[0094] The embodiment of the present application will be described by taking CTU as a block as an example.

[0095] In an embodiment of the present application, during the process of loop filtering the reconstructed image of the current frame, the decoder can parse the bitstream to first parse out the sequence-level permission flag (sps_nnlf_enable_flag). The sequence-level permission flag is a switch for whether the filtering function is turned on for the entire video sequence to be processed. When the sequence-level permission flag indicates permission, the decoder parses the syntax elements of the current frame to obtain the frame-level usage flag based on the neural network filtering model. The frame-level usage flag is used to indicate whether the current frame uses filtering. When the frame-level usage flag indicates use, it indicates that some or all blocks in the current frame need to be filtered, and when the frame-level usage flag indicates unused, it indicates that all blocks in the current frame do not need to be filtered. The decoder can continue to traverse other filtering methods to output a complete reconstructed image.

[0096] It should be noted that, by default, the relevant syntax elements are set to an initial value or a negative state.

[0097] It should be noted that the frame-level usage identification bit based on the neural network filtering model is not limited to a limited form, and can be a letter or a symbol, etc., and the embodiments of the present application do not impose any restrictions.

[0098] Exemplarily, the value of the frame-level usage flag based on the neural network filtering model can use 1 to represent use and 0 to represent unuse. The embodiment of the present application does not limit the expression form and meaning of the value of the frame-level usage flag.

[0099] In some embodiments of the present application, the frame level usage flag for the current frame may be embodied by one or more flags. When multiple flags are embodied, different color components of the current frame may each correspond to a respective frame level usage flag, that is, the frame level usage flag of the color component. The frame level usage flag of a color component indicates whether filtering is required in the block of the current frame under the color component.

[0100] It should be noted that the decoder traverses the frame level of each color component of the current frame and uses the identification bit to determine whether to perform filtering processing on the blocks under each color component.

[0101] S102, when the frame level usage flag indicates usage, obtaining a frame level switch flag and a frame level quantization parameter adjustment flag; the frame level switch flag is used to determine whether all blocks in the current frame are filtered;

[0102] In the embodiment of the present application, the decoder determines the frame level of the current frame using the flag characterization, and can also parse the frame level switch flag and the frame level quantization parameter adjustment flag from the bit stream. The frame level switch flag is used to determine whether each block in the current frame is filtered.

[0103] The blocks here may be coding tree units of the current frame.

[0104] The frame-level switch flag may correspond to each color component and may also indicate whether all coding tree units under the current color component are filtered using a loop filtering technique based on a neural network.

[0105] In an embodiment of the present application, if the frame-level switch flag is on, it means that all coding tree units under the current color component are filtered using the neural network-based loop filtering technology, that is, the coding tree unit-level usage flag of all coding tree units of the current frame under the color component is automatically set to use; if the frame-level switch flag is not on, it means that some coding tree units under the current color component use the neural network-based loop filtering technology, and some coding tree units do not use the neural network-based loop filtering technology. If the frame-level switch flag is not on, it is necessary to further parse the coding tree unit-level usage flag of all coding tree units of the current frame under the color component.

[0106] It should be noted that, in the embodiment of the present application, when the coding tree unit is used as a block, the coding tree unit level usage flag can also be understood as the block level usage flag.

[0107] Exemplarily, the value of the frame-level switch flag can be 1 to represent being enabled, and 0 to represent being disabled. The embodiment of the present application does not limit the form and meaning of the value of the frame-level switch flag.

[0108] In an embodiment of the present application, the frame-level quantization parameter adjustment flag indicates whether the quantization parameters (BaseQP and SliceQP) of the current frame have been adjusted. If the frame-level quantization parameter adjustment flag indicates that it is used, it indicates that the quantization parameter of the current frame has been adjusted, and it is necessary to continue to obtain the analysis and obtain the frame-level quantization parameter adjustment index for subsequent filtering processing. If the frame-level quantization parameter adjustment flag indicates that it is not used, it indicates that the quantization parameter of the current frame has been adjusted, and the code stream can continue to be used. The quantization parameter parsed in the code stream is used to implement the subsequent processing process.

[0109] Exemplarily, the value of the frame-level quantization parameter adjustment flag can be 1 to represent use, and 0 to represent unuse. The embodiment of the present application does not limit the expression form and meaning of the value of the frame-level quantization parameter adjustment flag.

[0110] In some embodiments of the present application, the decoder can choose whether to adjust the quantization parameter of the current frame according to the different coded frame types. The quantization parameter needs to be adjusted for the first type, and the quantization parameter is not adjusted for the second type frame, wherein the second type frame is a type frame other than the first type frame. Then, during decoding, the decoder can obtain the frame-level quantization parameter adjustment flag parsed in the code stream when the current frame can be filtered and the current frame is a first type frame.

[0111] In some embodiments of the present application, after obtaining the frame-level usage flag based on the neural network filtering model and before obtaining the adjusted frame-level quantization parameter, the decoder obtains the frame-level switch flag and the frame-level quantization parameter adjustment flag when the frame-level usage flag represents usage and the current frame is a first type frame.

[0112] It should be noted that, in the embodiment of the present application, the first type of frame may be a B frame or a P frame, and the embodiment of the present application does not limit this.

[0113] It should be noted that the decoder can parse and obtain the frame-level switch flag and the frame-level quantization parameter adjustment flag at the same time.

[0114] S103, when the frame-level switch flag indicates that it is turned on, and the frame-level quantization parameter adjustment flag indicates that it is used, obtaining an adjusted frame-level quantization parameter;

[0115] After parsing and obtaining the frame level switch flag and the frame level quantization parameter adjustment flag, the decoder obtains the adjusted frame level quantization parameter when the frame level switch flag indicates that it is turned on and the frame level quantization parameter adjustment flag indicates that it is used.

[0116] It should be noted that when the frame-level switch flag is turned on, it indicates that there is a situation where the coding tree unit needs to be filtered under the current color component. In this case, when the frame-level quantization parameter adjustment flag is used, the adjusted frame-level quantization parameter needs to be obtained for use in filtering the coding tree unit level.

[0117] In the embodiment of the present application, the frame-level quantization parameter adjustment flag is characterized by use, and the decoder can obtain the frame-level quantization adjustment index from the bit stream, and determine the adjusted quantization parameter based on the frame-level quantization adjustment index.

[0118] In some embodiments of the present application, the decoder determines a frame-level quantization bias parameter based on a frame-level quantization parameter adjustment index obtained in a bitstream; and determines an adjusted frame-level quantization parameter based on the obtained frame-level quantization parameter and the frame-level quantization bias parameter.

[0119] Here, the adjustment amplitudes of all the coding tree units of the current frame are the same, that is, the quantization parameter inputs of all the coding tree units are the same.

[0120] It should be noted that when the encoder is encoding, if it is determined that the quantization parameter needs to be adjusted, the serial number corresponding to the frame-level quantization bias parameter will be transmitted to the bitstream as the frame-level quantization adjustment index. The decoder stores the correspondence between the serial number and the quantization bias parameter. In this way, the decoder can determine the frame-level quantization bias parameter based on the frame-level quantization adjustment index. The decoder uses the frame-level quantization bias parameter to adjust the frame-level quantization parameter, and the adjusted frame-level quantization parameter can be obtained. The quantization parameter can be obtained from the bitstream.

[0121] Exemplarily, if the frame-level quantization parameter index adjustment flag of the current frame is used, the quantization parameter is adjusted according to the frame-level quantization parameter adjustment index. For example, if the quantization parameter adjustment index points to offset1, BaseQP is superimposed with the offset parameter offset1 to obtain BasseQPFinal, which replaces BaseQP as the quantization parameter of all coding tree units of the current frame and is input into the network model.

[0122] In some embodiments of the present application, the decoder obtains the adjusted frame-level quantization parameter from the bitstream.

[0123] That is to say, the encoder can directly transmit the adjusted quantization parameter to the decoder through the bit stream for use by the decoder during decoding.

[0124] S104: Filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block.

[0125] After the decoder obtains the adjusted frame-level quantization parameters, since the frame-level switch flag indicates that it is turned on, the decoder can filter all coding tree units of the current frame. For the filtering of a coding tree unit, it is necessary to traverse the filtering processing of each color component before decoding the next coding tree unit.

[0126] In the embodiment of the present application, a neural network filtering model is used to filter the current block of the current frame with the adjusted frame-level quantization parameter to obtain the first residual information of the current block, where the current block is the current coding tree unit.

[0127] In an embodiment of the present application, the decoder obtains the reconstruction value of the current block before filtering the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block. The neural network filtering model is used to filter the reconstruction value of the current block and the adjusted frame-level quantization parameter to obtain the first residual information of the current block to complete the filtering of the current block.

[0128] In some embodiments of the present application, the decoder filters the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model, and obtains at least one of the predicted value, block partition information and deblocking filter boundary strength of the current block, as well as the reconstructed value of the current block before obtaining the first residual information of the current block.

[0129] In some embodiments of the present application, the decoder uses a neural network filtering model to filter the prediction value of the current block, at least one of the block partition information and the deblocking filter boundary strength, the reconstructed value of the current block, and the adjusted frame-level quantization parameter to obtain the first residual information of the current block to complete the filtering of the current block.

[0130] It should be noted that, during the filtering process, the input parameters input to the neural network filtering model may include: the predicted value of the current block, block partition information, deblocking filter boundary strength, the reconstruction value of the current block, and the adjusted frame-level quantization parameter (or quantization parameter), and the present application does not limit the type of information of the input parameters. However, the predicted value of the current block, the block partition information, and the deblocking filter boundary strength are not necessarily required every time, and need to be determined according to the actual situation.

[0131] In some embodiments of the present application, after the decoder filters the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filter model to obtain the first residual information of the current block, the decoder may also obtain a second residual scaling factor in the bitstream; based on the second residual scaling factor, scale the first residual information of the current block to obtain the first target residual information;

[0132] A first target reconstruction value of the current block is determined based on the first target residual information and the reconstruction value of the current block.

[0133] It should be noted that when the encoder obtains the residual information, it can use the second residual scaling factor to scale the first residual information to obtain the first residual information. Therefore, the decoder needs to scale the first residual information of the current block based on the second residual scaling factor to obtain the first target residual information, and determine the first target reconstruction value of the current block based on the first target residual information and the reconstruction value of the current block. However, if the encoder does not use the residual factor during encoding, but also needs to input the quantization parameter (or the adjusted quantization parameter) during filtering, the filtering method provided in the embodiment of the present application is also applicable, except that the residual information does not need to be scaled using the residual factor.

[0134] It should be noted that each color component has corresponding residual information and residual factors.

[0135] It can be understood that the decoder can determine whether the quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameters, thereby improving decoding efficiency.

[0136] In some embodiments of the present application, during the filtering process of encoding and decoding, part of the data in the input parameters of the neural network filtering model may be adjusted using the aforementioned principles before filtering.

[0137] In an embodiment of the present application, at least one of the quantization parameter, the prediction value of the current block, the block partitioning information and the deblocking filter boundary strength in the input parameters can be adjusted and processed, and the embodiment of the present application is not limited.

[0138] In some embodiments of the present application, when the frame-level usage flag indicates usage, a frame-level switch flag and a frame-level input parameter adjustment flag are obtained; the frame-level input parameter adjustment flag indicates whether any one of the parameters of the prediction value, the block partition information, and the deblocking filter boundary strength is adjusted;

[0139] When the frame-level switch flag indicates that it is turned on and the frame-level input parameter adjustment flag indicates that it is used, obtaining the adjusted block-level input parameters;

[0140] Based on the adjusted block-level input parameters, the acquired frame-level quantization parameters and the neural network filtering model, the current block of the current frame is filtered to obtain third residual information of the current block.

[0141] When the frame-level switch flag indicates that it is not turned on, it is necessary to obtain the block-level usage flag, and then determine whether to filter the current block. When it is determined that filtering is required, the decoder can filter according to the adjusted block-level input parameters.

[0142] It can be understood that the decoder can adjust the flag based on the frame-level input parameter to determine whether the input parameters of the input neural network filter model need to be adjusted, thereby achieving flexible selection and diverse change processing of the input parameters, thereby improving decoding efficiency.

[0143] In some embodiments of the present application, a filtering method provided by an embodiment of the present application may further include:

[0144] S101, parsing the bitstream to obtain a frame-level usage identification bit based on a neural network filtering model;

[0145] S102, when the frame level usage flag indicates usage, obtaining a frame level switch flag and a frame level quantization parameter adjustment flag; the frame level switch flag is used to determine whether all blocks in the current frame are filtered;

[0146] It should be noted that S101 and S102 have been described above and will not be repeated here.

[0147] Among them, the current block may be a coding tree unit, which is not limited in the embodiment of the present application.

[0148] S105, when the frame-level switch flag indicates that it is not turned on, obtaining a block-level use flag;

[0149] S106, when the block-level usage flag indicates that any color component of the current block is used, and the frame-level quantization parameter adjustment flag indicates that it is used, obtaining an adjusted frame-level quantization parameter;

[0150] In the embodiment of the present application, when the frame-level switch flag indicates that it is not turned on, it is necessary to obtain the block-level usage flag from the bitstream.

[0151] It should be noted that the block-level usage flag of the current block includes the block-level usage flag corresponding to each color component.

[0152] In an embodiment of the present application, when the block-level usage flag indicates that any color component of the current block is used, and the frame-level quantization parameter adjustment flag indicates that it is used, the adjusted frame-level quantization parameter is obtained; wherein, the process of obtaining the adjusted frame-level quantization parameter may be consistent with the aforementioned implementation means, and will not be repeated here.

[0153] It should be noted that, for the current block, as long as any color component is represented by a block-level usage flag, decoding needs to filter the current block to obtain the residual information corresponding to each color component. Therefore, for the current block, as long as any color component is represented by a block-level usage flag, it is necessary to obtain the adjusted frame-level quantization parameter for use in filtering.

[0154] S107 . Filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block.

[0155] In an embodiment of the present application, the decoder filters the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block.

[0156] The first residual information includes the residual information corresponding to each color component. The decoder determines the reconstruction value of the color component of the current block according to the block-level usage identifier corresponding to each color component. If the block-level usage identifier corresponding to the color component is used, the target reconstruction value corresponding to the color component is the sum of the reconstruction value of the color component of the current block and the residual information of the filtering output under the color component. If the block-level usage identifier corresponding to the color component is used, the target reconstruction value corresponding to the color component is the reconstruction value of the color component of the current block.

[0157] Exemplarily, if the coding tree unit level usage flags of all color components of the current coding tree unit are not all used, the current coding tree unit is filtered using a neural network-based loop filtering technology, and the reconstructed sample YUV of the current coding tree unit, the predicted sample YUV of the current coding tree unit, the division information YUV of the current coding tree unit, and the quantization parameter information are used as input to obtain the residual information of the current coding tree unit. The quantization parameter information is adjusted according to the frame-level quantization parameter adjustment flag and the frame-level quantization parameter adjustment index. The residual information is scaled, and the residual scaling factor has been obtained from the aforementioned parsed code stream. The scaled residual is superimposed on the reconstructed sample to obtain the reconstructed sample YUV after the neural network loop filtering. According to the coding tree unit usage flag of each color component of the current coding tree unit, the reconstructed sample is selected as the output of the neural network-based loop filtering technology. If the coding tree unit usage flag of the corresponding color component is used, the reconstructed sample of the corresponding color component after the neural network loop filtering is used as the output; otherwise, the reconstructed sample that has not been filtered by the neural network loop filtering is used as the output of the color component. After traversing all the coding tree units of the current frame, the neural network-based loop filtering module ends.

[0158] It can be understood that the decoder can determine whether the quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameters, thereby improving decoding efficiency.

[0159] In some embodiments of the present application, after the decoder obtains the block-level usage flag, it obtains the block-level quantization parameter adjustment flag;

[0160] When the block-level usage flag indicates that any color component of the current block is used, and the block-level quantization parameter adjustment flag indicates that it is used, the adjusted block-level quantization parameter is obtained; based on the adjusted block-level quantization parameter and the neural network filtering model, the current block of the current frame is filtered to obtain the second residual information of the current block.

[0161] In some embodiments of the present application, the decoder determines a block-level quantization offset parameter based on a block-level quantization parameter index obtained in a bitstream; and determines an adjusted block-level quantization parameter according to the obtained block-level quantization parameter and the block-level quantization offset parameter.

[0162] It should be noted that the block-level quantization parameter obtained by the decoder can be the block-level quantization parameter index corresponding to the block-level quantization bias parameter parsed from the bitstream, and the block-level quantization bias parameters corresponding to different blocks are superimposed based on the quantization parameter to obtain the block-level quantization parameter corresponding to the current block. Then, based on the adjusted block-level quantization parameter and the neural network filter model, the current block of the current frame is filtered to obtain the second residual information of the current block.

[0163] In the embodiment of the present application, the adjustments between different coding tree units may be different, that is, the quantization parameter inputs of different coding tree units may be different.

[0164] In some embodiments of the present application, after the decoder obtains the block-level usage flag, when the block-level usage flag represents the use of any color component of the current block, the block-level quantization parameter corresponding to the current block is obtained; based on the adjusted block-level quantization parameter and the neural network filtering model, the current block of the current frame is filtered to obtain the second residual information of the current block.

[0165] It should be noted that each identification bit in the present application can be 1 for used or allowed state, and 0 for unused or not allowed state, and the embodiments of the present application are not limited thereto.

[0166] It should be noted that the block-level quantization parameter corresponding to the current block can be parsed from the bitstream.

[0167] In some embodiments of the present application, the decoder filters the current block of the current frame based on the adjusted block-level quantization parameter and the neural network filter model, obtains the second residual information of the current block, and then obtains the second residual scaling factor in the code stream; scales the second residual information of the current block based on the second residual scaling factor to obtain the second target residual information; when the block-level usage flag characterization is used, the second target reconstruction value of the current block is determined based on the second target residual information and the reconstruction value of the current block. When the block-level usage flag characterization is not used, the reconstruction value of the current block is determined as the second target reconstruction value.

[0168] It should be noted that the decoding end continues to traverse other loop filtering methods and outputs a complete reconstructed image after completion.

[0169] It can be understood that the decoder can determine whether the block-level quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of block-level quantization parameters, and the adjustment amplitude of each block can be different, thereby improving decoding efficiency.

[0170] The present application embodiment provides a filtering method, which is applied to an encoder, such as Figure 8 As shown, the method may include:

[0171] S201, obtaining a sequence-level permission-to-use identification bit;

[0172] S202, when the sequence level allows the use of a flag to represent the permission, obtaining an original value of a current block in a current frame, a reconstructed value of the current block, and a frame-level quantization parameter;

[0173] S203, performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a first reconstruction value;

[0174] In an embodiment of the present application, the encoder traverses intra-frame or inter-frame prediction to obtain the prediction block of each coding unit. The residual of the coding unit can be obtained by subtracting the original image block from the prediction block. The residual obtains the frequency domain residual coefficient by various transformation modes, and then quantizes and inversely quantizes. After inverse transformation, the distortion residual information is obtained. The distortion residual information is superimposed with the prediction block to obtain the reconstructed block. After the image is encoded, the loop filter module filters the image with the coding tree unit level as the basic unit. The embodiment of the present application describes the coding tree unit as a block, but the block is not limited to CTU, but can also be CU, which is not limited by the embodiment of the present application. The encoder obtains the sequence-level permission flag based on the neural network filtering model, that is, sps_nnlf_enable_flag. If the sequence-level permission flag is allowed, the use of the neural network-based loop filtering technology is allowed; if the sequence-level permission flag is not allowed, the use of the neural network-based loop filtering technology is not allowed. The sequence-level permission flag needs to be written into the bitstream when encoding the video sequence.

[0175] In an embodiment of the present application, when the sequence level allows the use of an identification bit to indicate permission, the encoding end tries the loop filtering technology based on the neural network filtering model, and the encoder obtains the original value of the current block in the current frame, the reconstructed value of the current block, and the frame-level quantization parameter; if the sequence level based on the neural network filtering model allows the use of an identification bit to indicate that it is not allowed, the encoding end does not try the loop filtering technology based on the neural network; it continues to try other loop filtering tools, such as LF filtering, and outputs a complete reconstructed image after completion.

[0176] In an embodiment of the present application, for the current frame, the current block is filtered and estimated based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine the first estimated residual information; determine the first residual scaling factor; use the first residual scaling factor to scale the first estimated residual value to obtain the first scaled residual information; combine the first scaled residual information with the reconstruction value of the current block to determine the first reconstruction value.

[0177] In an embodiment of the present application, before the encoder determines the first residual scaling factor, it obtains, for the current frame, a prediction value of the current block, at least one of the block partition information and the deblocking filter boundary strength, and a reconstructed value of the current block; and uses a neural network filtering model to perform filtering estimation on the prediction value of the current block, at least one of the block partition information and the deblocking filter boundary strength, the reconstructed value of the current block, and a frame-level quantization parameter to obtain a first estimated residual information of the current block.

[0178] It should be noted that the input parameters input into the neural network filtering model can be determined according to actual conditions, and the embodiments of the present application are not limited thereto.

[0179] S204, performing rate-distortion cost estimation based on the first reconstructed value and the original value of the current block to obtain the rate-distortion cost of the current block, and traversing the current frame to determine a first rate-distortion cost of the current frame;

[0180] In an embodiment of the present application, after obtaining the first reconstruction value of the current block, the encoder estimates the rate-distortion cost based on the first reconstruction value and the original value of the current block to obtain the rate-distortion cost of the current block, and continues encoding the next block until the rate-distortion costs of all blocks of the current frame are obtained, then the rate-distortion costs of all blocks are added together to obtain the first rate-distortion cost of the current frame.

[0181] Exemplarily, the encoding end attempts a loop filtering technique based on a neural network, and uses the reconstructed sample YUV, predicted sample YUV, YUV with partition information, and quantization parameters (BaseQP and SliceQP) of the current coding tree unit to input into the neural network filtering model for reasoning. The neural network filtering model outputs the estimated residual information after filtering of the current coding tree unit, and scales the estimated residual information. The scaling factor in the scaling operation is calculated and obtained based on the original image samples of the current frame, the reconstructed samples that have not been filtered by the neural network loop, and the reconstructed samples that have been filtered by the neural network loop. The scaling factors of different color components are different and when necessary, they are all written into the code stream and transmitted to the decoding end. The encoder superimposes the scaled residual information on the reconstructed samples that have not been filtered by the neural network loop for output. The encoder calculates the rate-distortion cost value based on the coding tree unit samples after the neural network loop filtering and the original image samples of the coding tree unit, which is recorded as the first rate-distortion cost of the current frame, costNN.

[0182] S205, based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and the reconstruction value of the current block in the current frame, perform at least one filtering estimation on the current frame to determine at least one second rate-distortion cost of the current frame;

[0183] The encoder attempts to perform at least one filter estimation by changing at least one input parameter input to the neural network filter model to obtain at least one second rate-distortion cost (costOffset) of the current frame.

[0184] It should be noted that the input parameter may be at least one of the predicted value of the current block, block partition information and deblocking filter boundary strength, the reconstructed value of the current block, and the frame-level quantization parameter, and may also include other information, which is not limited in the embodiments of the present application. The encoder may adjust any one of the frame-level quantization parameter, the predicted value of the current block, the block partition information and the deblocking filter boundary strength to perform filter estimation, which is not limited in the embodiments of the present application.

[0185] In some embodiments of the present application, when the sequence level allows the use of a flag to indicate permission, obtaining a prediction value of a current block, at least one of block partition information and a deblocking filter boundary strength, a reconstructed value of the current block, and a frame-level quantization parameter;

[0186] Performing filtering estimation on the current block based on the prediction value of the current block, at least one of the block partitioning information and the deblocking filter boundary strength, the neural network filter model, the reconstruction value of the current block and the frame-level quantization parameter, to determine a sixth reconstruction value;

[0187] Performing a rate-distortion cost estimation based on the sixth reconstructed value and the original value of the current block to obtain a rate-distortion cost of the current block, and traversing the current frame to determine a seventh rate-distortion cost of the current frame;

[0188] Based on at least one of the prediction value of the current block, the block partition information and the deblocking filter boundary strength, the neural network filter model, at least one frame-level input bias parameter, and the reconstruction value of the current block in the current frame, perform at least one filter estimation on the current frame to determine at least one eighth rate-distortion cost of the current frame;

[0189] Based on the first rate-distortion cost and at least one eighth rate-distortion cost, a frame-level input parameter adjustment flag is determined.

[0190] When the input parameter is a frame-level quantization parameter, the frame-level input parameter adjustment flag may be understood as a frame-level quantization parameter adjustment flag.

[0191] It can be understood that the encoder can adjust the flag based on the frame-level input parameter to determine whether the input parameters of the input neural network filter model need to be adjusted, thereby achieving flexible selection and diverse change processing of the input parameters, thereby improving the encoding efficiency.

[0192] Exemplarily, the adjustment of the frame-level quantization parameter is implemented as follows:

[0193] The encoder obtains the i-th frame-level quantization bias parameter, adjusts the frame-level quantization parameter based on the i-th frame-level quantization bias parameter, and obtains the i-th adjusted frame-level quantization parameter; i is a positive integer greater than or equal to 1; based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter, the current block is filtered and estimated to obtain the i-th second reconstruction value; based on the i-th second reconstruction value and the original value of the current block, the rate-distortion cost is estimated, all blocks of the current frame are traversed, and the i-th second rate-distortion cost is obtained, and the i+1th filtering estimation is continued based on the i+1th frame-level quantization bias parameter until it is completed at least once, thereby determining at least one second rate-distortion cost of the current frame.

[0194] In an embodiment of the present application, the encoder estimates the rate-distortion cost based on the i-th second reconstruction value and the original value of the current block, traverses all blocks of the current frame, adds the rate-distortion costs of all blocks to obtain the i-th second rate-distortion cost, and continues to perform the i+1-th filtering estimation based on the i+1-th frame-level quantization bias parameter until the filtering estimation of all blocks is completed and the first rate-distortion cost of the current frame is obtained, until at least one round of filtering is completed and at least one second rate-distortion cost of the current frame is obtained.

[0195] In an embodiment of the present application, the encoder performs filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter, and the implementation of obtaining the i-th second reconstruction value includes: performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter, respectively, to obtain the i-th second estimated residual information; determining the i-th second residual scaling factor corresponding to the i-th adjusted frame-level quantization parameter; using the i-th second residual scaling factor, scaling the i-th second estimated residual information to obtain the i-th second scaled residual information; combining the i-th second scaled residual information with the reconstruction value of the current block to determine the i-th second reconstruction value.

[0196] In some embodiments of the present application, the encoder may also obtain a prediction value of the current block, at least one of block partition information and deblocking filter boundary strength, and a reconstructed value of the current block; and use a neural network filtering model to perform frame-level filtering estimation on the prediction value of the current block, at least one of block partition information and deblocking filter boundary strength, the reconstructed value of the current block, and the i-th adjusted frame-level quantization parameter to obtain the i-th second estimated residual information of the current block.

[0197] In some embodiments of the present application, the encoder may choose whether to adjust the quantization parameter of the current frame according to different encoding frame types. The quantization parameter needs to be adjusted for the first type, and the quantization parameter does not need to be adjusted for the second type frame, wherein the second type frame is a type frame other than the first type frame. Then, during encoding, the encoder can adjust the frame-level quantization parameter when the current frame is the first type frame to perform filter estimation.

[0198] In some embodiments of the present application, when the current frame is a first type frame, based on a neural network filtering model, at least one frame-level quantization bias parameter, a frame-level quantization parameter, and a reconstructed value of a current block in the current frame, at least one filtering estimation is performed on the current frame to determine at least one second rate-distortion cost of the current frame.

[0199] It should be noted that, in the embodiment of the present application, the first type of frame may be a B frame or a P frame, and the embodiment of the present application does not limit this.

[0200] Exemplarily, the encoder may adjust the BaseQP and SliceQP used as input, so that the encoding end has more options to try, thereby improving encoding efficiency.

[0201] The above adjustment of BaseQP and SliceQP includes unified adjustment of all coding tree units in a frame, and also includes separate adjustment of coding tree units. For unified adjustment of all coding tree units in a frame, that is, the current frame can be adjusted regardless of whether it is an I frame or a B frame, and the adjustment amplitude of all coding tree units in the current frame is the same, that is, the quantization parameter input of all coding tree units is the same; for separate adjustment of coding tree units, that is, the current frame can be adjusted regardless of whether it is an I frame or a B frame, and the adjustment amplitude of all coding tree units in the current frame can be selected by rate-distortion optimization at the encoding end according to the current coding tree unit, and the adjustment between different coding tree units can be different, that is, the quantization parameter input of different coding tree units can be different.

[0202] It can be understood that the encoder can determine whether the block-level quantization parameters of the input neural network filter model need to be adjusted based on the block-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of block-level quantization parameters, thereby improving encoding efficiency.

[0203] S206: Determine a frame-level quantization parameter adjustment flag based on the first rate-distortion cost and at least one second rate-distortion cost.

[0204] In an embodiment of the present application, the encoder may determine a frame-level quantization parameter adjustment flag based on a first rate-distortion cost and at least one second rate-distortion cost, that is, determine whether the frame-level quantization parameter needs to be adjusted during filtering.

[0205] Exemplarily, the above adjustment of BaseQP and SliceQP can be controlled by a frame-level flag, where there is at least one frame-level flag. For example, different frame-level quantization parameter adjustment flags can be set for different color components, a frame-level quantization parameter adjustment flag can be set for the brightness component, and a frame-level quantization parameter adjustment flag can also be set for the chrominance component. The expansion of the frame-level quantization parameter adjustment flag can also be to use one or more flags to identify whether all coding tree units in the current frame need to make quantization parameter adjustments, or whether all coding tree units have the same adjustment on the quantization parameters, which is not limited in the embodiments of the present application.

[0206] In some embodiments of the present application, the encoder determines the implementation of a frame-level quantization parameter adjustment flag based on a first rate-distortion cost and at least one second rate-distortion cost, including: determining a first minimum rate-distortion cost (bestCostNN) from the first rate-distortion cost and at least one second rate-distortion cost; if the first minimum rate-distortion cost is the first rate-distortion cost, determining that the frame-level quantization parameter adjustment flag is unused; if the first minimum rate-distortion cost is any one of at least one second rate-distortion cost, determining that the frame-level quantization parameter adjustment flag is used.

[0207] In some embodiments of the present application, after the encoder determines the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and at least one second rate-distortion cost, if the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost, then from at least one frame-level quantization bias parameter, the frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost is written into the bitstream, or the block-level quantization parameter index (offset sequence number) of the frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost is written into the bitstream.

[0208] In some embodiments of the present application, if the first minimum rate distortion cost is any one of the at least one second rate distortion cost, a second residual scaling factor corresponding to the first minimum rate distortion cost is written into the bitstream. If the first minimum rate distortion cost is the first rate distortion cost, the first residual scaling factor is written into the bitstream.

[0209] It should be noted that the writing here means waiting to be written, and the first minimum rate distortion cost needs to be compared with costOrg and costCTU later. If the first minimum rate distortion cost is the smallest, the writing operation is performed.

[0210] Exemplarily, the encoding end continues to try the neural network-based loop filtering technology, and the process is the same as the second round, but adjustments are made to the input part, and this round of attempts can be repeated multiple times. If the first attempt chooses to adjust the BaseQP quantization parameter, the BaseQP is superimposed with the adjustment bias parameter offset1 to obtain BaseQPFinal to replace BaseQP as the input, and the rest remains unchanged. Similarly, the rate-distortion cost value is calculated in the case of offset1, recorded as costOffset1; continue to try the second bias parameter offset2, the process is the same as before, and the rate-distortion cost value is calculated, recorded as costOffset2; in this example, this round tries the BaseQP offset twice, and does not try to adjust SliceQP. After obtaining costNN, costOffset1 and costOffset2, the encoder compares them. If costNN is the smallest, the frame-level quantization parameter is adjusted to the unused flag to be written into the bitstream; if costOffset1 is the smallest, the frame-level quantization parameter is adjusted to the used flag, and the frame-level quantization parameter is adjusted to the serial number representing the current offset1 to be written into the bitstream, and the residual scaling factor of the bitstream to be written is replaced with the residual scaling factor under the current offset1.

[0211] It can be understood that the encoder can determine whether the quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby achieving flexible selection and diverse change processing of the quantization parameters, thereby improving encoding efficiency.

[0212] In some embodiments of the present application, a filtering method provided by the encoder may further include:

[0213] S207 . When the sequence level allows the use of a flag to represent the permission, a rate-distortion cost is estimated based on the original value and the reconstructed value of the current block in the current frame to obtain a third rate-distortion cost.

[0214] When the sequence level allows the use of a flag to represent the permission, the encoder does not perform filtering processing, and a rate-distortion cost is estimated based on the original value and the reconstructed value of the current block in the current frame to obtain a third rate-distortion cost (costOrg).

[0215] In some embodiments of the present application, after the encoder determines the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and at least one second rate-distortion cost, the method further includes:

[0216] S208, performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a third reconstruction value;

[0217] It should be noted that the implementation principle of S208 is the same as that of S203, and will not be repeated here.

[0218] S209, performing rate-distortion cost estimation based on the third reconstructed value and the original value of the current block to obtain a fourth rate-distortion cost (costCTUorg) of the current block;

[0219] It should be noted that the implementation principle of S209 is the same as that of S204 and will not be repeated here.

[0220] S210, performing filtering estimation on the current block based on the neural network filtering model, the target reconstruction value corresponding to the first minimum rate-distortion cost, and the frame-level quantization parameter to obtain a fourth reconstruction value;

[0221] It should be noted that the implementation principle of S210 is the same as that of S203, except that what is input here is the target reconstruction value corresponding to the first minimum rate-distortion cost, rather than the reconstruction value of the current block.

[0222] S211, performing rate-distortion cost estimation based on the fourth reconstructed value and the original value of the current block to obtain a fifth rate-distortion cost (costCTUnn) of the current block;

[0223] It should be noted that the implementation principle of S211 is the same as that of S204 and will not be repeated here.

[0224] S212, determining a block-level usage flag based on the fourth rate-distortion cost and the fifth rate-distortion cost;

[0225] In the embodiment of the present application, if the fourth rate-distortion cost is less than the fifth rate-distortion cost, the block-level usage flag is determined to be unused; if the fourth rate-distortion cost is greater than or equal to the fifth rate-distortion cost, the block-level usage flag is determined to be used.

[0226] It should be noted that the block-level use flag indicates whether the current block or coding tree unit needs filtering.

[0227] Exemplarily, the value of the block-level usage flag can be 1 to represent use, and 0 to represent unuse. The embodiment of the present application does not limit the expression form and meaning of the value of the block-level usage flag.

[0228] S213 , traverse the blocks in the current frame, and determine the sum of the minimum rate-distortion costs of all the blocks in the current frame as the sixth rate-distortion cost (costCTU) of the current frame.

[0229] In an embodiment of the present application, the encoder adds up the minimum rate-distortion costs of each color component corresponding to the blocks in the current frame to obtain the frame-level rate-distortion cost of each color component, and then adds the rate-distortion costs of each color component to obtain the sixth rate-distortion cost of the current frame.

[0230] Exemplarily, the encoding end attempts to optimize the selection at the coding tree unit level, attempts to switch the coding tree unit level, and each component can be controlled separately. The encoder traverses the current coding tree unit, calculates the rate-distortion cost value of the reconstructed sample without using the neural network loop filter and the original sample of the current coding tree unit, recorded as costCTUorg; calculates the rate-distortion cost value of the reconstructed sample using the neural network loop filter and the original sample of the current coding tree unit, recorded as costCTUnn. If costCTUorg is less than costCTUnn, the block usage flag of the neural network loop filter at the coding tree unit level is set to use, to be written into the bitstream; otherwise, the block usage flag of the neural network loop filter at the coding tree unit level is set to use, to be written into the bitstream; if all coding tree units in the current frame have been traversed, the rate-distortion cost value of the reconstructed sample of the current frame and the original image sample in this case is calculated, recorded as costCTU.

[0231] In some embodiments of the present application, the encoder estimates the rate-distortion cost based on the third reconstruction value and the original value of the current block to obtain the fourth rate-distortion cost of the current block, and before determining the block-level usage identification bit based on the fourth rate-distortion cost and the fifth rate-distortion cost, performs at least one filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block, at least one frame-level quantization bias parameter and the frame-level quantization parameter, and determines at least one fifth reconstruction value; (similar to the principle of the third round) based on at least one fifth reconstruction value and the original value of the current block, determines the fifth rate-distortion cost with the minimum rate-distortion cost.

[0232] It should be noted that the process of performing at least one filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block, at least one frame-level quantization bias parameter and a frame-level quantization parameter, and determining at least one fifth reconstruction value is the same as the principle in S205 and will not be repeated here.

[0233] In some embodiments of the present application, when the encoder obtains the third rate distortion cost (costOrg), the first minimum rate distortion cost (bestCostNN) and the sixth rate distortion cost (costCTU), if the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the third rate distortion cost, it determines that the frame level usage flag is unused; and writes the frame level usage flag into the bitstream.

[0234] If the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the first minimum rate distortion cost, the frame level usage flag is determined to be used and the frame level switch flag is turned on; and the frame level usage flag and the frame level switch flag are written into the bitstream. Also, the frame level quantization offset parameter corresponding to the first minimum rate distortion cost is written into the bitstream, or the block level quantization parameter index (offset sequence number) of the frame level quantization offset parameter corresponding to the first minimum rate distortion cost is written into the bitstream.

[0235] If the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the sixth rate distortion cost, then determine that the frame-level usage flag is used and the frame-level switch flag is not turned on; and write the frame-level usage flag, the frame-level switch flag, and the block-level usage flag into the bitstream.

[0236] Exemplarily, each color component is traversed. If the value of costOrg is the smallest, the frame-level usage flag based on the neural network loop filter corresponding to the color component is set to used, written into the bitstream, and no neural network loop filtering is performed; if the value of bestCostNN is the smallest, the frame-level usage flag based on the neural network loop filter corresponding to the color component is set to used, the frame-level switch flag is set to used, and the frame-level quantization parameter adjustment flag decided in the third round, as well as the index information and the residual scaling factor are written into the bitstream; if the value of costCTU is the smallest, the frame-level usage flag based on the neural network loop filter corresponding to the color component is set to used, the frame-level switch flag is set to unused, and the frame-level quantization parameter adjustment flag and the frame-level quantization parameter adjustment index residual scaling factor decided in the third round are written into the bitstream. In addition, the block usage flag of each coding tree unit level needs to be written into the bitstream.

[0237] It can be understood that the encoder can determine whether the quantization parameter of the input neural network filter model needs to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameter, thereby improving the coding efficiency.

[0238] For example, the loop filter part of the codec end integrates the embodiment of the present application into the reference software of JVET EE1, which uses VTM10.0 as the platform basis and has the same basic performance as VVC. The test results of the integrated extraction under the general test conditions RA (Table 1) and LDB (Table 2) are shown in the table.

[0239] Table 1

[0240]

[0241]

[0242] Table 2

[0243]

[0244] It can be seen from Tables 1 and 2 above that the filtering method provided by the present application can achieve stable performance improvements regardless of the test conditions of RA or LDB. From classA1 to classE, it can be seen that RA has an average performance gain of more than 0.2% BD-rate, and LDB performs better in certain classes, with a maximum BD-rate performance gain of 0.57%, which mainly depends on Y. The filtering method provided by the present application does not bring additional complexity to the decoding end, and there is no additional complexity increase. At the decoding end, it only needs to adjust the quantization parameters once when decoding the current frame, which does not increase the complexity and can bring stable gains.

[0245] The present application embodiment provides a decoder 1, such as Fig. 9 As shown, the decoder 1 may include:

[0246] The parsing part 10 is configured to parse the bitstream and obtain a frame-level usage identification bit based on a neural network filtering model;

[0247] The first determination part 11 is configured to obtain a frame level switch flag and a frame level quantization parameter adjustment flag when the frame level usage flag indicates usage; the frame level switch flag is used to determine whether all blocks in the current frame are filtered;

[0248] The first adjustment part 12 is configured to obtain an adjusted frame level quantization parameter when the frame level switch flag indicates that it is turned on and the frame level quantization parameter adjustment flag indicates that it is used;

[0249] The first filtering part 13 is configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block.

[0250] In some embodiments of the present application, the parsing part 10 is further configured to obtain a block-level usage flag when the frame-level switch flag representation is not turned on;

[0251] The first determining part 11 is further configured to obtain an adjusted frame-level quantization parameter when the block-level use flag indicates that any color component of the current block is used and the frame-level quantization parameter adjustment flag indicates that it is used;

[0252] The first filtering part 13 is further configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block.

[0253] In some embodiments of the present application, the parsing part 10 is further configured to obtain a block-level quantization parameter adjustment flag after obtaining the block-level usage flag;

[0254] The first determining part 11 is further configured to obtain an adjusted block-level quantization parameter when the block-level use flag indicates that any color component of the current block is used and the block-level quantization parameter adjustment flag indicates that it is used;

[0255] The first filtering part 13 is further configured to filter the current block of the current frame based on the adjusted block-level quantization parameter and the neural network filtering model to obtain second residual information of the current block.

[0256] In some embodiments of the present application, the first determining part 11 is further configured to, after obtaining the block-level usage flag, obtain a block-level quantization parameter corresponding to the current block when the block-level usage flag indicates that any color component of the current block is used;

[0257] The first filtering part 13 is further configured to filter the current block of the current frame based on the adjusted block-level quantization parameter and the neural network filtering model to obtain second residual information of the current block.

[0258] In some embodiments of the present application, the parsing part 10 is also configured to parse the bitstream, after obtaining the frame-level usage flag based on the neural network filtering model and before obtaining the adjusted frame-level quantization parameter, when the frame-level usage flag represents usage and the current frame is a first type frame, obtain the frame-level switch flag and the frame-level quantization parameter adjustment flag.

[0259] In some embodiments of the present application, the first determination part 11 is also configured to determine the frame level quantization bias parameter based on the frame level quantization parameter adjustment index obtained in the code stream; and determine the adjusted frame level quantization parameter based on the obtained frame level quantization parameter and the frame level quantization bias parameter.

[0260] In some embodiments of the present application, the parsing part 10 is further configured to obtain the adjusted frame-level quantization parameter from the bit stream.

[0261] In some embodiments of the present application, the first determination part 11 is also configured to determine the block-level quantization bias parameter based on the block-level quantization parameter index obtained in the code stream; and determine the adjusted block-level quantization parameter based on the obtained block-level quantization parameter and the block-level quantization bias parameter.

[0262] In some embodiments of the present application, the first determination part 11 is also configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model, and obtain a reconstructed value of the current block before obtaining the first residual information of the current block.

[0263] In some embodiments of the present application, the first filtering part 13 is also configured to use the neural network filtering model to filter the reconstructed value of the current block and the adjusted frame-level quantization parameter to obtain the first residual information of the current block to complete the filtering of the current block.

[0264] In some embodiments of the present application, the first determination part 11 is also configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model, and obtain at least one of the prediction value, block partition information and deblocking filter boundary strength of the current block, and the reconstructed value of the current block before obtaining the first residual information of the current block.

[0265] In some embodiments of the present application, the first filtering part 13 is also configured to use the neural network filtering model to filter the prediction value of the current block, at least one of the block partitioning information and the deblocking filter boundary strength, the reconstructed value of the current block, and the adjusted frame-level quantization parameter to obtain the first residual information of the current block to complete the filtering of the current block.

[0266] In some embodiments of the present application, the first determining part 11 is further configured to filter the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block, or filter the current block of the current frame based on the adjusted block-level quantization parameter and the neural network filtering model to obtain the second residual information of the current block.

[0267] Obtain a second residual scaling factor in the code stream; scale the first residual information or the second residual information of the current block based on the second residual scaling factor to obtain first target residual information or second target residual information; determine a first target reconstruction value of the current block based on the first target residual information and the reconstruction value of the current block; or, when the block level is characterized by an identification bit, determine a second target reconstruction value of the current block based on the second target residual information and the reconstruction value of the current block.

[0268] In some embodiments of the present application, the first determining part 11 is further configured to determine the reconstruction value of the current block as the second target reconstruction value when the block-level usage identification bit representation is not used.

[0269] In some embodiments of the present application, the first determining part 11 is further configured to obtain at least one of a predicted value of the current block, block partition information and a deblocking filter boundary strength, and a reconstructed value of the current block after obtaining the frame-level usage identification bit based on the neural network filtering model;

[0270] The parsing part 10 is further configured to obtain a frame-level switch flag and a frame-level input parameter adjustment flag when the frame-level use flag indicates use; the frame-level input parameter adjustment flag indicates whether any one of the predicted value, block partition information and deblocking filter boundary strength is adjusted;

[0271] The first determining part 11 is further configured to obtain the adjusted block-level input parameter when the frame-level switch flag indicates that it is turned on and the frame-level input parameter adjustment flag indicates that it is used;

[0272] The first filtering part 13 is also configured to filter the current block of the current frame based on the adjusted block-level input parameters, the acquired frame-level quantization parameters and the neural network filtering model to obtain third residual information of the current block.

[0273] In some embodiments of the present application, the parsing part 10 is further configured to parse out a sequence-level permission-to-use flag; when the sequence-level permission-to-use flag indicates permission, parse the frame-level usage flag based on a neural network filtering model.

[0274] The present application embodiment also provides a decoder 1, such as Fig.10 As shown, the decoder 1 may include:

[0275] A first memory 14, configured to store a computer program that can be run on the first processor 15;

[0276] The first processor 15 is configured to execute the method described in the decoder when running the computer program.

[0277] It can be understood that the decoder can determine whether the quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameters (input parameters), thereby improving decoding efficiency.

[0278] Among them, the first processor 15 can be implemented by software, hardware, firmware or a combination thereof, and can use circuits, single or multiple application specific integrated circuits (ASICs), single or multiple general-purpose integrated circuits, single or multiple microprocessors, single or multiple programmable logic devices, or a combination of the aforementioned circuits or devices, or other suitable circuits or devices, so that the first processor 15 can execute the corresponding steps of the filtering method on the decoder side in the aforementioned embodiment.

[0279] The present application embodiment provides an encoder 2, such as Fig.11 As shown, the encoder 2 may include:

[0280] The second determining part 20 is configured to obtain a sequence level permission flag; and when the sequence level permission flag indicates permission, obtain the original value of the current block in the current frame, the reconstructed value of the current block and the frame level quantization parameter;

[0281] The second filtering part 21 is configured to perform filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a first reconstruction value;

[0282] The second determining part 20 is further configured to perform rate-distortion cost estimation based on the first reconstructed value and the original value of the current block to obtain the rate-distortion cost of the current block, and traverse the current frame to determine a first rate-distortion cost of the current frame;

[0283] The second filtering part 21 is further configured to perform at least one filtering estimation on the current frame based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and a reconstruction value of a current block in the current frame to determine at least one second rate-distortion cost of the current frame;

[0284] The second determining part 20 is further configured to determine a frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost.

[0285] In some embodiments of the present application, the second determining part 20 is further configured to obtain an i-th frame-level quantization offset parameter, and adjust the frame-level quantization parameter based on the i-th frame-level quantization offset parameter to obtain an i-th adjusted frame-level quantization parameter; i is a positive integer greater than or equal to 1;

[0286] Performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter to obtain an i-th second reconstruction value;

[0287] A rate-distortion cost is estimated based on the i-th second reconstruction value and the original value of the current block, and all blocks of the current frame are traversed to obtain the i-th second rate-distortion cost. The i+1-th filtering estimation is continued based on the i+1-th frame-level quantization bias parameter until it is completed at least once, thereby determining at least one second rate-distortion cost of the current frame.

[0288] In some embodiments of the present application, the second determining part 20 is further configured to determine a first minimum rate-distortion cost from the first rate-distortion cost and the at least one second rate-distortion cost;

[0289] If the first minimum rate-distortion cost is the first rate-distortion cost, determining that the frame-level quantization parameter adjustment flag is unused;

[0290] If the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost, the frame-level quantization parameter adjustment flag is determined to be used.

[0291] In some embodiments of the present application, the second determination part 20 is further configured to estimate the rate-distortion cost based on the original value and the reconstructed value of the current block in the current frame to obtain a third rate-distortion cost when the sequence level allows the use of an identification bit to represent the permission.

[0292] In some embodiments of the present application, the second filtering part 21 is further configured to determine, based on the first rate-distortion cost and the at least one second rate-distortion cost, after determining the frame-level quantization parameter adjustment flag, perform filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a third reconstruction value;

[0293] The second determining part 20 is further configured to perform rate-distortion cost estimation based on the third reconstructed value and the original value of the current block to obtain a fourth rate-distortion cost of the current block;

[0294] The second filtering part 21 is further configured to perform filtering estimation on the current block based on the neural network filtering model, the target reconstruction value corresponding to the first minimum rate distortion cost and the frame-level quantization parameter to obtain a fourth reconstruction value;

[0295] The second determination part 20 is further configured to perform rate-distortion cost estimation based on the fourth reconstructed value and the original value of the current block to obtain a fifth rate-distortion cost of the current block; determine a block-level usage identification bit based on the fourth rate-distortion cost and the fifth rate-distortion cost; traverse the blocks in the current frame, and determine the sum of the minimum rate-distortion costs of all blocks in the current frame as the sixth rate-distortion cost of the current frame.

[0296] In some embodiments of the present application, the second determining part 20 is further configured to determine that the block-level usage flag is unused if the fourth rate-distortion cost is less than the fifth rate-distortion cost;

[0297] If the fourth rate-distortion cost is greater than or equal to the fifth rate-distortion cost, the block-level usage flag is determined to be used.

[0298] In some embodiments of the present application, the encoder 2 further includes: a writing part 22; the second determining part 20 is further configured to determine that the frame-level usage flag is unused if the minimum rate-distortion cost among the third rate-distortion cost, the first minimum rate-distortion cost and the sixth rate-distortion cost is the third rate-distortion cost;

[0299] The writing part 22 is configured to write the frame level usage identification bit into the bit stream;

[0300] The second determining part 20 is further configured to determine that the frame level use flag is used and the frame level switch flag is turned on if the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the first minimum rate distortion cost;

[0301] The writing part 22 is configured to write the frame level usage flag and the frame level switch flag into the bitstream;

[0302] The second determining part 20 is further configured to determine that the frame level use flag is used and the frame level switch flag is not turned on if the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the sixth rate distortion cost;

[0303] The writing part 22 is configured to write the frame level usage flag, the frame level switch flag, and the block level usage flag into the bitstream.

[0304] In some embodiments of the present application, the writing part 22 is configured to, after determining the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost, if the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost, write the frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost from at least one frame-level quantization bias parameter into the bitstream, or write the block-level quantization parameter index of the frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost into the bitstream.

[0305] In some embodiments of the present application, the second filtering part 21 is also configured to perform filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter for the current frame, and determine first estimated residual information; determine a first residual scaling factor; scale the first estimated residual value using the first residual scaling factor to obtain first scaled residual information; and combine the first scaled residual information with the reconstruction value of the current block to determine the first reconstruction value.

[0306] In some embodiments of the present application, the second determining part 20 is further configured to obtain, for the current frame, at least one of a predicted value, block partitioning information, and a deblocking filter boundary strength of the current block, and a reconstructed value of the current block before determining the first residual scaling factor;

[0307] The second filtering part 21 is also configured to use the neural network filtering model to perform filtering estimation on the prediction value of the current block, at least one of the block division information and the deblocking filter boundary strength, the reconstructed value of the current block, and the frame-level quantization parameter to obtain the first estimated residual information of the current block.

[0308] In some embodiments of the present application, the writing part 22 is configured to write the first residual scaling factor into the bitstream if the first minimum rate-distortion cost is the first rate-distortion cost after determining the first residual scaling factor.

[0309] In some embodiments of the present application, the second filtering part 21 is further configured to perform a filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter, to obtain the i-th second estimated residual information; determine the i-th second residual scaling factor corresponding to the i-th adjusted frame-level quantization parameter; use the i-th second residual scaling factor to scale the i-th second estimated residual information to obtain the i-th second scaled residual information; combine the i-th second scaled residual information with the reconstruction value of the current block to determine the i-th second reconstruction value.

[0310] In some embodiments of the present application, the writing part 22 is configured to, after determining the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost, write a second residual scaling factor corresponding to the first minimum rate-distortion cost into the bitstream if the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost.

[0311] In some embodiments of the present application, the second determining part 20 is further configured to obtain at least one of a predicted value, block partitioning information and a deblocking filter boundary strength of the current block, and a reconstructed value of the current block before determining the i-th second residual scaling factor corresponding to the i-th adjusted frame-level quantization parameter;

[0312] The second filtering part 21 is also configured to use the neural network filtering model to perform frame-level filtering estimation on the prediction value of the current block, at least one of the block division information and the deblocking filter boundary strength, the reconstructed value of the current block, and the i-th adjusted frame-level quantization parameter to obtain the i-th second estimated residual information of the current block.

[0313] In some embodiments of the present application, the second filtering part 21 is also configured to perform at least one filtering estimation on the current frame based on the neural network filtering model, at least one frame-level quantization bias parameter, the frame-level quantization parameter, and the reconstructed value of the current block in the current frame when the current frame is a first type frame, and determine at least one second rate-distortion cost of the current frame.

[0314] In some embodiments of the present application, the second filtering part 21 is further configured to perform at least one filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block, at least one frame-level quantization offset parameter and the frame-level quantization parameter, and determine at least one fifth reconstruction value after performing rate-distortion cost estimation based on the third reconstruction value and the original value of the current block to obtain a fourth rate-distortion cost of the current block and before determining the block-level use flag based on the fourth rate-distortion cost and the fifth rate-distortion cost;

[0315] The second determining part 20 is further configured to determine the fifth rate-distortion cost with the minimum rate-distortion cost based on the at least one fifth reconstructed value and the original value of the current block.

[0316] In some embodiments of the present application, the second determining part 20 is further configured to obtain, when the sequence level allows the use of the flag bit to represent the permission, at least one of the predicted value of the current block, the block partition information and the deblocking filter boundary strength, the reconstructed value of the current block and the frame level quantization parameter;

[0317] The second filtering part 21 is further configured to perform filtering estimation on the current block based on the prediction value of the current block, at least one of the block partition information and the deblocking filter boundary strength, the neural network filtering model, the reconstruction value of the current block and the frame level quantization parameter, to determine a sixth reconstruction value;

[0318] The second determining part 20 is further configured to perform rate-distortion cost estimation based on the sixth reconstructed value and the original value of the current block to obtain the rate-distortion cost of the current block, and traverse the current frame to determine the seventh rate-distortion cost of the current frame;

[0319] The second filtering part 21 is further configured to perform at least one filtering estimation on the current frame based on the prediction value of the current block, the block partitioning information and at least one of the deblocking filter boundary strength, the neural network filtering model, at least one frame-level input bias parameter, and the reconstruction value of the current block in the current frame, to determine at least one eighth rate-distortion cost of the current frame;

[0320] The second determining part 20 is further configured to determine a frame-level input parameter adjustment flag based on the first rate-distortion cost and the at least one eighth rate-distortion cost.

[0321] The present application embodiment provides an encoder 2, such as Fig.12 As shown, the encoder 2 may include:

[0322] A second memory 23, configured to store a computer program that can be run on the second processor 24;

[0323] The second processor 24 is configured to execute the method described by the encoder when running the computer program.

[0324] It can be understood that the encoder can determine whether the quantization parameters of the input neural network filter model need to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameters (input parameters), thereby improving decoding efficiency.

[0325] An embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a first processor, the method described in the decoder is implemented, or when the computer program is executed by a second processor, the method described in the encoder is implemented.

[0326] The components in the embodiments of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software function modules.

[0327] If the integrated unit is implemented in the form of a software function module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute all or part of the steps of the method described in this embodiment. The aforementioned computer-readable storage medium includes: ferromagnetic random access memory (FRAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface storage, optical disk, or compact disc read-only memory (CD-ROM), etc., which can store program codes, and the embodiments of the present disclosure are not limited thereto.

[0328] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0329] Industrial Applicability

[0330] The embodiment of the present application provides a filtering method, an encoder, a decoder and a storage medium, which obtains a frame-level usage flag based on a neural network filtering model by parsing a bit stream; when the frame-level usage flag indicates usage, obtains a frame-level switch flag and a frame-level quantization parameter adjustment flag; the frame-level switch flag is used to determine whether each block in the current frame is filtered; when the frame-level switch flag indicates that it is turned on and the frame-level quantization parameter adjustment flag indicates that it is used, obtain the adjusted frame-level quantization parameter; based on the adjusted frame-level quantization parameter and the neural network filtering model, filter the current block of the current frame to obtain the first residual information of the current block. In this way, it is possible to determine whether the quantization parameter of the input neural network filtering model needs to be adjusted based on the frame-level quantization parameter adjustment flag, thereby realizing flexible selection and diverse change processing of the quantization parameter (input parameter), thereby improving decoding efficiency.

Claims

1. A filtering method, applied to a decoder, comprising: Parse the bitstream to obtain a frame-level switch flag and a frame-level quantization parameter adjustment flag; the frame-level switch flag is used to determine whether each block in the current frame is subjected to neural network filtering; the frame-level quantization parameter adjustment flag is used to indicate whether to adjust the quantization parameter for the current frame; When the frame-level switch flag indicates that neural network filtering is performed on each block in the current frame, and the frame-level quantization parameter adjustment flag indicates that the quantization parameter is adjusted for the current frame, obtaining the adjusted frame-level quantization parameter; Based on the adjusted frame-level quantization parameter and the neural network filtering model, a current block of the current frame is filtered to obtain first residual information of the current block.

2. The method according to claim 1, wherein: The method further comprises: When the frame-level switch flag indicates that neural network filtering is performed on only some blocks in the current frame, obtaining a block-level usage flag; When the block-level usage flag indicates that neural network filtering is used for any color component of the current block, and the frame-level quantization parameter adjustment flag indicates that a quantization parameter is adjusted for the current frame, obtaining an adjusted frame-level quantization parameter; Based on the adjusted frame-level quantization parameter and the neural network filtering model, a current block of the current frame is filtered to obtain first residual information of the current block.

3. The method according to claim 1 or 2, wherein: The obtaining of the adjusted frame-level quantization parameter comprises: Determining a frame-level quantization bias parameter based on a frame-level quantization parameter adjustment index obtained from a bitstream; The adjusted frame level quantization parameter is determined according to the acquired frame level quantization parameter and the frame level quantization offset parameter.

4. The method according to claim 1 or 2, wherein: Before filtering the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block, the method further includes: Get the reconstruction value of the current block.

5. The method according to claim 1 or 2, wherein: Before filtering the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block, the method further includes: At least one of a prediction value of a current block, block partition information, and a deblocking filter boundary strength, and a reconstruction value of the current block are obtained.

6. The method according to claim 5, wherein: The filtering of the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain first residual information of the current block includes: Using the neural network filtering model, the predicted value of the current block, at least one of the block partitioning information and the deblocking filter boundary strength, the reconstructed value of the current block, and the adjusted frame-level quantization parameter are filtered to obtain the first residual information of the current block.

7. The method according to any one of claims 1 or 2, wherein: After filtering the current block of the current frame based on the adjusted frame-level quantization parameter and the neural network filtering model to obtain the first residual information of the current block, the method further includes: Obtaining a second residual scaling factor in the bitstream; Scaling the first residual information of the current block based on the second residual scaling factor to obtain first target residual information; Based on the first target residual information and the reconstruction value of the current block, a first target reconstruction value of the current block is determined.

8. A filtering method, applied to an encoder, the method comprising: Get the sequence-level permission flag; When the sequence level allows the use of a flag indicating that a neural network-based loop filtering technology is allowed, obtaining an original value of a current block in a current frame, a reconstructed value of the current block, and a frame-level quantization parameter; Performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a first reconstruction value; Performing a rate-distortion cost estimation based on the first reconstructed value and the original value of the current block to obtain a rate-distortion cost of the current block, and traversing the current frame to determine a first rate-distortion cost of the current frame; Based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and a reconstruction value of a current block in the current frame, performing at least one filtering estimation on the current frame to determine at least one second rate-distortion cost of the current frame; A frame-level quantization parameter adjustment flag is determined based on the first rate-distortion cost and the at least one second rate-distortion cost.

9. The method according to claim 8, wherein: The step of performing at least one filtering estimation on the current frame based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and a reconstruction value of a current block in the current frame to determine at least one second rate-distortion cost of the current frame includes: Obtaining an i-th frame-level quantization offset parameter, and adjusting the frame-level quantization parameter based on the i-th frame-level quantization offset parameter to obtain an i-th adjusted frame-level quantization parameter; i is a positive integer greater than or equal to 1; Performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the i-th adjusted frame-level quantization parameter to obtain an i-th second reconstruction value; A rate-distortion cost is estimated based on the i-th second reconstruction value and the original value of the current block, and all blocks of the current frame are traversed to obtain the i-th second rate-distortion cost. The i+1-th filtering estimation is continued based on the i+1-th frame-level quantization bias parameter until it is completed at least once, thereby determining at least one second rate-distortion cost of the current frame.

10. The method according to claim 8 or 9, wherein: The determining, based on the first rate-distortion cost and the at least one second rate-distortion cost, a frame-level quantization parameter adjustment flag includes: determining a first minimum rate-distortion cost from the first rate-distortion cost and the at least one second rate-distortion cost; If the first minimum rate-distortion cost is the first rate-distortion cost, determining that the frame-level quantization parameter adjustment flag indicates that the quantization parameter is not adjusted for the current frame; If the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost, determining the frame-level quantization parameter adjustment flag indicates adjusting the quantization parameter for the current frame.

11. The method according to claim 8 or 9, wherein: The method further comprises: When the sequence level allows the use of a flag to indicate that a neural network-based loop filtering technique is allowed, a rate-distortion cost is estimated based on the original value and the reconstructed value of the current block in the current frame to obtain a third rate-distortion cost.

12. The method according to claim 10, wherein: After determining the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost, the method further includes: Performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine a third reconstruction value; Performing rate-distortion cost estimation based on the third reconstructed value and the original value of the current block to obtain a fourth rate-distortion cost of the current block; Performing filtering estimation on the current block based on the neural network filtering model, the target reconstruction value corresponding to the first minimum rate-distortion cost, and the frame-level quantization parameter to obtain a fourth reconstruction value; Performing rate-distortion cost estimation based on the fourth reconstructed value and the original value of the current block to obtain a fifth rate-distortion cost of the current block; Determining a block-level usage flag based on the fourth rate-distortion cost and the fifth rate-distortion cost; The blocks in the current frame are traversed, and the sum of the minimum rate-distortion costs of all the blocks in the current frame is determined as the sixth rate-distortion cost of the current frame.

13. The method according to claim 12, wherein: The method further comprises: If the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the first minimum rate distortion cost, determining that a frame-level switch flag indicates that neural network filtering is performed on each block in the current frame, and writing the frame-level switch flag into a bitstream; If the minimum rate distortion cost among the third rate distortion cost, the first minimum rate distortion cost and the sixth rate distortion cost is the sixth rate distortion cost, it is determined that the frame-level switch flag indicates that neural network filtering is performed only on part of the blocks in the current frame, and the frame-level switch flag and the block-level usage flag are written into the bitstream.

14. The method according to claim 9, wherein: After determining the frame-level quantization parameter adjustment flag based on the first rate-distortion cost and the at least one second rate-distortion cost, the method further includes: If the first minimum rate-distortion cost is any one of the at least one second rate-distortion cost, a frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost is written into the bitstream from at least one frame-level quantization bias parameter, or a block-level quantization parameter index of the frame-level quantization bias parameter corresponding to the first minimum rate-distortion cost is written into the bitstream.

15. The method according to claim 8, wherein: The filtering and estimating the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine the first reconstruction value includes: For a current frame, performing filtering estimation on the current block based on the neural network filtering model, the reconstruction value of the current block and the frame-level quantization parameter to determine first estimation residual information; determining a first residual scaling factor; Scaling the first estimated residual value using the first residual scaling factor to obtain first scaled residual information; The first scaled residual information is combined with the reconstruction value of the current block to determine the first reconstruction value.

16. The method according to claim 15, wherein: Before determining the first residual scaling factor, the method further includes: For a current frame, obtaining at least one of a prediction value of a current block, block partition information, and a deblocking filter boundary strength, and a reconstruction value of the current block; Using the neural network filtering model, filtering estimation is performed on the prediction value of the current block, at least one of the block partitioning information and the deblocking filter boundary strength, the reconstructed value of the current block, and the frame-level quantization parameter to obtain the first estimated residual information of the current block.

17. The method according to claim 8, wherein: The step of performing at least one filtering estimation on the current frame based on the neural network filtering model, at least one frame-level quantization offset parameter, the frame-level quantization parameter, and a reconstruction value of a current block in the current frame to determine at least one second rate-distortion cost of the current frame includes: When the current frame is a first type frame, based on the neural network filtering model, at least one frame-level quantization bias parameter, the frame-level quantization parameter and the reconstructed value of the current block in the current frame, at least one filtering estimation is performed on the current frame to determine at least one second rate-distortion cost of the current frame.

18. The method according to claim 8, wherein: The method further comprises: When the sequence level allows the use of a flag indicating that a neural network-based loop filtering technology is allowed, obtaining a prediction value of a current block, at least one of block partition information and a deblocking filter boundary strength, a reconstruction value of the current block, and a frame-level quantization parameter; Performing filtering estimation on the current block based on at least one of the predicted value of the current block, the block partitioning information and the deblocking filter boundary strength, the neural network filtering model, the reconstruction value of the current block and the frame level quantization parameter, to determine a sixth reconstruction value; Performing rate-distortion cost estimation based on the sixth reconstructed value and the original value of the current block to obtain a rate-distortion cost of the current block, and traversing the current frame to determine a seventh rate-distortion cost of the current frame; Based on the predicted value of the current block, the block partition information and at least one of the deblocking filter boundary strength, the neural network filter model, at least one frame-level input bias parameter, and the reconstructed value of the current block in the current frame, perform at least one filtering estimation on the current frame to determine at least one eighth rate-distortion cost of the current frame; Based on the first rate-distortion cost and the at least one eighth rate-distortion cost, a frame-level input parameter adjustment flag is determined.

19. A computer-readable storage medium, wherein: The computer-readable storage medium stores a code stream, wherein the code stream is generated according to the method according to any one of claims 8 to 18.

Images