Encoding and decoding method, encoder, decoder and storage medium

Abstract

The embodiment of the invention provides a decoding method, and the method comprises the steps: decoding a code stream through a decoder, and determining first identification information; if the first identification information indicates a scaling factor generated by using a neural network on a current color component of a current frame of the current sequence, determining second identification information; if the second identification information indicates a scalability factor generated by using a neural network for the current block of the current color component, determining a scalability factor corresponding to the current block by using a scalability factor generation network; wherein the scaling factor generation network is used for predicting scaling factors of the image blocks; and determining a reconstructed block of the current block according to the scaling factor.

Description

Coding and decoding method, encoder, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a coding and decoding method, an encoder, a decoder, and a storage medium. Background Art

[0002] In recent years, with the development of deep learning technology, the exploration of neural network-based loop filters has gradually expanded. For loop filtering modules, many high-performance neural network loop filtering tools have been proposed to optimize the filtering of the input reconstructed image. To achieve even better filtering optimization results, a human-defined scaling factor (SF) has been introduced. SF scales the residual between the network output image and the original input reconstructed image, making the final output image closer to the original image.

[0003] However, the current common use process of scaling factors has poor flexibility and applicability, which reduces coding performance, and the compression and transmission of scaling factors also greatly reduce coding efficiency.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which enhance the flexibility and applicability of the scaling factor, can improve coding performance, and improve coding efficiency.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Decoding the code stream to determine the first identification information;

[0009] If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information;

[0010] If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block;

[0011] A reconstructed block of the current block is determined according to the scaling factor.

[0012] In a second aspect, an embodiment of the present application provides an encoding method, applied to an encoder, the method comprising:

[0013] determining first identification information;

[0014] If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information;

[0015] If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block;

[0016] A reconstructed block of the current block is determined according to the scaling factor.

[0017] In a third aspect, an embodiment of the present application provides a decoder, comprising: a decoding part, a first determining part,

[0018] The decoding part is configured to decode the code stream;

[0019] The first determination part is configured to determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, then determine second identification information; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, then use a scaling factor generation network to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and determine a reconstructed block of the current block based on the scaling factor.

[0020] In a fourth aspect, an embodiment of the present application provides a decoder, which includes a first processor and a first memory storing instructions executable by the first processor. When the instructions are executed, the first processor implements the decoding method described above.

[0021] In a fifth aspect, an embodiment of the present application provides an encoder, comprising: a second determining part,

[0022] The second determination part is configured to determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, then determine second identification information; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, then use a scaling factor generation network to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and determine a reconstructed block of the current block based on the scaling factor.

[0023] In a sixth aspect, an embodiment of the present application provides an encoder, which includes a second processor and a second memory storing instructions executable by the second processor. When the instructions are executed, the second processor implements the encoding method described above.

[0024] In the seventh aspect, an embodiment of the present application provides a computer storage medium, which stores a computer program. When the computer program is executed by a first processor, it implements the decoding method described in the first aspect, or when it is executed by a second processor, it implements the encoding method described in the second aspect.

[0025] Embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a storage medium. The decoder decodes a bitstream and determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. The encoder determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the codec can use the scaling factor generation network on the codec side to predict and obtain a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image. This not only enhances the flexibility and applicability of the scaling factor, but also improves the coding performance and coding efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] FIG1 is a schematic diagram of an application of a coding framework provided by related art;

[0027] FIG2 is a schematic diagram of a network structure in which a loop filter of a multi-layer convolutional network performs filtering optimization on an input reconstructed image;

[0028] FIG3 is a schematic diagram of a network structure using a multi-layer residual network;

[0029] FIG4 is a block diagram of a system composition of an encoder provided in an embodiment of the present application;

[0030] FIG5 is a block diagram of a system composition of a decoder provided in an embodiment of the present application;

[0031] FIG6 is a schematic diagram of a first implementation flow of a decoding method;

[0032] FIG7 is a second schematic diagram of the implementation flow of the decoding method;

[0033] FIG8 is a schematic diagram of the network structure of the scaling factor generation network;

[0034] FIG9 is a second schematic diagram of the network structure of the scaling factor generation network;

[0035] FIG10 is a schematic diagram of an application of a coding framework proposed in an embodiment of the present application;

[0036] FIG11 is a schematic diagram of the overall framework of the loop filter module;

[0037] FIG12 is a schematic diagram of the implementation flow of the encoding method;

[0038] FIG13 is a schematic diagram of the first structure of a decoder according to an embodiment of the present application;

[0039] FIG14 is a second schematic diagram of the structure of the decoder proposed in an embodiment of the present application;

[0040] FIG15 is a schematic diagram of the first structure of the encoder proposed in an embodiment of the present application;

[0041] FIG16 is a second schematic diagram of the composition structure of the encoder proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0042] The following will be combined with the accompanying drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. It should be understood that the specific embodiments described herein are only used to explain the related applications and are not intended to limit the applications. It should also be noted that for ease of description, only the parts relevant to the related applications are shown in the drawings.

[0043] In the following description, reference is made to "some embodiments," which describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. It should also be noted that the terms "first, second, and third" in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It is understood that "first, second, and third" may be interchanged in a specific order or sequential order where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Digital video compression technology primarily compresses massive amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards can save significant amounts of video data, there is still a need for better digital video compression technologies to reduce bandwidth and traffic pressures associated with digital video transmission.

[0045] During digital video encoding, the encoder reads unequal pixels from the original video sequence in different color formats, including the luminance and chrominance components. In other words, the encoder reads a black and white or color image. It then divides the image into blocks and passes the block data to the encoder for encoding.

[0046] Common video codec standards all use a block-based hybrid coding framework. Each frame is divided into square Largest Coding Units (LCUs) or Coding Tree Units (CTUs) of the same size (e.g., 128×128, 64×64, etc.). Each LCU or CTU can be further divided into rectangular Coding Units (CUs) based on rules. Coding Units may also be further divided into smaller Prediction Units (PUs) and Transform Units (TUs), etc.

[0047] FIG1 is a schematic diagram of an application of a coding framework provided by related technologies. As shown in FIG1 , the hybrid coding framework may include modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module may include intra-frame prediction and inter-frame prediction, and inter-frame prediction may include motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels within a frame of a video image, the use of intra-frame prediction in video coding and decoding technology can eliminate spatial redundancy between adjacent pixels. However, since there is also a strong similarity between adjacent frames in a video image, the use of inter-frame prediction in video coding and decoding technology can eliminate temporal redundancy between adjacent frames, thereby improving coding and decoding efficiency. The basic process of a video codec is as follows: On the encoder side, a frame is divided into blocks. Intra-frame prediction or inter-frame prediction is used on the current block to generate a predicted block for the current block. The predicted block is subtracted from the original block to obtain a residual block. The residual block is transformed and quantized to obtain a quantization coefficient matrix. This quantization coefficient matrix is ​​entropy encoded and output to the bitstream. On the decoder side, intra-frame prediction or inter-frame prediction is used on the current block to generate a predicted block for the current block. The decoded bitstream is then decoded to obtain a quantization coefficient matrix. This quantization coefficient matrix is ​​inversely quantized and inversely transformed to obtain a residual block. The predicted block and residual block are added together to obtain a reconstructed block. The reconstructed blocks form a reconstructed image, which is then subjected to image-based or block-based loop filtering to obtain the decoded image. The encoder side also performs similar operations to the decoder side to obtain the decoded image. The decoded image can serve as a reference frame for inter-frame prediction in subsequent frames. Block division information, prediction, transform, quantization, entropy coding, loop filtering, and other mode or parameter information determined by the encoder are output to the bitstream if necessary. The decoding end determines the same block division information as the encoding end by parsing and analyzing the existing information, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end. The decoded image obtained by the encoding end is also usually called a reconstructed image. The current block can be divided into prediction units during prediction, and can be divided into transformation units during transformation. The division of prediction units and transformation units can be different. The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The embodiment of the present application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0048] It can be understood that intra-frame prediction only refers to the information of the same frame image, predicts the pixel information within the current partition block, and is used to eliminate spatial redundancy; inter-frame prediction can refer to the image information of different frames, and use motion estimation to search for the motion vector information that best matches the current partition block, which is used to eliminate temporal redundancy; the transformation converts the predicted image block into the frequency domain, redistributes the energy, and combines quantization to remove information that the human eye is not sensitive to, which is used to eliminate visual redundancy; entropy coding can eliminate character redundancy based on the current context model and the probability information of the binary code stream; loop filtering mainly processes the pixels after inverse transformation and inverse quantization to compensate for the distortion information and provide a better reference for subsequent encoded pixels.

[0049] In Versatile Video Coding (VVC), traditional loop filtering modules primarily include tools such as the Deblocking Filter (DBF), Sample Adaptive Offset (SAO), and Adaptive Loop Filter (ALF). In recent years, with the advancement of deep learning technology, research on neural network-based loop filters has also begun.

[0050] As part of VVC research, a Neural Network-based Video Coding (NNVC) Exploration Experiment (EE) group was established. Figure 2 shows the network architecture of a multi-layer convolutional network loop filter that optimizes the filtering of the input reconstructed image. As shown in Figure 2, a manually defined scaling factor (SF) is introduced to compare the network output image with the original input reconstructed image and perform scaling calculations on the residual between the two, making the final output image closer to the original image and achieving better coding performance.

[0051] Figure 3 shows a schematic diagram of a multi-layer residual network architecture. As shown in Figure 3, block partitioning information, prediction information, and quantization parameter (QP) information are introduced as network inputs, achieving good coding performance. The network also uses the same scaling factor to achieve even better performance.

[0052] In the above embodiment, a certain calculation formula is defined to derive the scaling factor. The specific formula is as follows:

[0053] Assume that the width of an image is W and the height is H. (x, y) represents the coordinates of the pixel, where x is the horizontal coordinate and y is the vertical coordinate. The reconstructed image input to the neural network is rec(x, y), the filtered image output by the neural network is cnn(x, y), and the original image is org(x, y).

[0054] The calculation of the scaling factor is based on the coding unit. For example, for the Coding Tree Unit (CTU), assuming that the current image has a total of N CTUs and the current CTU is numbered i, the residual between the CTUs at the corresponding positions of the two images is first calculated as follows:

[0055] orgResi(i)=[org(x i ,y i )-rec(x i ,y i )] (1)

[0056] cnnResi(i)=[cnn(x i ,y i )-rec(x i ,y i )] (2)

[0057] Then traverse all CTU units and perform weighted summation of the residual of the entire image in various forms, as shown in the following formula:

[0058]

[0059]

[0060]

[0061]

[0062] Finally, the least squares method is used to calculate the statistical one-dimensional scaling factor SF at the image level, as shown in the following formula:

[0063]

[0064] According to the derived SF, the residuals of the reconstructed image and the filtered image are corrected to obtain the final output image output(x i ,y i ), as follows:

[0065] output(x i ,y i )=rec(x i ,y i )+SF×[cnn(xi ,y i )-rec(x i ,y i )] (8)

[0066] The scaling factor itself needs to be derived at the encoding end and then encoded using a certain number of bits before it can be used after being parsed at the decoding end.

[0067] To achieve better performance, neural network-based loop filtering tools all introduce scaling factors. However, the scaling factors are often manually defined and calculated in a fixed manner. The statistically derived image-level scaling factor is applied to all CTUs of the entire image, but the scaling factor applied to the entire image is not necessarily the most appropriate scaling factor for each CTU. Furthermore, encoding the scaling factor requires a certain number of bits, and calculating the image-level scaling factor requires traversing all CTUs of the entire image before processing the current image, resulting in significant latency.

[0068] That is, in the above embodiments, the flexibility and applicability of the scaling factor are poor, which reduces the coding performance, and the compression and transmission of the scaling factor also greatly reduces the coding efficiency.

[0069] To address the above-mentioned issues, embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a storage medium. The decoder decodes a bitstream and determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. The encoder determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the codec can use the scaling factor generation network on the codec side to predict and obtain a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image. This not only enhances the flexibility and applicability of the scaling factor, but also improves the coding performance and coding efficiency.

[0070] Refer to Figure 4, which shows an example of a system composition block diagram of an encoder provided in an embodiment of the present application. As shown in Figure 4, the encoder 100 may include: a segmentation unit 101, a prediction unit 102, a first adder 107, a transform unit 108, a quantization unit 109, an inverse quantization unit 110, an inverse transform unit 111, a second adder 112, a filtering unit 113, a decoded picture buffer (DPB) unit 114 and an entropy coding unit 115. Here, the input of the encoder 100 can be a video consisting of a series of pictures or a static picture, and the output of the encoder 100 can be a bit stream (also referred to as a "code stream") for representing a compressed version of the input video.

[0071] Among them, the segmentation unit 101 segments the picture in the input video into one or more Coding Tree Units (CTUs). The segmentation unit 101 divides the picture into multiple tiles (or tiles), and can further divide a tile into one or more bricks. Here, a tile or a brick may include one or more complete and / or partial CTUs. In addition, the segmentation unit 101 can form one or more slices, where a slice may include one or more tiles arranged in a grid order in the picture, or one or more tiles covering a rectangular area in the picture. The segmentation unit 101 can also form one or more sub-pictures, where a sub-picture may include one or more slices, tiles or bricks.

[0072] During the encoding process of encoder 100, segmentation unit 101 transmits the CTU to prediction unit 102. Generally, prediction unit 102 may be composed of block segmentation unit 103, motion estimation (ME) unit 104, motion compensation (MC) unit 105, and intra prediction unit 106. Specifically, block segmentation unit 103 iteratively uses quadtree segmentation, binary tree segmentation, and ternary tree segmentation to further divide the input CTU into smaller coding units (CUs). Prediction unit 102 may use ME unit 104 and MC unit 105 to obtain inter-frame prediction blocks for the CU. Intra-frame prediction unit 106 may use various intra-frame prediction modes, including MIP mode, to obtain intra-frame prediction blocks for the CU. In an example, a rate-distortion optimized motion estimation method may be used by ME unit 104 and MC unit 105 to obtain inter-frame prediction blocks, and a rate-distortion optimized mode determination method may be used by intra-frame prediction unit 106 to obtain intra-frame prediction blocks.

[0073] The prediction unit 102 outputs the prediction block of the CU, and the first adder 107 calculates the difference between the CU in the output of the segmentation unit 101 and the prediction block of the CU, i.e., the residual CU. The transform unit 108 reads the residual CU and performs one or more transform operations on the residual CU to obtain coefficients. The quantization unit 109 quantizes the coefficients and outputs the quantized coefficients (i.e., levels). The inverse quantization unit 110 performs a scaling operation on the quantized coefficients to output reconstructed coefficients. The inverse transform unit 111 performs one or more inverse transforms corresponding to the transform in the transform unit 108 and outputs the reconstructed residual. The second adder 112 calculates the reconstructed CU by adding the reconstructed residual and the prediction block of the CU from the prediction unit 102. The second adder 112 also sends its output to the prediction unit 102 for use as an intra-frame prediction reference. After all CUs in the picture or sub-picture are reconstructed, the filtering unit 113 performs loop filtering on the reconstructed picture or sub-picture. Here, the filtering unit 113 includes one or more filters, such as a deblocking filter, a sample adaptive offset (SAO) filter, an adaptive loop filter (ALF), a luma mapping and chroma scaling (LMCS) filter, and a neural network-based filter. Alternatively, when the filtering unit 113 determines that a CU is not used as a reference for encoding other CUs, the filtering unit 113 performs loop filtering on one or more target pixels in the CU.

[0074] The output of the filtering unit 113 is a decoded picture or sub-picture, which is cached to the DPB unit 114. The DPB unit 114 outputs the decoded picture or sub-picture based on the timing and control information. Here, the picture stored in the DPB unit 114 can also be used as a reference for the prediction unit 102 to perform inter-frame prediction or intra-frame prediction. Finally, the entropy coding unit 115 converts the parameters required for decoding the picture from the encoder 100 (such as control parameters and supplementary information, etc.) into binary form and writes this binary form into the code stream according to the syntax structure of each data unit. That is, the encoder 100 finally outputs the code stream.

[0075] Furthermore, encoder 100 may include a first processor and a first memory storing a computer program. When the first processor reads and executes the computer program, encoder 100 reads the input video and generates a corresponding bitstream. Alternatively, encoder 100 may be a computing device comprising one or more chips. These units, implemented as integrated circuits on the chip, have similar connectivity and data exchange functions as the corresponding units in FIG8 .

[0076] Referring to Figure 5 , an example block diagram of a system composition of a decoder provided in an embodiment of the present application is shown. As shown in Figure 5 , the decoder 200 may include: a parsing unit 201, a prediction unit 202, an inverse quantization unit 205, an inverse transform unit 206, an adder 207, a filtering unit 208, and a decoded picture buffer unit 209. Here, the input of the decoder 200 is a bitstream representing a compressed version of a video or a still picture, and the output of the decoder 200 may be a decoded video consisting of a series of pictures or a decoded still picture.

[0077] The input codestream to decoder 200 may be the codestream generated by encoder 100. Parsing unit 201 parses the input codestream and obtains syntax element values ​​from the input codestream. Parsing unit 201 converts the binary representation of the syntax elements into digital values ​​and sends the digital values ​​to units within decoder 200 to obtain one or more decoded pictures. Parsing unit 201 may also parse one or more syntax elements from the input codestream to display decoded pictures.

[0078] During the decoding process of the decoder 200 , the parsing unit 201 sends the values ​​of the syntax elements and one or more variables set or determined according to the values ​​of the syntax elements and used to obtain one or more decoded pictures to the units in the decoder 200 .

[0079] The prediction unit 202 determines a prediction block for the current decoding block (e.g., CU). Here, the prediction unit 202 may include a motion compensation unit 203 and an intra-frame prediction unit 204. Specifically, when the inter-frame decoding mode is indicated for decoding the current decoding block, the prediction unit 202 passes the relevant parameters from the parsing unit 201 to the motion compensation unit 203 to obtain an inter-frame prediction block; when the intra-frame prediction mode (including the MIP mode indicated based on the MIP mode index value) is indicated for decoding the current decoding block, the prediction unit 202 passes the relevant parameters from the parsing unit 201 to the intra-frame prediction unit 204 to obtain an intra-frame prediction block.

[0080] The inverse quantization unit 205 has the same function as the inverse quantization unit 110 in the encoder 100. The inverse quantization unit 205 performs a scaling operation on the quantization coefficients (ie, levels) from the parsing unit 201 to obtain reconstructed coefficients.

[0081] The inverse transform unit 206 has the same function as the inverse transform unit 111 in the encoder 100. The inverse transform unit 206 performs one or more transform operations (ie, inverse operations of one or more transform operations performed by the inverse transform unit 111 in the encoder 100) to obtain a reconstructed residual.

[0082] The adder 207 performs an addition operation on its input (the prediction block from the prediction unit 202 and the reconstructed residual from the inverse transform unit 206) to obtain a reconstructed block of the current decoded block. The reconstructed block is also sent to the prediction unit 202 to be used as a reference for other blocks encoded in the intra prediction mode.

[0083] After all CUs in the picture or sub-picture are reconstructed, the filtering unit 208 performs loop filtering on the reconstructed picture or sub-picture. The filtering unit 208 includes one or more filters, such as a deblocking filter, a sample adaptive offset filter, an adaptive loop filter, a luminance mapping and chroma scaling filter, and a neural network-based filter. Alternatively, when the filtering unit 208 determines that the reconstructed block is not used as a reference for decoding other blocks, the filtering unit 208 performs loop filtering on one or more target pixels in the reconstructed block. Here, the output of the filtering unit 208 is a decoded picture or sub-picture, which is cached to the DPB unit 209. The DPB unit 209 outputs the decoded picture or sub-picture based on timing and control information. The picture stored in the DPB unit 209 can also be used as a reference for performing inter-frame prediction or intra-frame prediction by the prediction unit 202.

[0084] Furthermore, the decoder 200 can be a second memory having a second processor and a computer program. When the first processor reads and executes the computer program, the decoder 200 reads the input bit stream and generates the corresponding decoded video. In addition, the decoder 200 can also be a computing device having one or more chips. These units implemented as integrated circuits on the chip have similar connection and data exchange functions as the corresponding units in Figure 5.

[0085] It should also be noted that when the embodiment of the present application is applied to the encoder 100, the "current block" specifically refers to the current block to be encoded in the video image (which can also be simply referred to as the "encoding block"); when the embodiment of the present application is applied to the decoder 200, the "current block" specifically refers to the current block to be decoded in the video image (which can also be simply referred to as the "decoding block").

[0086] Based on FIG4 , the encoding method in the embodiment of the present application is mainly applied to the “filtering unit 113 ” part in the encoder 100 .

[0087] Based on FIG. 5 , the decoding method in the embodiment of the present application is mainly applied to the “filtering unit 208 ” part in the decoder 200 .

[0088] That is to say, the encoding and decoding method in the embodiments of the present application can be applied to a video encoding system (referred to as "encoder" for short), a video decoding system (referred to as "decoder" for short), and can even be applied to both a video encoding system and a video decoding system at the same time, but no limitation is made here.

[0089] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0090] One embodiment of the present application provides a decoding method, which is applied to a decoder. FIG6 is a first schematic diagram of an implementation flow of the decoding method. As shown in FIG6 , the decoding method performed by the decoder may include the following steps:

[0091] Step 101: Decode the code stream and determine first identification information.

[0092] In the embodiment of the present application, the decoder may determine the first identification information by decoding the code stream.

[0093] It should be noted that, in the implementation of the present application, the first identification information can be used to determine whether to use a scaling factor generated by a neural network for the current color component of the current frame of the current sequence. If a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, a scaling factor corresponding to at least one image block in the current color component of the current frame of the current sequence can be generated, and subsequent decoding processing can be performed using the scaling factor corresponding to the image block.

[0094] In an embodiment of the present application, the first identification information can be understood as a flag indicating whether a scaling factor generated by a neural network is used for at least one image block in a current color component of a current frame of a current sequence, or the first identification information can be understood as a flag indicating whether a scaling factor corresponding to the image block is used to determine a reconstructed block of the image block for at least one image block in a current color component of a current frame of the current sequence. Specifically, when a decoder decodes a bitstream, it can determine a variable serving as the first identification information, thereby determining the first identification information by taking the value of the variable.

[0095] It should be noted that, in the present application, the value of the first identification information may be the first value or the second value. Different values ​​of the first identification information may lead to different methods for determining the reconstructed block of at least one image block in the current color component of the current frame of the current sequence.

[0096] It can be understood that in the embodiments of the present application, based on the value of the first identification information, when decoding the current color component, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0097] For example, in the present application, if the value of the first identification information is a first value, then the scaling factor generated by the neural network may be selected for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current color component. Meanwhile, if the value of the first identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may not be used in the decoding process of the current color component.

[0098] For example, in the present application, if the value of the first identification information is a first value, then the scaling factor generated by the neural network may be selected for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current color component. Alternatively, if the value of the first identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may not be used in the decoding process of the current color component.

[0099] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0100] Furthermore, in an embodiment of the present application, the flag picture_nnsf_enable_flag[compIdx] may be used to represent the first identification information, where compIdx may be used to represent the color component of the current frame of the current sequence.

[0101] For example, in this application, if the value of picture_nnsf_enable_flag[compIdx] is 1, it can indicate that the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence, and then the scaling factor determined based on the neural network is used to complete the decoding process of the current color component.

[0102] For example, in the present application, if the value of picture_nnsf_enable_flag[compIdx] is 0, it may indicate that the scaling factor generated by the neural network is not used for the current color component of the current frame of the current sequence, and thus the scaling factor determined based on the neural network is not used in the decoding process of the current color component.

[0103] It should be noted that, in the embodiment of the present application, the first identification information may be a picture-level flag, wherein the first identification information may be determined by decoding a code stream.

[0104] Furthermore, in an embodiment of the present application, FIG7 is a second schematic diagram of a decoding method implementation flow. As shown in FIG7 , the method for the decoder to perform decoding processing may further include the following steps:

[0105] Step 105: Determine the third identification information.

[0106] Step 106: If the third identification information indicates that the scaling factor generated by the neural network is used for the current sequence, the determination process of the first identification information is executed.

[0107] In an embodiment of the present application, the third identification information may be determined first, and then whether to further determine the first identification information is determined based on the third identification information. If the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, then the first identification information may be determined by decoding the bitstream.

[0108] It should be noted that, in the implementation of this application, the third identification information can be used to determine whether to use the scaling factor generated by the neural network for the current sequence. That is, if the scaling factor generated by the neural network is used for the current sequence, a scaling factor corresponding to an image block can be generated for at least one image frame in the current sequence, and the scaling factor corresponding to the image block can then be used for subsequent decoding processing.

[0109] In embodiments of the present application, the third identification information can be understood as a flag indicating whether to use a scaling factor generated by a neural network for at least one image frame in the current sequence, or as a flag indicating whether to use a scaling factor corresponding to an image block to determine a reconstructed block of the image block for at least one image frame in the current sequence. Specifically, a variable can be determined as the third identification information, so that the third identification information can be determined by the value of the variable.

[0110] It should be noted that, in the present application, the value of the third identification information may be the first value or the second value. Different values ​​of the third identification information may lead to different ways of determining the reconstructed block of the image block of at least one image frame in the current sequence.

[0111] It can be understood that in the embodiments of the present application, based on the value of the third identification information, when decoding at least one image frame in the current sequence, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0112] For example, in this application, if the value of the third identification information is a first value, then the scaling factor generated by the neural network may be selected for the current sequence, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current sequence. Meanwhile, if the value of the third identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may not be used in the decoding process of the current color component.

[0113] For example, in the present application, if the value of the third identification information is a first value, then the scaling factor generated by the neural network may be selected for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current color component. Alternatively, if the value of the third identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current sequence, and the scaling factor determined based on the neural network may not be used in the decoding process of the current sequence.

[0114] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0115] Furthermore, in an embodiment of the present application, the flag nnsf_enable_flag may be used to represent the third identification information.

[0116] For example, in this application, if the value of nnsf_enable_flag is 1, it can indicate that the scaling factor generated by the neural network is used for the current sequence, and then the scaling factor determined based on the neural network is used to complete the decoding process of the current sequence.

[0117] For example, in this application, if the value of nnsf_enable_flag is 0, it may indicate that the scaling factor generated by the neural network is not used for the current sequence, and thus the scaling factor determined based on the neural network is not used in the decoding process of the current sequence.

[0118] It should be noted that, in the embodiment of the present application, the third identification information may be a sequence-level flag, wherein the third identification information may be pre-set and does not need to be determined by decoding the code stream.

[0119] It can be seen that in the present application, the determination of the first identification information can depend on the determination of the third identification information. Of course, the determination of the first identification information can also depend on some other flags, that is, picture_nnsf_enable_flag[compIdx] may depend on some other flags, which is not specifically limited in the embodiments of the present application.

[0120] It should be noted that in the embodiments of the present application, the third identification information can be a sequence-level flag, or even a higher-level flag, such as video usability information (VUI), supplemental enhancement information (SEI), etc., which is not specifically limited in the embodiments of the present application.

[0121] Step 102: If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined.

[0122] In an embodiment of the present application, after determining the first identification information by decoding the code stream, if the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, then the second identification information can be further determined.

[0123] It can be understood that in an embodiment of the present application, if the first identification information is used to determine that the scaling factor generated by the neural network is used to generate the scaling factor of the image block for the current color component of the current frame of the current sequence, then the second identification information needs to be further determined to determine which image blocks in all the image blocks in the current color component use the scaling factor generated by the neural network based on the second identification information.

[0124] That is, in an embodiment of the present application, the second identification information can be used to determine whether to use a scaling factor generated by a neural network for the current block of the current color component of the current frame of the current sequence. If a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, a scaling factor corresponding to the current block can be generated for the current block, and subsequent decoding processing can be performed using the scaling factor corresponding to the current block.

[0125] In an embodiment of the present application, the second identification information can be understood as a flag indicating whether to use a scaling factor generated by a neural network for the current block in the current color component of the current frame of the current sequence, or the second identification information can be understood as a flag indicating whether to use the scaling factor corresponding to the current block to determine a reconstructed block of the current block in the current color component of the current frame of the current sequence. Specifically, the decoder can determine a variable as the second identification information, so that the second identification information can be determined by the value of the variable.

[0126] It should be noted that, in this application, the value of the second identification information may be the first value or the second value. Different values ​​of the second identification information may lead to different ways of determining the reconstructed block of the current block in the current color component of the current frame of the current sequence.

[0127] It can be understood that in the embodiments of the present application, based on the value of the second identification information, when decoding the current block of the current color component, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0128] For example, in the present application, if the value of the second identification information is a first value, then the scaling factor generated by the neural network may be selected for the current block of the current color component, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current block. Meanwhile, if the value of the second identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current block of the current color component, and the scaling factor determined based on the neural network may not be used in the decoding process of the current block.

[0129] For example, in the present application, if the value of the second identification information is a first value, then the scaling factor generated by the neural network may be selected for the current block of the current color component, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current block. Alternatively, if the value of the second identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current block of the current color component, and the scaling factor determined based on the neural network may not be used in the decoding process of the current block.

[0130] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0131] Furthermore, in an embodiment of the present application, taking the coding tree unit CTU as an example, the flag patch_nnsf_enable_flag[compIdx][ctuIdx] can be used to represent the second identification information, where compIdx can be used to represent the color component of the current frame of the current sequence, and ctuIdx can be used to represent the image block of the current color component of the current frame of the current sequence.

[0132] For example, in the present application, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 1, it may indicate that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, and then the scaling factor determined based on the neural network is used to complete the decoding process of the current block.

[0133] For example, in the present application, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 0, it may indicate that the scaling factor generated by the neural network is not used for the current block of the current color component of the current frame of the current sequence, and thus the scaling factor determined based on the neural network is not used in the decoding process of the current block.

[0134] It should be noted that, in the embodiment of the present application, the second identification information may be a block-level flag, wherein the second identification information may be determined by decoding the code stream.

[0135] Furthermore, in an embodiment of the present application, the second identification information may be a block-level flag, and the block-level flag may include a maximum coding unit (LCU) level flag, a coding tree unit (CTU) level flag, a coding unit (CU) level flag, or other block-level flags. In the embodiment of the present application, the coding tree unit CTU is mainly used as an example for illustration, but the specific form of the block-level flag is not limited.

[0136] Step 103: If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame, the scaling factor generation network is used to determine the scaling factor corresponding to the current block of the current color component.

[0137] In an embodiment of the present application, after determining the second identification information, if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, then the scaling factor generation network can be further used to generate a scaling factor corresponding to the current block.

[0138] It should be noted that in the embodiments of the present application, the scaling factor generation network can be used to predict the scaling factor of an image block. That is, the scaling factor generation network can be a network that predicts the scaling factor of an image block. Therefore, the scaling factor generation network can be used to perform a scaling factor determination process for an image block, and the generated scaling factor is corresponding to the image block, that is, a block-level scaling factor is generated.

[0139] It can be understood that in an embodiment of the present application, when using a scaling factor generation network to determine the scaling factor corresponding to the current block of the current color component, the filtered image block and the reconstructed image block corresponding to the current block can be optionally input into the scaling factor generation network, so that the scaling factor of the current block can be output.

[0140] It should be noted that in the embodiments of the present application, the input of the scaling factor generation network may include a filtered image block generated by a neural network based loop filter (NNLF), and may also include a reconstructed image block determined by sample adaptive offset (SAO). Accordingly, the output of the scaling factor generation network may be the scaling factor corresponding to the current block.

[0141] Furthermore, in an embodiment of the present application, when using a scaling factor generation network to determine a scaling factor corresponding to a current block of a current color component, the filtered image block, the reconstructed image block, and feature information corresponding to the current block may be input into the scaling factor generation network to output a scaling factor for the current block. The feature information corresponding to the current block may include one or more of block partition information, prediction information, and quantization parameter information.

[0142] That is, in the present application, when determining the scaling factor of the current block based on the scaling factor generation network, in addition to processing the filtered image block and the reconstructed image block of the current block, more feature information of the current block can also be input. For example, the block partition information of the current block, and / or the prediction information of the current block, and / or the QP information of the current block can be used as one of the inputs of the scaling factor generation network.

[0143] Furthermore, in embodiments of the present application, the network structure of the scaling factor generation network may include at least one of a convolutional layer, a nonlinear activation function PReLU, a Concat layer, a maximum pooling layer, and a fully connected FullConnect layer. That is, the network structure of the scaling factor generation network may include a convolutional layer, and / or a nonlinear activation function PReLU, and / or a Concat layer, and / or a maximum pooling layer, and / or a fully connected FullConnect layer.

[0144] Accordingly, in the embodiments of the present application, the data format of the scaling factor generated by the scaling factor generation network may also vary based on different network structures. For example, the data format of the scaling factor may be a two-dimensional matrix or a one-dimensional variable, which is not specifically limited in this application.

[0145] Exemplarily, in an embodiment of the present application, the network structure of the scaling factor generation network may include a convolution layer, a nonlinear activation function PReLU layer, and a Concat layer. Accordingly, the data form of the scaling factor of the image block generated by the scaling factor generation network may be a two-dimensional matrix, that is, the scaling factor has the same dimension as the image block. For example, FIG8 is a schematic diagram of the network structure of the scaling factor generation network. As shown in FIG8 , the structure of the scaling factor generation network consists of multiple convolution layers. For the input filtered image block and the reconstructed image block, each needs to undergo a layer of convolution and a nonlinear activation function PReLU; then the feature maps are spliced ​​together through the Concat layer; finally, after passing through K layers of convolution and K-1 layers of nonlinear activation function PReLU, the scaling factor is predicted. Wherein, K is an integer greater than 1.

[0146] Exemplarily, in an embodiment of the present application, the network structure of the scaling factor generation network may further include a convolution layer, a nonlinear activation function PReLU layer, a Concat layer, a maximum pooling MaxPooling layer, and a fully connected FullConnect layer. Accordingly, the data form of the scaling factor of the image block generated by the scaling factor generation network may be a one-dimensional variable. For example, FIG9 is a second schematic diagram of the network structure of the scaling factor generation network. As shown in FIG9 , the structure of the scaling factor generation network consists of multiple convolution layers. For the input filtered image block and the reconstructed image block, each needs to undergo a layer of convolution and a nonlinear activation function PReLU; then the feature maps are spliced ​​together through the Concat layer; then K layers of convolution and K layers of maximum pooling MaxPooling are passed; finally, after passing through L layers of fully connected layers FullConnect and L-1 layers of nonlinear activation function PreLU, the scaling factor is predicted. Wherein, L is an integer greater than 1.

[0147] It should be noted that in the embodiments of the present application, the neural network loop filter generally performs training iterations on the network by calculating the minimum mean square error loss between the network output image and the original image. Since the training goal of the filter network is clear, that is, the output of the trained network is closer to the original image (i.e., the label), using the minimum mean square error as the loss function is more effective.

[0148] Furthermore, in an embodiment of the present application, a scaling factor set can be used to train a scaling factor generation network, so that data labels can be determined from the scaling factor set; then, the loss of the scaling factor generation network can be determined based on the output data and data labels of the scaling factor generation network; and finally, the network parameters of the scaling factor generation network can be updated based on the loss.

[0149] For the neural network scaling factor structure shown in FIG9 above, as proposed in the embodiments of this application, the training goal is to obtain a scaling factor that is closer to the theoretically optimal scaling factor (direct goal), thereby making the final corrected image closer to the original image (indirect goal). The original image (indirect goal) can be directly obtained from the dataset; however, the theoretically optimal scaling factor (direct goal) is actually unknown before training.

[0150] In other words, in order to better train for the direct goal, the embodiments of the present application propose an adaptive label update training method. By finding the optimal scaling factor during training, the labels are adaptively updated to achieve better training results.

[0151] During training, for the current input reconstructed image block rec(x i ,y i ), first load a certain neural network filter and get the filtered image block cnn(x i ,y i Then, referring to the correction formula of the scaling factor, the scaling factors SF_i within a specific range [SFmin, SFmax] (i.e., the scaling factor set) are traversed (for example, the traditional scaling factor calculation method can be referred to, and the range is set to [16, 320]), and the output reconstructed block output (x i ,y i ) and the original image org(x i ,y i ) to find the optimal scaling factor SF_best, which is used as the label SF_label (data label) of the current training data. The specific training pseudo code is as follows:

[0152] minloss=maxVALUE

[0153] SF_best=-1

[0154] for SF_i in range(SFmin,SFmax):

[0155] output=rec+SF_i*(cnn-rec)

[0156] loss = MSE (output, org)

[0157] if(loss <minloss):

[0158] minloss=loss

[0159] SF_best=SF_i

[0160] SF_label=SF_best

[0161] After finding SF_label, the label of the current training data is updated adaptively.

[0162] Finally, by calculating the mean absolute error MAE between the output of the scaling factor network [NNSF]_b and the label SF_label, the network loss is calculated, and training iterations and parameter updates are performed.

[0163] Step 104: Determine a reconstructed block of the current block according to the scaling factor.

[0164] In an embodiment of the present application, after the scaling factor corresponding to the current block of the current color component is determined using the scaling factor generation network, a reconstructed block of the current block can be further determined according to the scaling factor.

[0165] It should be noted that, in the embodiment of the present application, when determining the reconstructed block of the current block according to the scaling factor, the reconstructed block of the current block can be determined according to the scaling factor, the filtered image block and the reconstructed image block.

[0166] It is understandable that, in the embodiments of the present application, the scaling factor can be used to correct the residual between the filtered image block and the reconstructed image block, thereby completing the reconstruction of the current block and obtaining the corresponding reconstructed block.

[0167] Furthermore, in an embodiment of the present application, when determining the reconstructed block of the current block based on the scaling factor, the filtered image block, and the reconstructed image block, the residual value corresponding to the current block can be first determined based on the filtered image block and the reconstructed image block; then, the residual value is corrected based on the scaling factor to obtain a corrected result corresponding to the current block; finally, the reconstructed block of the current block can be determined based on the corrected result and the reconstructed image block.

[0168] For example, in the embodiment of the present application, for the current block (x i ,y i ), if the scaling factor generated by the scaling factor generation network is a two-dimensional matrix NNSF a (x i ,y i), then use the filter image block cnn(x i ,y i ) and reconstructed image block rec(x i ,y i ) Determine the reconstructed block output(x) of the current block i ,y i ) process can be expressed as the following formula:

[0169] output(x i ,y i )=rec(x i ,y i )+NNSF a (x i ,y i )×[cnn(x i ,y i )-rec(x i ,y i )] (9)

[0170] For example, in the embodiment of the present application, for the current block (x i ,y i ), if the scaling factor predicted by the scaling factor generating network is a one-dimensional variable NNSF b , then use the filter image block cnn(x i ,y i ) and reconstructed image block rec(x i ,y i ) Determine the reconstructed block output(x) of the current block i ,y i ) process can be expressed as the following formula:

[0171] output(x i ,y i )=rec(x i ,y i )+NNSF b ×[cnn(x i ,y i )-rec(x i ,y i )] (10)

[0172] It is understood that in the embodiments of the present application, the above methods are all based on the example of using the technology uniformly for all components in the video, such as the R, G, and B components of an RGB format video, or the Y, U, and V (Y, Cb, Cr) components of a YUV format. The above methods can also be used for only one component, such as only the Y component of a YUV format. The above methods can also be used for each component separately, that is, each component can be controlled separately. This application does not impose any specific restrictions.

[0173] Furthermore, in an embodiment of the present application, Figure 10 is an application diagram of a coding framework proposed in an embodiment of the present application. As shown in Figure 10, a scaling factor module for scaling factor processing is added to the inloop filter (Inloop Filter) module of the hybrid coding framework, namely, a neural network based scaling factor (NNSF) module. The scaling factor module can be used to predict the scaling factor of the image block using a neural network (scaling factor generation network), and complete the reconstruction of the image block based on the scaling factor to obtain the corresponding reconstructed block.

[0174] It should be noted that, in the embodiment of the present application, the scaling factor module NNSF may be located after the neural network-based loop filter NNLF in the loop filtering module and before the adaptive correction filter ALF.

[0175] It can be understood that, in the embodiment of the present application, the use of the scaling factor module NNSF does not depend on the switches of the deblocking filter DBF, the sample adaptive offset SAO, the neural network-based loop filter NNLF, and the adaptive correction filter ALF.

[0176] That is to say, whether the learning-based scaling factor module NNSF is turned on can be determined by the encoder, so in the embodiment of the present application, the scaling factor module can be set to switch control at the frame level and the block (such as CTU) level. Among them, at the encoding end, it is possible to judge whether to turn on the scaling factor module NNSF by comparing the image rate distortion cost before and after the scaling factor processing, that is, to judge whether to use the scaling factor generated by the neural network, and the switching status of the scaling factor module NNSF will be encoded into the bitstream for the decoder to read. At the decoder end, when it is parsed that the scaling factor module is turned on, that is, when it is determined to use the scaling factor generated by the neural network, the scaling factor of the image block is used to optimize the image block; and when it is parsed that the scaling factor module is closed, that is, when it is determined not to use the scaling factor generated by the neural network, no related processing of the scaling factor is performed.

[0177] Furthermore, in an embodiment of the present application, the scaling factor obtained based on the scaling factor generation network prediction can be a learning-based, image block scaling factor, which is used to correct the residual between the filtered image block output by the NNLF and the reconstructed image block output by the SAO, so that the reconstructed block of the current block finally obtained is closer to the original image, so as to obtain better coding performance and solve the problems existing in the traditional scaling factor to a certain extent.

[0178] It is understood that in the embodiments of the present application, the network structure of the scaling factor generation network can be a relatively simple convolutional neural network, with the input being the filtered image block output by the NNLF and the reconstructed image block output by the SAO, and the output being the scaling factor of the current block (e.g., CTU unit). That is, the learning-based scaling factor of the current block is predicted by the neural network.

[0179] Furthermore, in an embodiment of the present application, a technical solution of using a neural network (scaling factor generation network) to predict the scaling factor of an image block is implemented at both the codec end and can act in a loop filter module.

[0180] Exemplarily, in an embodiment of the present application, the decoder acquires and parses the bitstream. When parsing the loop filter module, it processes the bitstream according to the specified filter order. When entering the scaling factor module, it first determines whether the neural network scaling factor module can be used in the current sequence based on the sequence-level flag nnsf_enable_flag (third identification information), that is, it determines whether to use the scaling factor generated by the neural network for at least one image frame in the current sequence based on the third identification information. If nnsf_enable_flag is "1", it is determined that the scaling factor module is processed for the current sequence, that is, it is determined that the third identification information indicates that the scaling factor generated by the neural network is used for at least one image frame in the current sequence, and then the first identification information can be further determined; if nnsf_enable_flag is "0", it is determined that the scaling factor module is not used for the current sequence, that is, it is determined that the third identification information indicates that the scaling factor generated by the neural network is not used for all image frames in the current sequence, and then ALF processing can continue.

[0181] Accordingly, in an embodiment of the present application, after determining that the third identification information indicates that a scaling factor generated by a neural network is used for at least one image frame of the current sequence, and determining the first identification information (image-level flag picture_nnsf_enable_flag[compIdx]), for the current frame of the current sequence, if picture_nnsf_enable_flag[compIdx] is "1", it is determined that the scaling factor module is processed for the current color component in the current frame of the current sequence, that is, it is determined that the first identification information indicates that the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence, then the second identification information can be further determined; if picture_nnsf_enable_flag[compIdx] is "0", it is determined that the scaling factor module is not used for the current color component in the current frame of the current sequence, that is, it is determined that the first identification information indicates that the scaling factor generated by the neural network is not used for the current color component in the current frame of the current sequence, then ALF processing can continue.

[0182] Accordingly, in an embodiment of the present application, after determining that the first identification information indicates that the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence, and determining the second identification information (block-level flag patch_nnsf_enable_flag[compIdx][ctuIdx]), for the current color component of the current frame of the current sequence, if patch_nnsf_enable_flag[compIdx][ctuIdx] is "1", it is determined that the scaling factor module is processed for the current block in the current color component, that is, it is determined that the second identification information indicates that the scaling factor generated by the neural network is used for the current block of the current color component of the current frame of the current sequence; if patch_nnsf_enable_flag[compIdx][ctuIdx] is "0", it is determined that the scaling factor module is not used for the current block in the current color component of the current frame of the current sequence, that is, it is determined that the second identification information indicates that the scaling factor generated by the neural network is not used for the current block in the current color component, and then ALF processing can continue.

[0183] It can be understood that in the implementation of the present application, after traversing all image blocks in the current color component of the current frame of the current sequence, the next color component of the current frame of the current sequence can be loaded, and then all color components of the current frame of the current sequence can be traversed according to the above method. After completing the decision of the neural network scaling factor module of the current frame, the next frame of the current sequence can be loaded for processing.

[0184] Furthermore, in an embodiment of the present application, Figure 11 is a schematic diagram of the overall framework of the loop filtering module. As shown in Figure 11, the loop filtering module mainly includes a deblocking filter DBF, a sample adaptive compensation SAO, a neural network-based loop filter NNLF, a neural network-based scaling factor NNSF module, an adaptive correction filter ALF and other tools.

[0185] Among them, the network structure of the NNSF module (scaling factor generation network) can be composed of a relatively simple convolutional neural network, which can output the scaling factor of the image block in the form of a two-dimensional matrix as shown in Figure 8 above; or it can output the scaling factor of the image block in the form of a one-dimensional variable as shown in Figure 9 above.

[0186] Exemplarily, in an embodiment of the present application, for an image block, the reconstructed image block corresponding to the image block output by the SAO can be input into the NNLF and NNSF modules respectively, and then the NNLF outputs the filtered image block corresponding to the image block. The filtered image block is input into the NNSF module, and then the scaling factor corresponding to the image block and generated by the neural network can be output, so that the scaling factor can be used to correct the residual between the filtered image block and the reconstructed image block, and finally the reconstructed block corresponding to the image block is output, and the reconstructed block continues to be input into the ALF to perform the subsequent filtering processing.

[0187] Furthermore, in an embodiment of the present application, the sequence header definition syntax is shown in Table 1:

[0188] Table 1

[0189]

[0190] Among them, grammatically, the permission flag of the scaling factor generated by the neural network is nnsf_enable_flag, that is, the third identification information used to indicate whether to use the scaling factor generated by the neural network for the current sequence is nnsf_enable_flag.

[0191] For example, if the value of nnsf_enable_flag is 1, it can be determined that the scaling factor module is processed for the current sequence, that is, it can be determined that the scaling factor generated by the neural network can be used for the current sequence; if the value of nnsf_enable_flag is 0, it can be determined that the scaling factor generated by the neural network is not used for the current sequence.

[0192] Furthermore, in an embodiment of the present application, the image header definition syntax is shown in Table 2:

[0193] Table 2

[0194]

[0195] Among them, grammatically, when the value of the neural network generated scaling factor enable flag nnsf_enable_flag is 1, that is, when the third identification information indicates that the scaling factor generated by the neural network is used for the current sequence, it can be determined that the image-level neural network scaling factor enable flag is picture_nnsf_enable_flag[compIdx], that is, the first identification information used to indicate whether the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence is picture_nnsf_enable_flag[compIdx].

[0196] For example, if the value of picture_nnsf_enable_flag[compIdx] is 1, it can be determined that the current color component of the current frame of the current sequence is processed by the scaling factor module, that is, it is determined that the scaling factor generated by the neural network can be used for the current color component of the current frame of the current sequence; if the value of picture_nnsf_enable_flag[compIdx] is 0, it can be determined that the current color component of the current frame of the current sequence does not use the scaling factor generated by the neural network.

[0197] Furthermore, in an embodiment of the present application, the slice definition syntax is shown in Table 3:

[0198] Table 3

[0199]

[0200] Among them, grammatically, when the value of the neural network generated scaling factor enable flag nnsf_enable_flag is 1, that is, when the third identification information indicates the use of the scaling factor generated by the neural network for the current sequence, and at the same time, it is determined that the value of the image-level neural network scaling factor enable flag picture_nnsf_enable_flag[compIdx] is 1, that is, when the first identification information indicates the use of the scaling factor generated by the neural network for the current color component of the current frame of the current sequence, it can be determined that the block-level neural network scaling factor enable flag is patch_nnsf_enable_flag[compIdx][ctuIdx], that is, the second identification information used to indicate whether the scaling factor generated by the neural network is used for the current block of the current color component of the current frame of the current sequence is patch_nnsf_enable_flag[compIdx][ctuIdx].

[0201] For example, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 1, it can be determined that the current block of the current color component of the current frame of the current sequence is processed by the scaling factor module, that is, it is determined that the scaling factor generated by the neural network can be used for the current block of the current color component of the current frame of the current sequence; if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 0, it can be determined that the current block of the current color component of the current frame of the current sequence does not use the scaling factor generated by the neural network.

[0202] In summary, in the embodiment of the present application, the decoding method proposed by the above steps 101 to 106, on the one hand, can predict the scaling factor of the image block through the scaling factor generation network, that is, generate a scaling factor corresponding to the image block, the calculation method is flexible, and the image block can be better optimized and corrected, thereby improving the encoding performance; on the other hand, the scaling factor can be predicted only based on the image block for which the scaling factor generated by the neural network is enabled, without traversing all the image blocks, thereby solving the delay problem; on the other hand, the scaling factor generation network is used at both the encoding and decoding ends to predict the scaling factor of the image block, and there is no need to encode and transmit the scaling factor, which saves the number of bits required for encoding the scaling factor and improves the coding efficiency.

[0203] Furthermore, in an embodiment of the present application, preliminary training and testing of the encoding and decoding method proposed in the present application are implemented based on the VTM11.0-nnvc reference platform.

[0204] First, a conventional NNLF was implemented, consisting of a multi-layer convolutional neural network based on residual learning. NNSF was then further trained (a network with scaling factors, such as the network structure shown in Figure 9, was used to generate the network, trained for the Y component of non-I frames), and its performance was tested. Under the general test condition Random Access configuration, a specified general sequence was tested, with the comparison anchor being VTM11.0-nnvc. The test results for some sequences are shown in Tables 4 and 5. Table 4 shows the RA performance of NNLF compared to the anchor, and Table 5 shows the RA performance of NNSF compared to the anchor.

[0205] Table 4

[0206]

[0207] Table 5

[0208]

[0209]

[0210] Among them, comparing the Y component performance data in Table 4 and Table 5, it can be seen that by introducing the learning-based scaling factor (i.e., the scaling factor of the image block predicted by the scaling factor generation network), it is possible to improve the encoding performance based on the neural network loop filter to a certain extent, and the performance can be further improved through subsequent optimization training.

[0211] It is understood that in the embodiments of the present application, the above methods are all based on the use of the technology for all components in the video, such as R, G, B in RGB format video or Y, U, V (Y, Cb, Cr) in YUV format. The above methods can also be used for only a certain component, such as only the Y component in YUV format. The above methods can also be used for each component separately, that is, each component can be controlled on or off separately.

[0212] An embodiment of the present application provides a decoding method, wherein a decoder decodes a bitstream and determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the decoder can use the scaling factor generation network on the decoding side to predict a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image. This enhances the flexibility and applicability of the scaling factor, improves coding performance, and also improves coding efficiency.

[0213] Based on the above embodiment, another embodiment of the present application proposes an encoding method, which is applied to an encoder. FIG12 is a schematic diagram of an implementation flow of the encoding method. As shown in FIG12 , the encoding method performed by the encoder may include the following steps:

[0214] Step 201: Determine first identification information.

[0215] In an embodiment of the present application, the encoder may first determine the first identification information.

[0216] It should be noted that, in the implementation of the present application, the first identification information can be used to determine whether to use a scaling factor generated by a neural network for the current color component of the current frame of the current sequence. If a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, a scaling factor corresponding to at least one image block in the current color component of the current frame of the current sequence can be generated, and subsequent encoding processing can be performed using the scaling factor corresponding to the image block.

[0217] In an embodiment of the present application, the first identification information can be understood as a flag indicating whether a scaling factor generated by a neural network is used for at least one image block in a current color component of a current frame of a current sequence. Alternatively, the first identification information can be understood as a flag indicating whether a scaling factor corresponding to the image block is used to determine a reconstructed block of the image block for at least one image block in a current color component of a current frame of the current sequence. Specifically, the encoder can determine a variable as the first identification information, so that the determination of the first identification information can be achieved by taking the value of the variable.

[0218] It should be noted that, in the present application, the value of the first identification information may be the first value or the second value. Different values ​​of the first identification information may lead to different methods for determining the reconstructed block of at least one image block in the current color component of the current frame of the current sequence.

[0219] It can be understood that in the embodiments of the present application, based on the value of the first identification information, when encoding the current color component, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0220] For example, in the present application, if the value of the first identification information is a first value, then the scaling factor generated by the neural network may be selected for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may be used to complete the encoding process of the current color component. Meanwhile, if the value of the first identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may not be used in the decoding process of the current color component.

[0221] For example, in the present application, if the value of the first identification information is a first value, then the scaling factor generated by the neural network may be selected for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may be used to complete the decoding process of the current color component. Alternatively, if the value of the first identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current color component of the current frame of the current sequence, and the scaling factor determined based on the neural network may not be used in the encoding process of the current color component.

[0222] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0223] Furthermore, in an embodiment of the present application, the flag picture_nnsf_enable_flag[compIdx] may be used to represent the first identification information, where compIdx may be used to represent the color component of the current frame of the current sequence.

[0224] For example, in this application, if the value of picture_nnsf_enable_flag[compIdx] is 1, it can indicate that the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence, and then the scaling factor determined based on the neural network is used to complete the encoding processing of the current color component.

[0225] For example, in the present application, if the value of picture_nnsf_enable_flag[compIdx] is 0, it may indicate that the scaling factor generated by the neural network is not used for the current color component of the current frame of the current sequence, and thus the scaling factor determined based on the neural network is not used in the encoding processing flow of the current color component.

[0226] It should be noted that, in the embodiment of the present application, the first identification information may be an image-level flag, wherein the first identification information may be written into the bitstream and transmitted to the decoding end.

[0227] Furthermore, in an embodiment of the present application, the method for encoding by the encoder may further include the following steps:

[0228] Step 205: Determine the third identification information.

[0229] Step 206: If the third identification information indicates that the scaling factor generated by the neural network is used for the current sequence, the determination process of the first identification information is executed.

[0230] In an embodiment of the present application, the third identification information may be determined first, and then whether to further determine the first identification information is determined based on the third identification information. If the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, then the first identification information may be determined.

[0231] It should be noted that, in the implementation of this application, the third identification information can be used to determine whether to use the scaling factor generated by the neural network for the current sequence. That is, if the scaling factor generated by the neural network is used for the current sequence, a scaling factor corresponding to an image block can be generated for at least one image frame in the current sequence, and the scaling factor corresponding to the image block can then be used for subsequent encoding processing.

[0232] In embodiments of the present application, the third identification information can be understood as a flag indicating whether to use a scaling factor generated by a neural network for at least one image frame in the current sequence, or as a flag indicating whether to use a scaling factor corresponding to an image block to determine a reconstructed block of the image block for at least one image frame in the current sequence. Specifically, a variable can be determined as the third identification information, so that the third identification information can be determined by the value of the variable.

[0233] It should be noted that, in the present application, the value of the third identification information may be the first value or the second value. Different values ​​of the third identification information may lead to different ways of determining the reconstructed block of the image block of at least one image frame in the current sequence.

[0234] It can be understood that in the embodiments of the present application, based on the value of the third identification information, when encoding at least one image frame in the current sequence, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0235] For example, in this application, if the value of the third identification information is a first value, then the scaling factor generated by the neural network may be selected for the current sequence, and the scaling factor determined based on the neural network may be used to complete the encoding process of the current sequence. Meanwhile, if the value of the third identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current sequence, and the scaling factor determined based on the neural network may not be used in the encoding process of the current sequence.

[0236] For example, in this application, if the value of the third identification information is the first value, then the scaling factor generated by the neural network may be selected for the current sequence, and the scaling factor determined based on the neural network may be used to complete the encoding process of the current sequence. Alternatively, if the value of the third identification information is the second value, then the scaling factor generated by the neural network may not be selected for the current sequence, and the scaling factor determined based on the neural network may not be used in the encoding process of the current sequence.

[0237] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0238] Furthermore, in an embodiment of the present application, the flag nnsf_enable_flag may be used to represent the third identification information.

[0239] For example, in this application, if the value of nnsf_enable_flag is 1, it can indicate that the scaling factor generated by the neural network is used for the current sequence, and then the scaling factor determined based on the neural network is used to complete the encoding process of the current sequence.

[0240] For example, in this application, if the value of nnsf_enable_flag is 0, it may indicate that the scaling factor generated by the neural network is not used for the current sequence, and thus the scaling factor determined based on the neural network is not used in the encoding process of the current sequence.

[0241] It should be noted that, in the embodiment of the present application, the third identification information may be a sequence-level flag, wherein the third identification information may be pre-set and does not need to be written into the code stream.

[0242] It can be seen that in the present application, the determination of the first identification information can depend on the determination of the third identification information. Of course, the determination of the first identification information can also depend on some other flags, that is, picture_nnsf_enable_flag[compIdx] may depend on some other flags, which is not specifically limited in the embodiments of the present application.

[0243] It should be noted that in the embodiments of the present application, the third identification information can be a sequence-level flag, or even a higher-level flag, such as video usability information (VUI), supplemental enhancement information (SEI), etc., which is not specifically limited in the embodiments of the present application.

[0244] Step 202: If the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, second identification information is determined.

[0245] In an embodiment of the present application, after determining the first identification information, if the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, then the second identification information can be further determined.

[0246] It can be understood that in an embodiment of the present application, if the first identification information is used to determine that the scaling factor generated by the neural network is used to generate the scaling factor of the image block for the current color component of the current frame of the current sequence, then the second identification information needs to be further determined to determine which image blocks in all the image blocks in the current color component use the scaling factor generated by the neural network based on the second identification information.

[0247] That is, in an embodiment of the present application, the second identification information can be used to determine whether to use a scaling factor generated by a neural network for the current block of the current color component of the current frame of the current sequence. If a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, a scaling factor corresponding to the current block can be generated for the current block, and the scaling factor corresponding to the current block can be used for subsequent encoding processing.

[0248] In an embodiment of the present application, the second identification information can be understood as a flag indicating whether a scaling factor generated by a neural network is used for a current block in a current color component of a current frame in a current sequence. Alternatively, the second identification information can be understood as a flag indicating whether a scaling factor corresponding to the current block is used to determine a reconstructed block of the current block in a current color component of a current frame in a current sequence. Specifically, the encoder can determine a variable as the second identification information, so that the second identification information can be determined by taking the value of the variable.

[0249] It should be noted that, in this application, the value of the second identification information may be the first value or the second value. Different values ​​of the second identification information may lead to different ways of determining the reconstructed block of the current block in the current color component of the current frame of the current sequence.

[0250] It can be understood that in the embodiments of the present application, based on the value of the second identification information, when encoding the current block of the current color component, you can choose to use the scaling factor generated by the neural network, or you can choose not to use the scaling factor generated by the neural network.

[0251] For example, in the present application, if the value of the second identification information is a first value, then the scaling factor generated by the neural network may be selected for the current block of the current color component, and the scaling factor determined based on the neural network may be used to complete the encoding process of the current block. Meanwhile, if the value of the second identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current block of the current color component, and the scaling factor determined based on the neural network may not be used in the encoding process of the current block.

[0252] For example, in the present application, if the value of the second identification information is a first value, then the scaling factor generated by the neural network may be selected for the current block of the current color component, and the scaling factor determined based on the neural network may be used to complete the encoding process of the current block. Alternatively, if the value of the second identification information is a second value, then the scaling factor generated by the neural network may be selected not to be used for the current block of the current color component, and the scaling factor determined based on the neural network may not be used in the encoding process of the current block.

[0253] It should be noted that, in the embodiments of the present application, the first value may be set to 1 and the second value may be set to 0; the first value may also be set to true and the second value may be set to false. This application does not impose any limitation.

[0254] Furthermore, in an embodiment of the present application, taking the coding tree unit CTU as an example, the flag patch_nnsf_enable_flag[compIdx][ctuIdx] can be used to represent the second identification information, where compIdx can be used to represent the color component of the current frame of the current sequence, and ctuIdx can be used to represent the image block of the current color component of the current frame of the current sequence.

[0255] For example, in the present application, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 1, it may indicate that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, and then the scaling factor determined based on the neural network is used to complete the encoding process of the current block.

[0256] For example, in the present application, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 0, it may indicate that the scaling factor generated by the neural network is not used for the current block of the current color component of the current frame of the current sequence, and thus the scaling factor determined based on the neural network is not used in the encoding process of the current block.

[0257] It should be noted that, in the embodiment of the present application, the second identification information may be a block-level flag, wherein the second identification information may be written into the code stream and transmitted to the decoding end.

[0258] Furthermore, in an embodiment of the present application, the second identification information may be a block-level flag, and the block-level flag may include a maximum coding unit (LCU) level flag, a coding tree unit (CTU) level flag, a coding unit (CU) level flag, or other block-level flags. In the embodiment of the present application, the coding tree unit CTU is mainly used as an example for illustration, but the specific form of the block-level flag is not limited.

[0259] It should be noted that, in an embodiment of the present application, after determining the third identification information, if the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, then the scaling factor generated by the neural network can be used for each image block of the current color component of the current frame of the current sequence, thereby obtaining the cost parameter of each image block.

[0260] It can be understood that in an embodiment of the present application, based on the third identification information, if it is determined that the current sequence uses a scaling factor generated by a neural network, then all image blocks of the current color component of the current frame of the current sequence can be traversed in sequence, and each image block can be processed using the scaling factor generated by the neural network to obtain the corresponding processed image block. Then, the cost parameters corresponding to each image block can be calculated based on the image block before processing and the image block after processing, respectively, where the cost parameters may include but are not limited to rate-distortion cost.

[0261] For example, in the present application, if the cost parameter is less than 0, that is, the cost of the image block after processing is less than the cost of the image block before processing, then it can be considered that the scaling factor generated by the neural network can be used for the image block; correspondingly, if the cost parameter is not less than 0, that is, the cost of the image block after processing is not less than the cost of the image block before processing, then it can be considered that there is no need to use the scaling factor generated by the neural network for the image block.

[0262] Furthermore, in an embodiment of the present application, if a cost parameter of an image block indicates that a scaling factor generated by a neural network is used for an image block, the value of the second identification information of the image block is set to a first value; if a cost parameter of an image block indicates that a scaling factor generated by a neural network is not used for an image block, the value of the second identification information of the image block is set to a second value.

[0263] That is, in an embodiment of the present application, if processing using the scaling factor generated by the neural network can reduce the cost of the image block, then the second identification information corresponding to the image block can be set to the first value, that is, the second identification information is set to indicate that the scaling factor generated by the neural network is used for the image block; if processing using the scaling factor generated by the neural network does not reduce the cost of the image block, then the second identification information corresponding to the image block can be set to the second value, that is, the second identification information is set to indicate that the scaling factor generated by the neural network is not used for the image block.

[0264] Furthermore, in an embodiment of the present application, if the second identification information of each image block of the current color component of the current frame of the current sequence is not all the second value, the value of the first identification information of the current color component is set to the first value; if the second identification information of each image block of the current color component of the current frame of the current sequence is all the second value, the value of the first identification information of the current color component is set to the second value.

[0265] That is, in an embodiment of the present application, for the current frame of the current sequence, if all the second identification information corresponding to all the image blocks of the current color component are not all the second values, then it is necessary to use the scaling factor generated by the neural network for the current color component of the current frame of the current sequence, and therefore the value of the first identification information of the current color component can be set to the first value, that is, the first identification information is set to indicate that the scaling factor generated by the neural network is used for the current color component; if all the second identification information corresponding to all the image blocks of the current color component are all the second values, then it is not necessary to use the scaling factor generated by the neural network for the current color component of the current frame of the current sequence, and therefore the value of the first identification information of the current color component can be set to the second value, that is, the first identification information is set to indicate that the scaling factor generated by the neural network is not used for the current color component.

[0266] Step 203: If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame, the scaling factor generation network is used to determine the scaling factor corresponding to the current block of the current color component.

[0267] In an embodiment of the present application, after determining the second identification information, if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component of the current frame of the current sequence, then the scaling factor generation network can be further used to generate a scaling factor corresponding to the current block.

[0268] It should be noted that in the embodiments of the present application, the scaling factor generation network can be used to predict the scaling factor of an image block. That is, the scaling factor generation network can be a network that predicts the scaling factor of an image block. Therefore, the scaling factor generation network can be used to perform a scaling factor determination process for an image block, and the generated scaling factor is corresponding to the image block, that is, a block-level scaling factor is generated.

[0269] It can be understood that in an embodiment of the present application, when using a scaling factor generation network to determine the scaling factor corresponding to the current block of the current color component, the filtered image block and the reconstructed image block corresponding to the current block can be optionally input into the scaling factor generation network, so that the scaling factor of the current block can be output.

[0270] It should be noted that in the embodiments of the present application, the input of the scaling factor generation network may include a filtered image block generated by a neural network-based loop filter (NNLF), and may also include a reconstructed image block determined by sample adaptive offset (SAO). Accordingly, the output of the scaling factor generation network may be the scaling factor corresponding to the current block.

[0271] Furthermore, in an embodiment of the present application, when using a scaling factor generation network to determine a scaling factor corresponding to a current block of a current color component, the filtered image block, the reconstructed image block, and feature information corresponding to the current block may be input into the scaling factor generation network to output a scaling factor for the current block. The feature information corresponding to the current block may include one or more of block partition information, prediction information, and quantization parameter information.

[0272] That is, in the present application, when determining the scaling factor of the current block based on the scaling factor generation network, in addition to processing the filtered image block and the reconstructed image block of the current block, more feature information of the current block can also be input. For example, the block partition information of the current block, and / or the prediction information of the current block, and / or the QP information of the current block can be used as one of the inputs of the scaling factor generation network.

[0273] Furthermore, in embodiments of the present application, the network structure of the scaling factor generation network may include at least one of a convolutional layer, a nonlinear activation function PReLU, a Concat layer, a maximum pooling layer, and a fully connected FullConnect layer. That is, the network structure of the scaling factor generation network may include a convolutional layer, and / or a nonlinear activation function PReLU, and / or a Concat layer, and / or a maximum pooling layer, and / or a fully connected FullConnect layer.

[0274] Accordingly, in the embodiments of the present application, the data format of the scaling factor generated by the scaling factor generation network may also vary based on different network structures. For example, the data format of the scaling factor may be a two-dimensional matrix or a one-dimensional variable, which is not specifically limited in this application.

[0275] Exemplarily, in an embodiment of the present application, the network structure of the scaling factor generation network may include a convolution layer, a nonlinear activation function PReLU layer, and a Concat layer. Accordingly, the data form of the scaling factor of the image block generated by the scaling factor generation network may be a two-dimensional matrix, that is, the scaling factor has the same dimension as the image block. For example, as shown in FIG8 , the structure of the scaling factor generation network consists of multiple convolution layers. For the input filtered image block and the reconstructed image block, each needs to undergo a layer of convolution and a nonlinear activation function PReLU; then the feature maps are spliced ​​together through the Concat layer; finally, after K layers of convolution and K-1 layers of nonlinear activation function PReLU, the scaling factor is predicted. Wherein, K is an integer greater than 1.

[0276] Exemplarily, in an embodiment of the present application, the network structure of the scaling factor generation network may further include a convolution layer, a nonlinear activation function PReLU layer, a Concat layer, a maximum pooling MaxPooling layer, and a fully connected FullConnect layer. Accordingly, the data form of the scaling factor of the image block generated by the scaling factor generation network may be a one-dimensional variable. For example, as shown in FIG9 , the structure of the scaling factor generation network is composed of multiple convolution layers. For the input filtered image block and the reconstructed image block, each needs to undergo a layer of convolution and a nonlinear activation function PReLU; then the feature maps are spliced ​​together through the Concat layer; then K layers of convolution and K layers of maximum pooling MaxPooling are passed; finally, after passing through L layers of fully connected layers FullConnect and L-1 layers of nonlinear activation function PreLU, the scaling factor is predicted. Wherein, L is an integer greater than 1.

[0277] It should be noted that in the embodiments of the present application, the neural network loop filter generally performs training iterations on the network by calculating the minimum mean square error loss between the network output image and the original image. Since the training goal of the filter network is clear, that is, the output of the trained network is closer to the original image (i.e., the label), using the minimum mean square error as the loss function is more effective.

[0278] Furthermore, in an embodiment of the present application, a scaling factor set can be used to train a scaling factor generation network, so that data labels can be determined from the scaling factor set; then, the loss of the scaling factor generation network can be determined based on the output data and data labels of the scaling factor generation network; and finally, the network parameters of the scaling factor generation network can be updated based on the loss.

[0279] For the neural network scaling factor structure shown in FIG9 above, as proposed in the embodiments of this application, the training goal is to obtain a scaling factor that is closer to the theoretically optimal scaling factor (direct goal), thereby making the final corrected image closer to the original image (indirect goal). The original image (indirect goal) can be directly obtained from the dataset; however, the theoretically optimal scaling factor (direct goal) is actually unknown before training.

[0280] In other words, in order to better train for the direct goal, the embodiments of the present application propose an adaptive label update training method. By finding the optimal scaling factor during training, the labels are adaptively updated to achieve better training results.

[0281] During training, for the current input reconstructed image block rec(x i ,y i ), first load a certain neural network filter and get the filtered image block cnn(x i ,y i Then, referring to the correction formula of the scaling factor, the scaling factors SF_i within a specific range [SFmin, SFmax] (i.e., the scaling factor set) are traversed (for example, the traditional scaling factor calculation method can be referred to, and the range is set to [16, 320]), and the output reconstructed block output (x i ,y i ) and the original image org(x i ,y i ) to find the optimal scaling factor SF_best, which is used as the label SF_label (data label) of the current training data. The specific training pseudo code is as follows:

[0282] minloss=maxVALUE

[0283] SF_best=-1

[0284] for SF_i in range(SFmin,SFmax):

[0285] output=rec+SF_i*(cnn-rec)

[0286] loss = MSE (output, org)

[0287] if(loss <minloss):

[0288] minloss=loss

[0289] SF_best=SF_i

[0290] SF_label=SF_best

[0291] After finding SF_label, the label of the current training data is updated adaptively.

[0292] Finally, by calculating the mean absolute error MAE between the output of the scaling factor network [NNSF]_b and the label SF_label, the network loss is calculated, and training iterations and parameter updates are performed.

[0293] Step 204: Determine a reconstructed block of the current block according to the scaling factor.

[0294] In an embodiment of the present application, after the scaling factor corresponding to the current block of the current color component is determined using the scaling factor generation network, a reconstructed block of the current block can be further determined according to the scaling factor.

[0295] It should be noted that, in the embodiment of the present application, when determining the reconstructed block of the current block according to the scaling factor, the reconstructed block of the current block can be determined according to the scaling factor, the filtered image block and the reconstructed image block.

[0296] It is understandable that, in the embodiments of the present application, the scaling factor can be used to correct the residual between the filtered image block and the reconstructed image block, thereby completing the reconstruction of the current block and obtaining the corresponding reconstructed block.

[0297] Furthermore, in an embodiment of the present application, when determining the reconstructed block of the current block based on the scaling factor, the filtered image block, and the reconstructed image block, the residual value corresponding to the current block can be first determined based on the filtered image block and the reconstructed image block; then, the residual value is corrected based on the scaling factor to obtain a corrected result corresponding to the current block; finally, the reconstructed block of the current block can be determined based on the corrected result and the reconstructed image block.

[0298] For example, in the embodiment of the present application, for the current block (x i ,y i ), if the scaling factor generated by the scaling factor generation network is a two-dimensional matrix NNSF a (x i ,y i), then use the filter image block cnn(x i ,y i ) and reconstructed image block rec(x i ,y i ) Determine the reconstructed block output(x) of the current block i ,y i ) can be expressed as formula (9).

[0299] For example, in the embodiment of the present application, for the current block (x i ,y i ), if the scaling factor predicted by the scaling factor generating network is a one-dimensional variable NNSF b , then use the filter image block cnn(x i ,y i ) and reconstructed image block rec(x i ,y i ) Determine the reconstructed block output(x) of the current block i ,y i ) can be expressed as formula (10).

[0300] It is understood that in the embodiments of the present application, the above methods are all based on the example of using the technology uniformly for all components in the video, such as the R, G, and B components of an RGB format video, or the Y, U, and V (Y, Cb, Cr) components of a YUV format. The above methods can also be used for only one component, such as only the Y component of a YUV format. The above methods can also be used for each component separately, that is, each component can be controlled separately. This application does not impose any specific restrictions.

[0301] Furthermore, in an embodiment of the present application, as shown in Figure 10, a scaling factor module for scaling factor processing is added to the inloop filter (Inloop Filter) module of the hybrid coding framework, namely, a neural network-based scaling factor NNSF module. The scaling factor module can be used to predict the scaling factor of the image block using a neural network (scaling factor generation network), and complete the reconstruction of the image block based on the scaling factor to obtain the corresponding reconstructed block.

[0302] It should be noted that, in the embodiment of the present application, the scaling factor module NNSF may be located after the neural network-based loop filter NNLF in the loop filtering module and before the adaptive correction filter ALF.

[0303] It can be understood that, in the embodiment of the present application, the use of the scaling factor module NNSF does not depend on the switches of the deblocking filter DBF, the sample adaptive offset SAO, the neural network-based loop filter NNLF, and the adaptive correction filter ALF.

[0304] That is to say, whether the learning-based scaling factor module NNSF is turned on can be determined by the encoder, so in the embodiment of the present application, the scaling factor module can be set to switch control at the frame level and the block (such as CTU) level. Among them, at the encoding end, it is possible to judge whether to turn on the scaling factor module NNSF by comparing the image rate distortion cost before and after the scaling factor processing, that is, to judge whether to use the scaling factor generated by the neural network, and the switching status of the scaling factor module NNSF will be encoded into the bitstream for the decoder to read. At the decoder end, when it is parsed that the scaling factor module is turned on, that is, when it is determined to use the scaling factor generated by the neural network, the scaling factor of the image block is used to optimize the image block; and when it is parsed that the scaling factor module is closed, that is, when it is determined not to use the scaling factor generated by the neural network, no related processing of the scaling factor is performed.

[0305] Furthermore, in an embodiment of the present application, the scaling factor obtained based on the scaling factor generation network prediction can be a learning-based, image block scaling factor, which is used to correct the residual between the filtered image block output by the NNLF and the reconstructed image block output by the SAO, so that the reconstructed block of the current block finally obtained is closer to the original image, so as to obtain better coding performance and solve the problems existing in the traditional scaling factor to a certain extent.

[0306] It is understood that in the embodiments of the present application, the network structure of the scaling factor generation network can be a relatively simple convolutional neural network, with the input being the filtered image block output by the NNLF and the reconstructed image block output by the SAO, and the output being the scaling factor of the current block (e.g., CTU unit). That is, the learning-based scaling factor of the current block is predicted by the neural network.

[0307] Furthermore, in an embodiment of the present application, a technical solution of using a neural network (scaling factor generation network) to predict the scaling factor of an image block is implemented at both the codec end and can act in a loop filter module.

[0308] Exemplarily, in an embodiment of the present application, when the encoder processes the loop filter module, it processes in accordance with the prescribed filter order. When entering the scaling factor module, it first determines whether the neural network scaling factor module can be used in the current sequence based on the sequence-level flag nnsf_enable_flag (third identification information), that is, it determines whether to use the scaling factor generated by the neural network for at least one image frame in the current sequence based on the third identification information. If nnsf_enable_flag is "1", it is determined that the scaling factor module is processed for the current sequence, that is, it is determined that the third identification information indicates that the scaling factor generated by the neural network is used for at least one image frame in the current sequence; if nnsf_enable_flag is "0", it is determined that the scaling factor module is not used for the current sequence, that is, it is determined that the third identification information indicates that all image frames in the current sequence do not use the scaling factor generated by the neural network, and then ALF processing can continue.

[0309] Accordingly, in an embodiment of the present application, after determining that the third identification information indicates the use of a neural network-generated scaling factor for at least one image frame of the current sequence, for the current frame of the current sequence, you can choose to traverse all encoded image blocks of all color components, try neural network scaling factor processing based on each encoded block, and compare it with the image block before processing, calculate the cost (i.e., cost parameter), and then set the third identification information of the image block under a color component according to the cost parameter.

[0310] For example, in this application, it is assumed that the cost parameter of the image block is the rate-distortion cost D, which is expressed as the following formula:

[0311] D=D nnsf -D before (10)

[0312] Among them, D nnsf is the distortion after processing, D before is the distortion before processing.

[0313] If the cost after processing is less than that before processing, that is, D<0, then the second identification information of the image block can be set to the first value (such as 1), that is, the value of the block-level flag patch_nnsf_enable_flag[compIdx][ctuIdx] is set to "1"; if the cost after processing is greater than or equal to that before processing, that is, D>=0, then the second identification information of the image block can be set to the second value (such as 0), that is, the value of the block-level flag patch_nnsf_enable_flag[compIdx][ctuIdx] is set to "0". If all image blocks of the current color component in the current frame have been traversed, the next color component in the current frame can be loaded for processing.

[0314] Furthermore, in an embodiment of the present application, for the current color component of the current frame of the current sequence, if the second identification information of all image blocks is the second value, that is, the values ​​of patch_nnsf_enable_flag[compIdx][ctuIdx] of all graphic blocks are all "0", then the value of the first identification information of the current color component of the current frame of the current sequence can be set to the second value (such as 0), that is, the value of the image-level flag picture_nnsf_enable_flag[compIdx] is set to "0". If the second identification information of all image blocks is not all the second value, that is, the values ​​of patch_nnsf_enable_flag[compIdx][ctuIdx] of all graphic blocks are not "0", then the value of the first identification information of the current color component of the current frame of the current sequence can be set to the first value (such as 1), that is, the value of the image-level flag picture_nnsf_enable_flag[compIdx] is set to "1". If all color components of the current frame have been traversed, the next frame in the current sequence can be loaded for processing.

[0315] It can be understood that in the implementation of the present application, after traversing all image blocks in the current color component of the current frame of the current sequence, the next color component of the current frame of the current sequence can be loaded, and then all color components of the current frame of the current sequence can be traversed according to the above method. After completing the decision of the neural network scaling factor module of the current frame, the next frame of the current sequence can be loaded for processing.

[0316] Furthermore, in an embodiment of the present application, as shown in FIG11 , the loop filtering module mainly includes tools such as a deblocking filter DBF, a sample adaptive compensation SAO, a neural network-based loop filter NNLF, a neural network-based scaling factor NNSF module, and an adaptive correction filter ALF.

[0317] Among them, the network structure of the NNSF module (scaling factor generation network) can be composed of a relatively simple convolutional neural network, which can output the scaling factor of the image block in the form of a two-dimensional matrix as shown in Figure 8 above; or it can output the scaling factor of the image block in the form of a one-dimensional variable as shown in Figure 9 above.

[0318] Exemplarily, in an embodiment of the present application, for an image block, the reconstructed image block corresponding to the image block output by the SAO can be input into the NNLF and NNSF modules respectively, and then the NNLF outputs the filtered image block corresponding to the image block. The filtered image block is input into the NNSF module, and then the scaling factor corresponding to the image block and generated by the neural network can be output, so that the scaling factor can be used to correct the residual between the filtered image block and the reconstructed image block, and finally the reconstructed block corresponding to the image block is output, and the reconstructed block continues to be input into the ALF to perform the subsequent filtering processing.

[0319] It can be understood that in an embodiment of the present application, after determining the first identification information and the second identification information, the encoder can write the first identification information and the second identification information into the code stream and transmit it to the decoding end, so that the decoder can determine the first identification information and the second identification information by decoding the code stream.

[0320] Furthermore, in an embodiment of the present application, the sequence header definition syntax is shown in Table 1. In the syntax, the permission flag of the scaling factor generated by the neural network is nnsf_enable_flag, that is, the third identification information used to indicate whether to use the scaling factor generated by the neural network for the current sequence is nnsf_enable_flag.

[0321] For example, if the value of nnsf_enable_flag is 1, it can be determined that the scaling factor module is processed for the current sequence, that is, it can be determined that the scaling factor generated by the neural network can be used for the current sequence; if the value of nnsf_enable_flag is 0, it can be determined that the scaling factor generated by the neural network is not used for the current sequence.

[0322] Furthermore, in an embodiment of the present application, the picture header definition syntax is shown in Table 2. In terms of syntax, when the value of the neural network generated scaling factor enable flag nnsf_enable_flag is 1, that is, when the third identification information indicates that the scaling factor generated by the neural network is used for the current sequence, the image-level neural network scaling factor enable flag can be determined to be picture_nnsf_enable_flag[compIdx], that is, the first identification information used to indicate whether the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence is picture_nnsf_enable_flag[compIdx].

[0323] For example, if the value of picture_nnsf_enable_flag[compIdx] is 1, it can be determined that the current color component of the current frame of the current sequence is processed by the scaling factor module, that is, it is determined that the scaling factor generated by the neural network can be used for the current color component of the current frame of the current sequence; if the value of picture_nnsf_enable_flag[compIdx] is 0, it can be determined that the current color component of the current frame of the current sequence does not use the scaling factor generated by the neural network.

[0324] Furthermore, in an embodiment of the present application, the slice definition syntax is as shown in Table 3. In terms of syntax, when the value of the neural network generated scaling factor enable flag nnsf_enable_flag is 1, that is, when the third identification information indicates that the scaling factor generated by the neural network is used for the current sequence, and at the same time, the image-level neural network scaling factor enable flag picture_nnsf_enable_flag[compIdx] is determined to be 1, that is, when the first identification information indicates that the scaling factor generated by the neural network is used for the current color component of the current frame of the current sequence, the block-level neural network scaling factor enable flag can be determined to be patch_nnsf_enable_flag[compIdx][ctuIdx], that is, the second identification information used to indicate whether the scaling factor generated by the neural network is used for the current block of the current color component of the current frame of the current sequence is patch_nnsf_enable_flag[compIdx][ctuIdx].

[0325] For example, if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 1, it can be determined that the current block of the current color component of the current frame of the current sequence is processed by the scaling factor module, that is, it is determined that the scaling factor generated by the neural network can be used for the current block of the current color component of the current frame of the current sequence; if the value of patch_nnsf_enable_flag[compIdx][ctuIdx] is 0, it can be determined that the current block of the current color component of the current frame of the current sequence does not use the scaling factor generated by the neural network.

[0326] In summary, in the embodiment of the present application, the encoding method proposed by steps 201 to 206 above, on the one hand, can predict the scaling factor of the image block through the scaling factor generation network, that is, generate a scaling factor corresponding to the image block, the calculation method is flexible, and the image block can be better optimized and corrected, thereby improving the encoding performance; on the other hand, the scaling factor can be predicted only based on the image block for which the scaling factor generated by the neural network is enabled, without traversing all the image blocks, thereby solving the delay problem; on the other hand, the scaling factor generation network is used at both the encoding and decoding ends to predict the scaling factor of the image block, and there is no need to encode and transmit the scaling factor, which saves the number of bits required for encoding the scaling factor and improves the encoding efficiency.

[0327] An embodiment of the present application provides an encoding method, wherein an encoder determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the encoder can use the scaling factor generation network on the encoding side to predict a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image, thereby enhancing the flexibility and applicability of the scaling factor, improving encoding performance, and improving encoding efficiency.

[0328] Based on the above embodiment, in yet another embodiment of the present application, FIG13 is a schematic diagram of the structure of a decoder proposed in the embodiment of the present application. As shown in FIG13 , the decoder 300 proposed in the embodiment of the present application may include a decoding part 301, a first determining part 302,

[0329] The decoding part 301 is configured to decode the code stream;

[0330] The first determination portion 302 is configured to determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, then determine second identification information; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, then use a scaling factor generation network to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of an image block; and determine a reconstructed block of the current block based on the scaling factor.

[0331] Figure 14 is a second schematic diagram of the composition structure of the decoder proposed in an embodiment of the present application. As shown in Figure 14, the decoder 300 proposed in an embodiment of the present application may also include a first processor 303, a first memory 304 storing executable instructions of the first processor 303, a first communication interface 305, and a first bus 306 for connecting the first processor 303, the first memory 304 and the first communication interface 305.

[0332] Furthermore, in an embodiment of the present application, the first processor 303 is configured to decode a code stream and determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor.

[0333] FIG15 is a schematic diagram of the first structure of the encoder proposed in an embodiment of the present application. As shown in FIG15 , the encoder 400 proposed in an embodiment of the present application may include a second determining part 401,

[0334] The second determination part 401 is configured to determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, then determine second identification information; if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, then use a scaling factor generation network to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and determine a reconstructed block of the current block based on the scaling factor.

[0335] Figure 16 is a second schematic diagram of the composition structure of the encoder proposed in an embodiment of the present application. As shown in Figure 16, the encoder 400 proposed in an embodiment of the present application may further include a second processor 402, a second memory 403 storing executable instructions of the second processor 402, a second communication interface 405, and a second bus 406 for connecting the second processor 402, the second memory 403 and the second communication interface 405.

[0336] Furthermore, in an embodiment of the present application, the second processor 402 is configured to determine first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor.

[0337] In addition, the functional modules in this embodiment may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated units may be implemented in the form of hardware or software functional modules.

[0338] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0339] An embodiment of the present application provides an encoder and a decoder, wherein the decoder decodes a bitstream and determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor. The encoder determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, a scaling factor generation network is used to determine the scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the codec can use the scaling factor generation network on the codec side to predict and obtain a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image. This not only enhances the flexibility and applicability of the scaling factor, but also improves the coding performance and coding efficiency.

[0340] The embodiments of the present application provide a computer-readable storage medium and a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, the method described in the above embodiments is implemented.

[0341] Specifically, the program instructions corresponding to a decoding method in this embodiment can be stored on a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to a decoding method in the storage medium are read or executed by an electronic device, the following steps are included:

[0342] Decoding the code stream to determine the first identification information;

[0343] If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information;

[0344] If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block;

[0345] A reconstructed block of the current block is determined according to the scaling factor.

[0346] Specifically, the program instructions corresponding to an encoding method in this embodiment can be stored on a storage medium such as an optical disk, a hard disk, or a USB flash drive. When the program instructions corresponding to an encoding method in the storage medium are read or executed by an electronic device, the following steps are included:

[0347] determining first identification information;

[0348] If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information;

[0349] If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block;

[0350] A reconstructed block of the current block is determined according to the scaling factor.

[0351] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage and optical storage, etc.) containing computer-usable program code.

[0352] The present application is described with reference to the implementation flow diagram and / or block diagram of the method, device (system) and computer program product according to the embodiment of the present application. It should be understood that each flow and / or box in the flow diagram and / or block diagram and the combination of the flow and / or box in the flow diagram and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processing machine or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for realizing the function specified in one flow diagram or multiple flows and / or one block or multiple blocks of the block diagram.

[0353] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a product including an instruction device that implements the functions specified in implementing one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0354] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0355] The above description is merely a preferred embodiment of the present application and is not intended to limit the scope of protection of the present application. Industrial Applicability

[0356] Embodiments of the present application provide a coding and decoding method, a decoder, an encoder, and a storage medium. The decoder decodes a bitstream and determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. The encoder determines first identification information; if the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for a current block of the current color component, a scaling factor generation network is used to determine a scaling factor corresponding to the current block; wherein the scaling factor generation network is used to predict a scaling factor of an image block; and a reconstructed block of the current block is determined based on the scaling factor. That is, in an embodiment of the present application, the codec can use the scaling factor generation network on the codec side to predict and obtain a learning-based scaling factor for the image block, which is used to correct the residual corresponding to the image block to obtain a reconstructed block that is closer to the original image. This not only enhances the flexibility and applicability of the scaling factor, but also improves the coding performance and coding efficiency.

Claims

1. A decoding method, applied to a decoder, comprising: Decoding the code stream to determine the first identification information; If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information; If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block; A reconstructed block of the current block is determined according to the scaling factor.

2. The method according to claim 1, wherein The using a scaling factor generation network to determine the scaling factor corresponding to the current block includes: The filtered image block and the reconstructed image block corresponding to the current block are input into the scaling factor generation network, and the scaling factor is output.

3. The method according to claim 2, wherein: The network structure of the scaling factor generation network includes at least one of a convolutional layer, a nonlinear activation function PReLU layer, a Concat layer, a maximum pooling MaxPooling layer, and a fully connected FullConnect layer.

4. The method according to claim 3, wherein: The data form of the scaling factor is a two-dimensional matrix or a one-dimensional variable.

5. The method according to claim 4, wherein The determining a reconstructed block of the current block according to the scaling factor includes: A reconstructed block of the current block is determined according to the scaling factor, the filtered image block, and the reconstructed image block.

6. The method according to claim 5, wherein: The determining, according to the scaling factor, the filtered image block, and the reconstructed image block, of a reconstructed block of the current block includes: Determine a residual value corresponding to the current block according to the filtered image block and the reconstructed image block; Correcting the residual value according to the scaling factor to obtain a correction result corresponding to the current block; A reconstructed block of the current block is determined according to the correction result and the reconstructed image block.

7. The method according to claim 4, wherein: The using a scaling factor generation network to determine the scaling factor corresponding to the current block includes: Inputting the filtered image block, reconstructed image block and feature information corresponding to the current block into the scaling factor generation network, and outputting the scaling factor; wherein the feature information includes at least one of block partition information, prediction information and quantization parameter information.

8. The method according to claim 1, wherein The method further comprises: determining third identification information; If the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, the determination process of the first identification information is executed.

9. The method according to claim 8, wherein If the value of the third identification information is the first value, applying the scaling factor generated by the neural network to the current sequence; and / or If the value of the third identification information is the second value, the scaling factor generated by the neural network is not used for the current sequence.

10. The method according to claim 8, wherein If the value of the first identification information is the first value, applying a scaling factor generated by a neural network to the current color component; and / or If the value of the first identification information is the second value, the scaling factor generated by the neural network is not used for the current color component.

11. The method according to claim 8, wherein If the value of the second identification information is the first value, applying a scaling factor generated by a neural network to the current block; and / or If the value of the second identification information is the second value, the scaling factor generated by the neural network is not used for the current block.

12. The method according to any one of claims 9 to 11, wherein: described The first value is 1, and the second value is 0; or, The first value is set to true, and the second value is set to false.

13. The method according to claim 4, wherein: The method further comprises: training the scaling factor generation network using the scaling factor set to determine data labels from the scaling factor set; determining a loss of the scaling factor generation network based on output data of the scaling factor generation network and the data label; Network parameters of the scaling factor generation network are updated based on the loss.

14. A coding method, applied to an encoder, comprising: determining first identification information; If the first identification information indicates that a scaling factor generated by a neural network is used for a current color component of a current frame of a current sequence, determining second identification information; If the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, determining a scaling factor corresponding to the current block using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block; A reconstructed block of the current block is determined according to the scaling factor.

15. The method according to claim 14, wherein The using a scaling factor generation network to determine the scaling factor corresponding to the current block includes: The filtered image block and the reconstructed image block corresponding to the current block are input into the scaling factor generation network, and the scaling factor is output.

16. The method according to claim 15, wherein The network structure of the scaling factor generation network includes at least one of a convolutional layer, a nonlinear activation function PReLU, a Concat layer, a maximum pooling MaxPooling layer, and a fully connected FullConnect layer.

17. The method according to claim 3, wherein: The data form of the scaling factor is a two-dimensional matrix or a one-dimensional variable.

18. The method according to claim 17, wherein The determining a reconstructed block of the current block according to the scaling factor includes: A reconstructed block of the current block is determined according to the scaling factor, the filtered image block, and the reconstructed image block.

19. The method according to claim 18, wherein The determining, according to the scaling factor, the filtered image block, and the reconstructed image block, of a reconstructed block of the current block includes: Determine a residual value corresponding to the current block according to the filtered image block and the reconstructed image block; Correcting the residual value according to the scaling factor to obtain a correction result corresponding to the current block; A reconstructed block of the current block is determined according to the correction result and the reconstructed image block.

20. The method according to claim 17, wherein The using a scaling factor generation network to determine the scaling factor corresponding to the current block includes: Inputting the filtered image block, reconstructed image block and feature information corresponding to the current block into the scaling factor generation network, and outputting the scaling factor; wherein the feature information includes at least one of block partition information, prediction information and quantization parameter information.

21. The method according to claim 14, wherein The method further comprises: determining third identification information; If the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, the determination process of the first identification information is executed.

22. The method according to claim 21, wherein If the value of the third identification information is the first value, applying the scaling factor generated by the neural network to the current sequence; and / or If the value of the third identification information is the second value, the scaling factor generated by the neural network is not used for the current sequence.

23. The method according to claim 22, wherein The method further comprises: If the third identification information indicates that a scaling factor generated by a neural network is used for the current sequence, then the scaling factor generated by the neural network is used for each image block of the current color component of the current frame of the current sequence to obtain a cost parameter for each image block; If a cost parameter of an image block indicates a scaling factor generated by a neural network for the image block, setting a value of the second identification information of the image block to a first value; If the cost parameter of an image block indicates that the scaling factor generated by the neural network is not used for the image block, the value of the second identification information of the image block is set to a second value.

24. The method according to claim 23, wherein The method further comprises: If the second identification information of each image block of the current color component is not all the second value, setting the value of the first identification information of the current color component to the first value; If the second identification information of each image block of the current color component is the second value, the value of the first identification information of the current color component is set to the second value.

25. The method according to any one of claims 22 to 24, wherein: described The first value is 1, and the second value is 0; or, The first value is set to true, and the second value is set to false.

26. The method according to claim 17, wherein The method further comprises: training the scaling factor generation network using the scaling factor set to determine data labels from the scaling factor set; determining a loss of the scaling factor generation network based on output data of the scaling factor generation network and the data label; Network parameters of the scaling factor generation network are updated based on the loss.

27. The method of claim 14, further comprising: The first identification information and the second identification information are written into a code stream.

28. A decoder, comprising: Decoding part, first determination part, The decoding part is configured to decode the code stream; The first determining part is configured to determine first identification information; If the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, the second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, the scaling factor corresponding to the current block is determined using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor.

29. A decoder, the encoder comprising a first processor and a first memory storing instructions executable by the first processor, wherein when the instructions are executed, the first processor implements the method according to any one of claims 1 to 13.

30. An encoder, comprising: The second part is to determine The second determining part is configured to determine the first identification information; If the first identification information indicates that a scaling factor generated by a neural network is used for the current color component of the current frame of the current sequence, the second identification information is determined; if the second identification information indicates that a scaling factor generated by a neural network is used for the current block of the current color component, the scaling factor corresponding to the current block is determined using a scaling factor generation network; wherein the scaling factor generation network is used to predict the scaling factor of the image block; and a reconstructed block of the current block is determined based on the scaling factor.

31. An encoder, wherein the decoder comprises a second processor and a second memory storing instructions executable by the second processor, and when the instructions are executed, the second processor implements the method according to any one of claims 14 to 27.

32. A computer storage medium, wherein: The computer storage medium stores a computer program, which implements the method according to any one of claims 1 to 13 when executed by a first processor, or implements the method according to any one of claims 14 to 27 when executed by a second processor.

Images