Coding method, decoding method, coder, decoder, and storage medium

Abstract

Disclosed in the present application are a coding method, a decoding method, a coder, a decoder, and a storage medium. The coding method comprises: at a coding and decoding end, determining that a current block uses or allows the use of a loop filtering technique based on a neural network; acquiring a first reconstructed image of the current block, wherein the first reconstructed image comprises a reconstructed sample of the current block; acquiring a first edge image of the current block, wherein the first edge image comprises edge information of the reconstructed sample; and inputting the first reconstructed image and the first edge image into a loop filtering model based on the neural network, so as to obtain a filtered image of the current block. Therefore, inputting into a loop filtering model an edge image as added edge information can provide sample-level edge information for a network, can adjust the learning of filtering intensity at a sample level and can improve a filtering effect, thus improving the quality and coding and decoding performance of a reconstructed image.

Description

Coding and decoding method, encoder, decoder and storage medium Technical Field

[0001] The embodiments of the present application relate to the field of video coding and decoding technology, and specifically to a coding and decoding method, an encoder, a decoder, and a storage medium. Background Art

[0002] As demand for video display quality increases, new video applications such as high-definition and ultra-high-definition video have emerged. The Joint Video Exploration Team (JVET) of the International Organization for Standardization (ISO / IEC) and ITU-T has developed the next-generation video coding standard H.266 / Versatile Video Coding (VVC).

[0003] Neural networks have been introduced into the field of video coding and decoding. Leveraging the powerful learning capabilities of neural networks, coding and decoding tools based on neural networks often have very high coding and decoding efficiency. For example, there are intra-frame prediction methods based on neural networks, inter-frame prediction methods based on neural networks, and loop filtering methods based on neural networks. Among them, the coding performance of loop filtering methods based on neural networks is the most outstanding. However, the current loop filtering methods based on neural networks do not fully utilize the advantages of neural network models. In some coding and decoding scenarios, the loop filtering methods based on neural networks do not significantly improve the filtering effect, and may even reduce the filtering efficiency. Therefore, the loop filtering methods based on neural networks need to be optimized.

[0004] Summary of the Invention

[0005] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which input edge images as new edge information into a loop filtering model to improve the filtering effect.

[0006] The technical solution of the embodiment of the present application can be implemented as follows:

[0007] In a first aspect, an embodiment of the present application provides a decoding method, applied to a decoder, the method comprising:

[0008] Decoding the code stream to determine first indication information;

[0009] Determining, according to the first indication information, that a neural network-based loop filtering technology is used for the current block;

[0010] Obtaining a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0011] Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0012] The first reconstructed image and the first edge image are input into a loop filtering model based on a neural network to obtain a filtered image of the current block.

[0013] In a second aspect, an embodiment of the present application provides an encoding method, applied to an encoder, the method comprising:

[0014] Determining whether the current block allows the use of a neural network-based loop filtering technique;

[0015] Obtaining a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0016] Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0017] Inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block;

[0018] Performing cost calculation based on the original image of the current block and the filtered image to determine a first generation value;

[0019] Determining whether a current block uses a neural network-based loop filtering technology according to the first generation value, and setting first indication information of the current block;

[0020] The first indication information is encoded, and the obtained encoded bits are written into a bitstream.

[0021] In a third aspect, an embodiment of the present application provides an encoder, comprising a first determination unit, a first filtering unit, a decision unit, and an encoding unit; wherein:

[0022] The first determining unit is configured to determine whether the current block allows the use of a neural network-based loop filtering technology;

[0023] The first determining unit is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block;

[0024] The first determining unit is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0025] The first filtering unit is configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block;

[0026] The decision unit is configured to perform cost calculation based on the original image of the current block and the filtered image to determine a first generation value;

[0027] The decision unit is further configured to determine whether a neural network-based loop filtering technology is used for the current block according to the first generation value, and set first indication information for the current block;

[0028] The encoding unit is configured to encode the first indication information and write the obtained encoded bits into a bit stream.

[0029] In a fourth aspect, an embodiment of the present application provides an encoder, comprising a first memory and a first processor; wherein,

[0030] a first memory for storing a computer program capable of running on the first processor;

[0031] The first processor is configured to execute the method according to the second aspect when running a computer program.

[0032] In a fifth aspect, an embodiment of the present application provides a decoder, comprising a decoding unit, a second determining unit, and a second filtering unit; wherein:

[0033] The decoding unit is configured to decode the code stream and determine the first indication information;

[0034] The second determining unit is configured to determine, according to the first indication information, whether to use a neural network-based loop filtering technology for the current block;

[0035] The second determining unit is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block;

[0036] The second determining unit is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0037] The second filtering unit is configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block.

[0038] In a sixth aspect, an embodiment of the present application provides a decoder, the decoder comprising a second memory and a second processor; wherein,

[0039] a second memory for storing a computer program capable of running on the second processor;

[0040] The second processor is configured to execute the method according to the first aspect when running a computer program.

[0041] In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a code stream generated by the encoding method described in the second aspect above.

[0042] In an eighth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed, implements the method as described in the first aspect or the method as described in the second aspect.

[0043] The embodiments of the present application provide a coding and decoding method, an encoder, a decoder, and a storage medium, which input edge images as new edge information into a loop filtering model, can provide sample-level edge information for the network, adjust the learning of the filtering strength at the sample level, improve the filtering effect, and thus improve the reconstructed image quality and coding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0045] FIG1 is a block diagram of an encoder according to an embodiment of the present application;

[0046] FIG2 is a schematic block diagram of a decoder provided in an embodiment of the present application;

[0047] FIG3 is a schematic diagram of a network architecture of a coding and decoding system provided in an embodiment of the present application;

[0048] FIG4 is a schematic diagram of a flow chart of an encoding method provided in an embodiment of the present application;

[0049] FIG5 is a schematic diagram of a reconstructed image provided by an embodiment of the present application;

[0050] FIG6 is a schematic diagram of an edge image provided in an embodiment of the present application;

[0051] FIG7 is a schematic diagram of the structure of a loop filtering model in an embodiment of the present application;

[0052] FIG8 is a schematic diagram of the composition structure of another loop filtering model provided in an embodiment of the present application;

[0053] FIG9 is a schematic diagram of a flow chart of a decoding method provided in an embodiment of the present application;

[0054] FIG10 is a schematic diagram of the structure of an encoder provided in an embodiment of the present application;

[0055] FIG11 is a schematic diagram of a specific hardware structure of an encoder provided in an embodiment of the present application;

[0056] FIG12 is a schematic diagram of the structure of a decoder provided in an embodiment of the present application;

[0057] FIG13 is a schematic diagram of a specific hardware structure of a decoder provided in an embodiment of the present application;

[0058] FIG14 is a schematic diagram of the composition structure of a coding and decoding system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following will describe the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0060] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0061] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0062] It should also be pointed out that the terms "first\second\third" involved in the embodiments of the present application are only used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0063] In video images, a coding block (CB) is generally represented by a first image component, a second image component, and a third image component. These three image components are a luminance component, a blue chrominance component, and a red chrominance component. Specifically, the luminance component is typically represented by the symbol Y, the blue chrominance component is typically represented by the symbols Cb or U, and the red chrominance component is typically represented by the symbols Cr or V. Thus, video images can be represented in either the YCbCr or YUV format.

[0064] Before further explaining the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained first. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations:

[0065] Moving Picture Experts Group (MPEG)

[0066] International Standardization Organization (ISO)

[0067] International Electrotechnical Commission (IEC)

[0068] Joint Video Experts Team (JVET)

[0069] Alliance for Open Media (AOM)

[0070] Next-generation video coding standard H.266 / Versatile Video Coding (VVC)

[0071] VVC Test Model (VTM)

[0072] Audio Video Standard (AVS)

[0073] AVS High-Performance Model (HPM)

[0074] Transform coefficients

[0075] Quantization Parameter (QP)

[0076] Neural-network based video coding (NNVC)

[0077] Sample Adaptive Offset (SAO)

[0078] Deblocking filter (DBF)

[0079] Digital video compression technology is understood to primarily compress massive amounts of digital video data for easier transmission and storage. With the surge in Internet video usage and increasing demand for higher-quality video, while existing digital video compression standards can conserve significant video data, there remains a need for improved digital video compression technologies to reduce bandwidth and traffic pressures associated with digital video transmission.

[0080] Currently, common video codec standards (such as H.266 / VVC) all use a block-based hybrid coding framework. Each frame in the video is divided into square maximum coding units (LCUs) of the same size (such as 128x128, 64x64, etc.). Each LCU can be divided into rectangular coding units (CUs) according to rules. Coding units may also be divided into prediction units (PUs) and transform units (TUs). The hybrid coding framework includes modules such as prediction, transform, quantization, entropy coding, and in-loop filtering. The prediction module includes intra-frame prediction and inter-frame prediction. Inter-frame prediction includes motion estimation and motion compensation. Since there is a strong correlation between adjacent pixels in a video frame, intra-frame prediction is used in video coding technology to eliminate spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, the inter-frame prediction method is used in video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0081] The basic process of a video codec is as follows. On the encoder side, a frame is divided into blocks. Intra-frame prediction or inter-frame prediction is used on the current block to generate a prediction block for the current block. The prediction block is subtracted from the original image block to obtain a residual block. This residual block is transformed and quantized to obtain a quantization coefficient matrix. This quantization coefficient matrix is ​​then entropy-encoded and output to the bitstream. On the decoder side, intra-frame prediction or inter-frame prediction is used on the current block to generate a prediction block for the current block. The bitstream is then parsed to obtain a quantization coefficient matrix. This quantization coefficient matrix is ​​then inversely quantized and inversely transformed to obtain a residual block. The prediction block and residual block are added together to form a reconstructed block. The reconstructed block is then loop-filtered on the reconstructed image based on an image or block basis to obtain the decoded image. The encoder side also performs similar operations to the decoder side to obtain the decoded image. The decoded image can serve as a reference frame for inter-frame prediction for subsequent frames. The block division information determined by the encoder, as well as information about the prediction, transform, quantization, entropy coding, loop filtering, and other modes or parameters, are output to the bitstream if necessary. The decoding end determines the same block division information as the encoding end by parsing and analyzing the existing information, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering, thereby ensuring that the decoded image obtained by the encoding end is the same as the decoded image obtained by the decoding end. The decoded image obtained by the encoding end is also usually called a reconstructed image. During prediction, the current block can be divided into prediction units, and during transformation, the current block can be divided into transformation units. The division of prediction units and transformation units can be different. The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of this framework or process may be optimized. The current block can be the current coding unit (CU) or the current prediction unit (PU), etc.

[0082] JVET, the international video coding standards organization, has established a group to research coding models that surpass H.266 / VVC. This model, or platform testing software, has been named ECM. Building on VTM10.0, ECM incorporates newer and more efficient compression algorithms, currently surpassing VVC's coding performance by approximately 13%. ECM not only increases the coding unit size for specific resolutions but also integrates numerous intra- and inter-frame prediction techniques.

[0083] Neural networks have been introduced into the video coding and decoding field. Leveraging their powerful learning capabilities, neural network-based codec tools often offer highly efficient coding and decoding. For example, neural network-based intra-frame prediction methods, neural network-based inter-frame prediction methods, and neural network-based loop filtering methods offer the most outstanding coding performance.

[0084] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0085] Referring to Figure 1, which shows a schematic block diagram of the composition of an encoder provided in an embodiment of the present application. As shown in Figure 1, the encoder (specifically, a "video encoder") 100 may include a transform and quantization unit 101, an intra-frame estimation unit 102, an intra-frame prediction unit 103, a motion compensation unit 104, a motion estimation unit 105, an inverse transform and inverse quantization unit 106, a filter control analysis unit 107, a filtering unit 108, an encoding unit 109, and a decoded image cache unit 110, etc., wherein the filtering unit 108 can implement deblocking filtering and sample adaptive offset (SAO) filtering, and the encoding unit 109 can implement header information encoding and context-based adaptive binary arithmetic coding (CABAC).For the input original video signal, a video coding block can be obtained by dividing the coding tree unit (CTU). Then, the residual pixel information obtained after intra-frame or inter-frame prediction is transformed by the transformation and quantization unit 101, including transforming the residual information from the pixel domain to the transform domain and quantizing the obtained transform coefficients to further reduce the bit rate; the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to perform intra-frame prediction on the video coding block. Specifically, the intra-frame estimation unit 102 and the intra-frame prediction unit 103 are used to determine the intra-frame prediction mode to be used to encode the video coding block; the motion compensation unit 104 and the motion estimation unit 105 are used to perform inter-frame prediction coding on the received video coding block relative to one or more blocks in one or more reference frames to provide temporal prediction information; the motion estimation performed by the motion estimation unit 105 is the process of generating a motion vector, which can estimate the motion of the video coding block, and the motion compensation unit 104 performs motion compensation based on the motion vector determined by the motion estimation unit 105; after determining the intra-frame After the prediction mode is determined, the intra-frame prediction unit 103 is further configured to provide the selected intra-frame prediction data to the encoding unit 109, and the motion estimation unit 105 also sends the calculated motion vector data to the encoding unit 109. In addition, the inverse transform and inverse quantization unit 106 is used to reconstruct the video coding block and reconstruct the residual block in the pixel domain. The reconstructed residual block passes through the filter control analysis unit 107 and the filtering unit 108. The filtering unit 108 implements neural network-based loop filtering, deblocking filtering, SAO filtering, etc., and then adds the reconstructed residual block to a predictive block in the frame of the decoded image buffer unit 110 to generate a reconstructed video coding block. The encoding unit 109 is used to encode various coding parameters and quantized transform coefficients. In the CABAC-based coding algorithm, the context content can be based on adjacent coding blocks and can be used to encode information indicating the determined intra-frame prediction mode, and output the code stream of the video signal. The decoded image buffer unit 110 is used to store the reconstructed video coding block for prediction reference. As the video image encoding proceeds, new reconstructed video encoding blocks are continuously generated, and these reconstructed video encoding blocks are stored in the decoded image buffer unit 110 .

[0086] Referring to Figure 2, a schematic block diagram of a decoder provided in an embodiment of the present application is shown. As shown in Figure 2, the decoder (specifically, a "video decoder") 200 includes a decoding unit 201, an inverse transform and inverse quantization unit 202, an intra-frame prediction unit 203, a motion compensation unit 204, a filtering unit 205, and a decoded image cache unit 206. The decoding unit 201 can implement header information decoding and CABAC decoding, and the filtering unit 205 can implement neural network-based loop filtering, deblocking filtering, SAO filtering, and the like. After the input video signal is encoded as shown in FIG1 , a code stream of the video signal is output; the code stream is input to the decoder 200 and first passes through the decoding unit 201 to obtain the decoded transform coefficients; the transform coefficients are processed by the inverse transform and inverse quantization unit 202 to generate a residual block in the pixel domain; the intra-frame prediction unit 203 can be used to generate prediction data for the current video decoding block based on the determined intra-frame prediction mode and the data of the previously decoded block from the current frame or picture; the motion compensation unit 204 determines the prediction information for the video decoding block by analyzing the motion vector and other associated syntax elements, and uses The prediction information is used to generate a predictive block for the video decoding block being decoded; a decoded video block is formed by summing the residual block from the inverse transform and inverse quantization unit 202 with the corresponding predictive block generated by the intra-frame prediction unit 203 or the motion compensation unit 204; the decoded video signal passes through the filtering unit 205 to remove blocking artifacts, thereby improving video quality; the decoded video block is then stored in the decoded image buffer unit 206, which stores reference images used for subsequent intra-frame prediction or motion compensation, and is also used for outputting the video signal, thereby obtaining the restored original video signal.

[0087] Furthermore, an embodiment of the present application also provides a network architecture of a coding and decoding system including an encoder and a decoder, wherein FIG3 shows a schematic diagram of a network architecture of a coding and decoding system provided by an embodiment of the present application. As shown in FIG3 , the network architecture includes one or more electronic devices 13 to 1N and a communication network 01, wherein the electronic devices 13 to 1N can perform video interaction through the communication network 01. During implementation, the electronic device can be various types of devices with video coding and decoding functions. For example, the electronic device can include a smart phone, a tablet computer, a personal computer, a personal digital assistant, a navigator, a digital phone, a video phone, a television, a sensing device, a server, etc., which are not specifically limited here. In addition, the decoder or encoder described in the embodiment of the present application can be the above-mentioned electronic device.

[0088] It should be noted that the method of the embodiment of the present application is mainly applied to the filtering unit 108 shown in Figure 1 and the filtering unit 205 shown in Figure 2. In other words, the embodiment of the present application can be applied to both the encoder and the decoder, or even to both the encoder and the decoder at the same time, but the embodiment of the present application is not specifically limited thereto.

[0089] It should also be noted that, when applied to the filtering unit 108 part, the "current block" specifically refers to the coding block currently to be intra-frame predicted; when applied to the filtering unit 205 part, the "current block" specifically refers to the decoding block currently to be intra-frame predicted.

[0090] To facilitate understanding of the technical solutions of the embodiments of the present application, the technical solutions of the present application are described in detail below through specific embodiments. The above related technologies can be combined arbitrarily with the technical solutions of the embodiments of the present application as optional solutions, and all of them fall within the scope of protection of the embodiments of the present application. The embodiments of the present application include at least part of the following contents.

[0091] An embodiment of the present application provides a coding and decoding method, which can be specifically a loop filtering method based on a neural network. The edge image is input into the loop filtering model as new edge information, which can provide the network with sample-level edge information, adjust the learning of the filtering strength at the sample level, improve the filtering effect, and thus improve the reconstructed image quality and coding and decoding performance.

[0092] In one embodiment of the present application, referring to FIG4 , a schematic flow chart of an encoding method provided by an embodiment of the present application is shown. As shown in FIG4 , the method may include:

[0093] S401: Determine whether the current block allows the use of a neural network-based loop filtering technology;

[0094] It should be noted that the current block can be any loop filtering processing unit based on a neural network. In some embodiments, the current block can be the current coding tree unit. In other embodiments, the current block can also be the current image block obtained by other division methods. In some other embodiments, the current block can also be the current frame.

[0095] It should also be noted that the current block includes at least a first color component and a second color component. For the first color component of the current block, the block can be simply referred to as a first color component block; and when the first color component is a luminance component, the first color component block can also be referred to as a luminance block. Similarly, for the second color component of the current block, the block can be simply referred to as a second color component block; and when the second color component is a chrominance component, the second color component block can also be referred to as a chrominance block.

[0096] In some embodiments, the method includes: determining whether the current block is allowed to use a technology based on a neural network loop filter according to a high-level syntax element identifier of the current block.

[0097] S402: Obtain a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0098] It should be noted that the reconstructed samples may be luminance component reconstructed samples, in which case the first reconstructed image is a luminance component reconstructed image. The reconstructed samples may also be chrominance reconstructed samples, in which case the first reconstructed image is a chrominance component reconstructed image.

[0099] In some embodiments, obtaining the first reconstructed image of the current block includes: obtaining an initial reconstructed image of the frame containing the current image; and obtaining the first reconstructed image of the current block from the initial reconstructed image. In other words, the encoder performs loop filtering on the reconstructed image based on an image or block, and inputs the initial reconstructed image into a loop filtering model.

[0100] In other embodiments, the initial reconstructed image is deblocked and the first reconstructed image of the current block is obtained from the filtered image; or the initial reconstructed image is sample-adaptively compensated and the first reconstructed image of the current block is obtained from the compensated image; or the initial reconstructed image is deblocked and sample-adaptively compensated and the first reconstructed image of the current block is obtained from the compensated image. In other words, the encoder performs loop filtering on the reconstructed image based on an image or a block, and the reconstructed image input to the loop filtering model can also be other reconstructed images, such as images after deblocking filtering or sample-adaptive compensation.

[0101] S403: Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0102] It should be noted that the first reconstructed image is the same size as the first edge image, and the first edge image contains edge information for each reconstructed sample in the first reconstructed image. The reconstructed samples can be luminance component reconstructed samples, and the first edge image is the luminance component edge image. The reconstructed samples can also be chrominance reconstructed samples, and the first edge image is the chrominance component edge image. Figure 5 is a schematic diagram of a reconstructed image provided in an embodiment of the present application, and Figure 6 is a schematic diagram of an edge image provided in an embodiment of the present application.

[0103] In some embodiments, the edge information may be binary data used to represent the edge of an object in an image. In other embodiments, the edge information may also include edge intensity, which represents not only the edge of an object in an image but also the intensity of the object edge.

[0104] In some embodiments, the method further includes: obtaining a second reconstructed image of the frame where the current block is located; performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; obtaining the first edge image of the current block, including: obtaining the first edge image of the current block from the second edge image.

[0105] In some embodiments, obtaining a second reconstructed image of the frame where the current block is located includes: obtaining an initial reconstructed image of the frame where the current block is located; using the initial reconstructed image as the second reconstructed image; or performing deblocking filtering on the initial reconstructed image to obtain the second reconstructed image; or performing sample adaptive compensation on the initial reconstructed image to obtain the second reconstructed image; or performing deblocking filtering and sample adaptive compensation on the initial reconstructed image to obtain the second reconstructed image. In other words, when the encoder performs loop filtering on the reconstructed image based on an image or a block, an edge image containing edge information of all reconstructed samples of the current block is determined by edge detection on the reconstructed image, wherein the reconstructed image can be an initial reconstructed image composed of reconstructed blocks, or can be other images such as an image after deblocking filtering or an image after sample adaptive compensation.

[0106] In some embodiments, the method further includes: traversing a list of edge detection operators to obtain a candidate edge detection operator as the first edge detection operator; if the minimum cost value is the first generation value, determining to use a neural network-based loop filtering technique; determining the index value of the optimal edge detection operator corresponding to the minimum cost value; and encoding the index value of the optimal edge detection operator. In other words, the encoder can use multiple existing edge detection operators for edge detection and make encoding decisions, determine the optimal edge detection operator based on the minimum cost value, encode the index value of the optimal edge detection operator, write the encoding information into the code stream and transmit it to the decoder; the decoder selects the optimal edge detection operator based on the parsed index value of the edge detection operator and performs edge detection on the reconstructed image.

[0107] In other embodiments, the first edge detection operator is a preset edge detection operator.

[0108] In the embodiment of the present application, the first edge detection operator may be one of the following: a Sobel operator, a Laplacian operator, a Roberts operator, and the like.

[0109] Exemplarily, the first edge detection operator is a Sobel operator;

[0110] Performing edge detection on the second reconstructed image based on the first edge detection operator to determine the second edge image includes: detecting gradient information of the second reconstructed image to determine transverse gradient information and longitudinal gradient information; and determining the second edge image based on the transverse gradient information and the longitudinal gradient information.

[0111] The detection principle of the Sobel operator is as follows:

[0112] As shown in the above formula, A represents the image to be extracted edge information, Gx is the horizontal gradient information, and Gy is the vertical gradient information. After extracting the gradient information of all sample points in image A, the sum of the horizontal and vertical gradient information is calculated:

[0113] In some embodiments, a second edge image is determined based on the transverse gradient information and the longitudinal gradient information, including: determining the absolute value of the gradient of the reconstructed sample based on the transverse gradient of the reconstructed sample in the transverse gradient information and the longitudinal gradient in the longitudinal gradient information; and using the absolute value of the gradient as the edge information of the reconstructed sample.

[0114] It should be noted that, in order to reduce computational complexity, G can be obtained by calculating the sum of the absolute values ​​of Gx and Gy instead of the above formula. The specific expression is as follows: G = |Gx| + |Gy|

[0115] In some embodiments, the method further includes: obtaining a second reconstructed image of the frame in which the current block is located; performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; performing denoising on the second edge image to obtain a denoised second edge image; and further, obtaining the first edge image of the current block from the denoised second edge image.

[0116] Exemplarily, the second edge image is denoised to obtain a denoised second edge image, including: if the edge strength of the first reconstructed sample is greater than or equal to a first threshold, retaining the edge strength of the first reconstructed sample; if the edge strength of the second reconstructed sample is less than the first threshold, setting the edge strength of the second reconstructed sample to zero.

[0117] It should be noted that, in the edge image obtained by calculation based on the edge detection operator, etc., extremely low gradient amplitude contributes little to enhancing high-frequency information or is noise, so the noise can be filtered out by the first threshold to obtain an updated edge image.

[0118] The first threshold may be a preset specific threshold or may be determined based on edge strength characteristics of the current edge image. In some embodiments, the method further comprises: determining an average value of edge strengths of all reconstructed samples in the second edge image; and determining the first threshold based on the average value.

[0119] In some embodiments, the first edge image and the second edge image are chroma component edge images, and the method further includes: determining a luminance component second edge image and a chroma component second edge image; and determining a final chroma component second edge image based on the luminance component second edge image and the chroma component second edge image.

[0120] It should be noted that since the chrominance component usually has a smaller number of samples, the edge contour of the chrominance component is not as clear as the edge contour of the luminance component. For the chrominance component edge image, the determination of the chrominance component edge image can be guided by the luminance component edge image, thereby improving the quality of the chrominance component edge image.

[0121] Exemplarily, the second edge image of the luminance component is downsampled to obtain a downsampled luminance component edge image, and the edge intensity mean of each sample of the downsampled luminance component edge image and the chrominance component edge image is calculated to obtain the final second edge image of the chrominance component.

[0122] In some other embodiments, the second edge image of the luminance component may be directly used as the second edge image of the chrominance component.

[0123] S404: Inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block;

[0124] It should be noted that video content often contains strong edge information, also known as high-frequency information. This is a crucial component of subjective vision, yet also difficult to gain. Inputting edge images as additional edge information into the loop filtering model provides the network with sample-level edge information, enabling it to adjust filter strength learning at the sample level and improve filtering effectiveness. If the first indication information determines that the current block does not utilize neural network-based loop filtering technology, then the first reconstructed image of the current block utilizes other filtering technologies.

[0125] In some embodiments, Figure 7 is a schematic diagram of the composition structure of a loop filtering model in an embodiment of the present application. The neural network-based loop filtering model 70 includes: an input unit 701, a feature extraction unit 702 and an output unit 703; the first reconstructed image Rec and the first edge image G are input into the input unit 701, and after passing through the feature extraction unit 702 and the output unit 703, the filtered image is output.

[0126] In other embodiments, the first reconstructed image Rec is input into the input unit 701, and then passes through the feature extraction unit 702. The feature extraction unit 702 inputs the feature information of the output first reconstructed image into the output unit 703; the first edge image G is input into the output unit 703. The output unit 703 processes the feature maps of the first edge image and the first reconstructed image, and outputs the filtered image.

[0127] In some other embodiments, the first edge image G and the first reconstructed image Rec are input to the input unit 701, and the first edge image G is input to the output unit 703. The output unit 703 processes the feature information of the first edge image and the first reconstructed image and outputs the filtered image.

[0128] It should be noted that the processing of edge information in the loop filtering model can be different from other input information. Since edge information reflects high-frequency information, it can be more effective at a later position in the loop filtering model. The edge information is directly input into the back-end output unit of the loop filtering model, and the original input edge information is used for image reconstruction to ensure the loop filtering auxiliary effect of the edge information.

[0129] In some embodiments, the method further includes: obtaining at least one of a predicted image, boundary strength information, slice type information, and quantization information of the current block; and simultaneously inputting at least one of the predicted image, boundary strength information, slice type information, and quantization information into a neural network-based loop filtering model to obtain a filtered image of the current block.

[0130] FIG8 is a schematic diagram of the composition structure of another loop filtering model provided by an embodiment of the present application. As shown in FIG8 , the input part mainly includes reconstructed samples rec, predicted samples pred, boundary information G, boundary strength information BS, slice type information IPB, and quantization parameter QP. Reconstructed samples and predicted samples can provide good residual information. The boundary strength information here can be understood as the strength information detected based on the deblocking filter. The boundary strength can help the neural network-based loop filtering tool learn the ability of the deblocking filter. The mode information indicates that the coding block where the sample is located is intra-frame prediction I, unidirectional inter-frame prediction P, and bidirectional inter-frame prediction B. Finally, the quantization parameter part also includes the basic quantization value BaseQP or the frame-level quantization value SliceQP. The mode information and quantization information adjust the learning of the filter strength from the local block level and the global frame level respectively. Usually, the neural network-based loop filtering tool is a single network model, that is, a single model is used to process different color components and different frame types. Therefore, the input information of the single network model also includes the different color components of the above input part, such as the luminance component reconstruction sample recY and luminance component prediction sample predY in the YUV domain, as well as the chrominance component reconstruction sample recUV and chrominance component prediction sample predUV.

[0131] For the output part, based on the description of the above input part, the input includes different color components, and the output part also includes different color components, namely, the filtered samples filteredRecY of the brightness component and the filtered samples filterRecUV of the chrominance component.

[0132] In some embodiments, the method further includes: performing edge detection on a training image based on multiple edge detection operators during a model training phase to obtain multiple edge images; constructing a training sample set based on the training image, the multiple edge images, and the true image; and training a neural network-based loop filter model using the training sample set to obtain a trained loop filter model. It should be noted that during the model training phase, different types of operators can be used to calculate edge images, a training dataset can be constructed using the multiple edge images, and the model can be trained using the training dataset to improve the robustness of the model.

[0133] S405: performing cost calculation based on the original image and the filtered image of the current block to determine the first generation value;

[0134] In some embodiments, the method further includes: determining a residual scaling parameter of the current block; determining a sample residual of the current block based on the reconstructed samples in the first reconstructed image and the reconstructed samples in the filtered image; scaling the sample residual according to the residual scaling parameter to obtain a scaled residual of the current block; determining a target reconstructed image of the current block based on the first reconstructed image and the scaled residual; performing a cost calculation based on the original image and the target reconstructed image of the current block to determine a second generation value; determining whether the current block uses a residual scaling technology based on the second generation value, and setting a second indication information; encoding the second indication information, and writing the obtained coded bits into the bitstream.

[0135] In some embodiments, the second indication information includes at least one of the following: a frame-level syntax element identifier, used to indicate whether the frame where the current block is located uses residual scaling technology; a slice-level syntax element identifier, used to indicate whether the slice where the current block is located uses residual scaling technology; a coding tree unit-level syntax element identifier, used to indicate whether the coding tree unit where the current block is located uses residual scaling technology.

[0136] In some embodiments, determining the residual scaling parameters of the current block includes: calculating the residual scaling parameters based on the original samples, the reconstructed samples before filtering, and the reconstructed samples after filtering to determine the residual scaling parameters of the current block; the method also includes: determining the residual scaling technology used for the current block based on the second-generation value, and encoding the residual scaling parameters.

[0137] In some embodiments, determining the residual scaling parameters of the current block includes: traversing the residual scaling parameter list to determine the residual scaling parameters of the current block; the method also includes: determining the residual scaling technology used for the current block based on the second generation value, determining the index value of the optimal residual scaling parameter; encoding the index value of the optimal residual scaling parameter.

[0138] In some embodiments, the method further includes encoding third indication information. The third indication information may be an index value of a residual scaling parameter, denoted as scaleIdx. When scaleIdx is 0, the residual scaling parameter is determined to be encoded. When scaleIdx is not 0, the index value of the residual scaling parameter is determined based on scaleIdx.

[0139] Furthermore, the method also includes: classifying samples of the current block based on the first edge image to determine the sample type of each sample in the current block; wherein each sample type corresponds to a residual scaling parameter. In other words, the edge image not only serves as additional side information for the loop filtering model, providing the network with sample-level edge information to adjust the learning of the filter strength at the sample level and improve the filtering effect, but also serves as sample classification information to classify and scale sample parameters, improving the accuracy of the scaled residual, thereby improving the reconstructed image quality and coding efficiency.

[0140] S406: Determine whether the current block uses a neural network-based loop filtering technology according to the first generation value, and set first indication information of the current block;

[0141] S407: Encode the first indication information, and write the obtained coded bits into the bitstream.

[0142] It should be noted that the first indication information is used to indicate whether the current block uses a neural network-based loop filtering technique. The current block may be any neural network-based loop filtering processing unit. In some embodiments, the current block may be a current coding tree unit. In other embodiments, the current block may be a current image block obtained by other partitioning methods. In still other embodiments, the current block may be a current frame.

[0143] It should also be noted that the current block includes at least a first color component and a second color component. For the first color component of the current block, the block can be simply referred to as a first color component block; and when the first color component is a luminance component, the first color component block can also be referred to as a luminance block. Similarly, for the second color component of the current block, the block can be simply referred to as a second color component block; and when the second color component is a chrominance component, the second color component block can also be referred to as a chrominance block.

[0144] In some embodiments, the first indication information includes a first syntax element identifier and a second syntax element identifier; wherein the first syntax element identifier is used to indicate whether the image unit where the current block is located is allowed to use the neural network-based loop filtering technology, and the second syntax element identifier is used to indicate whether the current block uses the neural network-based loop filtering technology.

[0145] In some embodiments, decoding the bitstream and determining the first indication information includes: decoding the bitstream and determining the first syntax element identifier, recorded as sps_nnlf_enable_flag; if the first syntax element identifier indicates that the current image sequence allows the use of the technology based on the neural network loop filter, decoding the bitstream and determining the second syntax element identifier information; if the first syntax element identifier indicates that the current image sequence does not allow the use of the technology based on the neural network loop filter, determining that the current sequence does not use the technology based on the neural network loop filter, and the first reconstructed image of the current block uses other filtering technologies.

[0146] In some embodiments, when the current block is a coding tree unit, the second syntax element identifier includes: a frame-level second syntax element identifier, used to indicate whether the frame where the current block is located uses a neural network-based loop filtering technology; a coding tree unit-level second syntax element identifier, used to indicate whether the coding tree unit where the current block is located uses a neural network-based loop filtering technology.

[0147] It should be noted that the second syntax element identifier at the frame level can be expressed as sh_nnlf_flag, and the second syntax element identifier at the coding tree unit level can be expressed as ctb_nnlf_flag. If sh_nnlf_flag is true, the identifier ctb_nnlf_flag used by the neural network-based loop filtering of all coding tree units in the current frame is parsed according to sh_nnlf_flag; if ctb_nnlf_flag is true, the current coding tree unit uses the neural network-based loop filtering technology, otherwise it is not used.

[0148] If sh_nnlf_flag is false, the neural network-based loop filtering usage flag ctb_nnlf_flag of all coding tree units in the current frame is false, which means that all coding tree units in the current frame do not use this technology.

[0149] In some other embodiments, the current block is not a coding tree unit, and the second syntax element identifier may further include: a block-level second syntax element identifier, used to indicate whether the current block uses a neural network-based loop filtering technology.

[0150] Based on the above embodiments, the encoding method provided in the embodiments of the present application is further illustrated.

[0151] In this embodiment, at the encoding end, the encoder obtains a reconstructed image after prediction, transformation, quantization, inverse quantization, and inverse transformation, and begins loop filtering to improve image quality. The neural network-based loop filtering enable flag, sps_nnlf_enable_flag, is parsed. If the enable flag is true, the neural network-based loop filtering tool is enabled; otherwise, the neural network-based loop filtering tool is not enabled.

[0152] 1. If the flag bit of the neural network-based loop filter tool that allows the use of the filter is true, proceed to step 2; otherwise, skip step 2 and proceed directly to step 3;

[0153] 2. Obtain the reconstructed sample image of the input network model, use the Sobel operator to calculate the corresponding horizontal gradient Gx and vertical gradient Gy for each sample of the reconstructed sample image, and take the square root of the sum of the squares of Gx and Gy to obtain the gradient amplitude of each sample point, that is, an edge image of the same size as the reconstructed sample image; the magnitude of the gradient amplitude is related to the speed of change of the pixel value in the image. In edge detection, the gradient amplitude can provide intensity information of the edge position. It should be noted that the edge image is calculated based on the Sobel operator or Laplace operator, etc., in which extremely low gradient amplitudes contribute little to enhancing high-frequency information or are noise. Therefore, in some embodiments, a threshold can be pre-set to filter out noise and obtain an updated edge image. Exemplarily, the mean value Gavg of all gradient amplitude samples in the edge image is solved, and the edge image is updated. For all samples in the edge image whose gradient amplitude samples are greater than Gavg, their gradient amplitudes are retained; otherwise, they are set to zero.

[0154] Initialize the network model according to the preset parameters, obtain the reconstructed sample rec, predicted sample pred, boundary strength BS, mode information IPB, quantization information BaseQP and SliceQP, and edge image of the current coding tree unit area, and input this information into the network model for inference calculation;

[0155] After network model inference, the filtered reconstructed sample filteredRec is obtained;

[0156] If the current frame or current coding tree unit uses residual scaling, the residual scaling parameters are calculated based on the original image, the reconstructed image, and the filtered image. The encoder calculates the rate-distortion cost of the obtained parameters and the default parameters and selects the optimal parameters. The parameters are multiplied by the residual of the filtered reconstructed sample filteredRec and the pre-filtered reconstructed sample rec, and the scaled residual is added back to the pre-filtered reconstructed sample rec to obtain the output sample output. If the current frame or current coding tree unit does not use residual scaling, the filtered reconstructed sample filteredRec is the output sample output.

[0157] 3. The encoder continues to operate other loop filtering technologies.

[0158] 4. After executing all loop filtering tools, the final output image is obtained and the bitstream information is output.

[0159] By adopting the above technical solution, at the encoding end, the edge image is input into the loop filtering model as new edge information, which can provide the network with sample-level edge information, adjust the learning of the filtering strength at the sample level, improve the filtering effect, and thus improve the reconstructed image quality and coding performance.

[0160] In yet another embodiment of the present application, see Figure 9, which shows a flowchart of a decoding method provided by an embodiment of the present application. As shown in Figure 9, the method may include:

[0161] S901: Decode the code stream and determine first indication information;

[0162] It should be noted that the first indication information is used to indicate whether the current block uses a neural network-based loop filtering technique. The current block may be any neural network-based loop filtering processing unit. In some embodiments, the current block may be a current coding tree unit. In other embodiments, the current block may be a current image block obtained by other partitioning methods. In still other embodiments, the current block may be a current frame.

[0163] It should also be noted that the current block includes at least a first color component and a second color component. For the first color component of the current block, the block can be simply referred to as a first color component block; and when the first color component is a luminance component, the first color component block can also be referred to as a luminance block. Similarly, for the second color component of the current block, the block can be simply referred to as a second color component block; and when the second color component is a chrominance component, the second color component block can also be referred to as a chrominance block.

[0164] In some embodiments, the first indication information includes a first syntax element identifier and a second syntax element identifier; wherein the first syntax element identifier is used to indicate whether the image unit where the current block is located is allowed to use the neural network-based loop filtering technology, and the second syntax element identifier is used to indicate whether the current block uses the neural network-based loop filtering technology.

[0165] In some embodiments, decoding the bitstream and determining the first indication information includes: decoding the bitstream and determining the first syntax element identifier, recorded as sps_nnlf_enable_flag; if the first syntax element identifier indicates that the current image sequence allows the use of the technology based on the neural network loop filter, decoding the bitstream and determining the second syntax element identifier information; if the first syntax element identifier indicates that the current image sequence does not allow the use of the technology based on the neural network loop filter, determining that the current sequence does not use the technology based on the neural network loop filter, and the first reconstructed image of the current block uses other filtering technologies.

[0166] In some embodiments, when the current block is a coding tree unit, the second syntax element identifier includes: a frame-level second syntax element identifier, used to indicate whether the frame where the current block is located uses a neural network-based loop filtering technology; a coding tree unit-level second syntax element identifier, used to indicate whether the coding tree unit where the current block is located uses a neural network-based loop filtering technology.

[0167] It should be noted that the second syntax element identifier at the frame level can be expressed as sh_nnlf_flag, and the second syntax element identifier at the coding tree unit level can be expressed as ctb_nnlf_flag. If sh_nnlf_flag is true, the identifier ctb_nnlf_flag used by the neural network-based loop filtering of all coding tree units in the current frame is parsed according to sh_nnlf_flag; if ctb_nnlf_flag is true, the current coding tree unit uses the neural network-based loop filtering technology, otherwise it is not used.

[0168] If sh_nnlf_flag is false, the neural network-based loop filtering usage flag ctb_nnlf_flag of all coding tree units in the current frame is false, which means that all coding tree units in the current frame do not use this technology.

[0169] In some other embodiments, the current block is not a coding tree unit, and the second syntax element identifier may further include: a block-level second syntax element identifier, used to indicate whether the current block uses a neural network-based loop filtering technology.

[0170] S902: Determine, according to the first indication information, whether to use a neural network-based loop filtering technology for the current block;

[0171] It should be noted that if the first indication information determines that the current block uses a neural network-based loop filtering technique, the edge image is input into the loop filtering model as additional edge information. This provides the network with sample-level edge information, allowing for adjustment of filter strength learning at the sample level to improve filtering effectiveness. If the first indication information determines that the current block does not use a neural network-based loop filtering technique, then the first reconstructed image of the current block uses another filtering technique.

[0172] S903: Obtain a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0173] It should be noted that the reconstructed samples may be luminance component reconstructed samples, in which case the first reconstructed image is a luminance component reconstructed image. The reconstructed samples may also be chrominance reconstructed samples, in which case the first reconstructed image is a chrominance component reconstructed image.

[0174] In some embodiments, obtaining a first reconstructed image of the current block includes: obtaining an initial reconstructed image of the frame containing the current image; and obtaining the first reconstructed image of the current block from the initial reconstructed image. Specifically, the decoder parses the bitstream to obtain a quantization coefficient matrix, performs inverse quantization and inverse transformation on the quantization coefficient matrix to obtain a residual block, adds the prediction block and the residual block to obtain a reconstructed block, and the reconstructed blocks form a reconstructed image. The reconstructed image is subjected to image-based or block-based loop filtering to obtain a decoded image.

[0175] In other embodiments, the initial reconstructed image is deblocked and the first reconstructed image of the current block is obtained from the filtered image; alternatively, the initial reconstructed image is sample-adaptively compensated and the first reconstructed image of the current block is obtained from the compensated image; alternatively, the initial reconstructed image is deblocked and sample-adaptively compensated and the first reconstructed image of the current block is obtained from the compensated image. In other words, the decoder performs loop filtering on the reconstructed image based on an image or a block, and the reconstructed image input to the loop filtering model may also be other reconstructed images, such as images after deblocking filtering or sample-adaptive compensation.

[0176] S904: Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0177] It should be noted that the first reconstructed image is the same size as the first edge image, and the first edge image contains edge information for each reconstructed sample in the first reconstructed image. The reconstructed samples can be luminance component reconstructed samples, and the first edge image is the luminance component edge image. The reconstructed samples can also be chrominance reconstructed samples, and the first edge image is the chrominance component edge image. Figure 5 is a schematic diagram of a reconstructed image provided in an embodiment of the present application, and Figure 6 is a schematic diagram of an edge image provided in an embodiment of the present application.

[0178] In some embodiments, the edge information may be binary data used to represent the edge of an object in an image. In other embodiments, the edge information may also include edge intensity, which represents not only the edge of an object in an image but also the intensity of the object edge.

[0179] In some embodiments, the method further includes: obtaining a second reconstructed image of the frame where the current block is located; performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; obtaining the first edge image of the current block, including: obtaining the first edge image of the current block from the second edge image.

[0180] In some embodiments, obtaining a second reconstructed image of the frame in which the current block is located includes: obtaining an initial reconstructed image of the frame in which the current block is located; using the initial reconstructed image as the second reconstructed image; or performing deblocking filtering on the initial reconstructed image to obtain the second reconstructed image; or performing sample adaptive compensation on the initial reconstructed image to obtain the second reconstructed image; or performing deblocking filtering and sample adaptive compensation on the initial reconstructed image to obtain the second reconstructed image. In other words, when the decoder performs loop filtering on the reconstructed image based on an image or a block, an edge image containing edge information of all reconstructed samples of the current block is determined by edge detection on the reconstructed image, wherein the reconstructed image can be an initial reconstructed image composed of reconstructed blocks, or can be other images such as an image after deblocking filtering or an image after sample adaptive compensation.

[0181] In some embodiments, the method further includes: decoding the bitstream to determine the index value of the first edge detection operator; and determining the first edge detection operator from a list of edge detection operators based on the index value of the first edge detection operator. In other words, the encoder can use multiple existing edge detection operators for edge detection, make encoding decisions, determine the optimal edge detection operator based on the minimum cost, encode the index value of the optimal edge detection operator, write the encoding information into the bitstream, and transmit it to the decoder; the decoder selects the optimal edge detection operator based on the parsed index value of the edge detection operator and performs edge detection on the reconstructed image.

[0182] In other embodiments, the first edge detection operator is a preset edge detection operator.

[0183] In the embodiment of the present application, the first edge detection operator may be one of the following: a Sobel operator, a Laplacian operator, a Roberts operator, and the like.

[0184] Exemplarily, the first edge detection operator is a Sobel operator;

[0185] Performing edge detection on the second reconstructed image based on the first edge detection operator to determine the second edge image includes: detecting gradient information of the second reconstructed image to determine transverse gradient information and longitudinal gradient information; and determining the second edge image based on the transverse gradient information and the longitudinal gradient information.

[0186] The detection principle of the Sobel operator is as follows:

[0187] As shown in the above formula, A represents the image to be extracted edge information, Gx is the horizontal gradient information, and Gy is the vertical gradient information. After extracting the gradient information of all sample points in image A, the sum of the horizontal and vertical gradient information is calculated:

[0188] In some embodiments, a second edge image is determined based on the transverse gradient information and the longitudinal gradient information, including: determining the absolute value of the gradient of the reconstructed sample based on the transverse gradient of the reconstructed sample in the transverse gradient information and the longitudinal gradient in the longitudinal gradient information; and using the absolute value of the gradient as the edge information of the reconstructed sample.

[0189] It should be noted that, in order to reduce computational complexity, G can be obtained by calculating the sum of the absolute values ​​of Gx and Gy instead of the above formula. The specific expression is as follows: G = |Gx| + |Gy|

[0190] In some embodiments, the method further includes: obtaining a second reconstructed image of the frame in which the current block is located; performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; performing denoising on the second edge image to obtain a denoised second edge image; and further, obtaining the first edge image of the current block from the denoised second edge image.

[0191] Exemplarily, the second edge image is denoised to obtain a denoised second edge image, including: if the edge strength of the first reconstructed sample is greater than or equal to a first threshold, retaining the edge strength of the first reconstructed sample; if the edge strength of the second reconstructed sample is less than the first threshold, setting the edge strength of the second reconstructed sample to zero.

[0192] It should be noted that, in the edge image obtained by calculation based on the edge detection operator, etc., extremely low gradient amplitude contributes little to enhancing high-frequency information or is noise, so the noise can be filtered out by the first threshold to obtain an updated edge image.

[0193] The first threshold may be a preset specific threshold or may be determined based on edge strength characteristics of the current edge image. In some embodiments, the method further comprises: determining an average value of edge strengths of all reconstructed samples in the second edge image; and determining the first threshold based on the average value.

[0194] In some embodiments, the first edge image and the second edge image are chroma component edge images, and the method further includes: determining a luminance component second edge image and a chroma component second edge image; and determining a final chroma component second edge image based on the luminance component second edge image and the chroma component second edge image.

[0195] It should be noted that since the chrominance component usually has a smaller number of samples, the edge contour of the chrominance component is not as clear as the edge contour of the luminance component. For the chrominance component edge image, the determination of the chrominance component edge image can be guided by the luminance component edge image, thereby improving the quality of the chrominance component edge image.

[0196] Exemplarily, the second edge image of the luminance component is downsampled to obtain a downsampled luminance component edge image, and the edge intensity mean of each sample of the downsampled luminance component edge image and the chrominance component edge image is calculated to obtain the final second edge image of the chrominance component.

[0197] In some other embodiments, the second edge image of the luminance component may be directly used as the second edge image of the chrominance component.

[0198] S905: Input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block.

[0199] It should be noted that video content often contains strong edge information, also known as high-frequency information. This is a crucial component of subjective vision, yet also difficult to gain. Inputting edge images as additional edge information into the loop filtering model provides the network with sample-level edge information, enabling it to adjust filter strength learning at the sample level and improve filtering effectiveness. If the first indication information determines that the current block does not utilize neural network-based loop filtering technology, then the first reconstructed image of the current block utilizes other filtering technologies.

[0200] In some embodiments, Figure 7 is a schematic diagram of the composition structure of a loop filtering model in an embodiment of the present application. The neural network-based loop filtering model 70 includes: an input unit 701, a feature extraction unit 702 and an output unit 703; the first reconstructed image Rec and the first edge image G are input into the input unit 701, and after passing through the feature extraction unit 702 and the output unit 703, the filtered image is output.

[0201] In other embodiments, the first reconstructed image Rec is input into the input unit 701, and then passes through the feature extraction unit 702. The feature extraction unit 702 inputs the feature information of the output first reconstructed image into the output unit 703; the first edge image G is input into the output unit 703. The output unit 703 processes the feature maps of the first edge image and the first reconstructed image, and outputs the filtered image.

[0202] In some other embodiments, the first edge image G and the first reconstructed image Rec are input to the input unit 701, and the first edge image G is input to the output unit 703. The output unit 703 processes the feature information of the first edge image and the first reconstructed image and outputs the filtered image.

[0203] It should be noted that the processing of edge information in the loop filtering model can be different from other input information. Since edge information reflects high-frequency information, it can be more effective at a later position in the loop filtering model. The edge information is directly input into the back-end output unit of the loop filtering model, and the original input edge information is used for image reconstruction to ensure the loop filtering auxiliary effect of the edge information.

[0204] In some embodiments, the method further includes: obtaining at least one of a predicted image, boundary strength information, slice type information, and quantization information of the current block; and simultaneously inputting at least one of the predicted image, boundary strength information, slice type information, and quantization information into a neural network-based loop filtering model to obtain a filtered image of the current block.

[0205] In some embodiments, the method further includes: performing edge detection on a training image based on multiple edge detection operators during a model training phase to obtain multiple edge images; constructing a training sample set based on the training image, the multiple edge images, and the true image; and training a neural network-based loop filter model using the training sample set to obtain a trained loop filter model. It should be noted that during the model training phase, different types of operators can be used to calculate edge images, a training dataset can be constructed using the multiple edge images, and the model can be trained using the training dataset to improve the robustness of the model.

[0206] In some embodiments, the method further includes: decoding the bitstream to determine second indication information; determining, based on the second indication information, to use a residual scaling technique for the current block, decoding the bitstream to determine a residual scaling parameter for the current block; determining a sample residual of the current block based on reconstructed samples in the first reconstructed image and reconstructed samples in the filtered image; scaling the sample residual based on the residual scaling parameter to obtain a scaled residual of the current block; and determining a target reconstructed image for the current block based on the first reconstructed image and the scaled residual.

[0207] In some embodiments, the second indication information includes at least one of the following: a frame-level syntax element identifier, used to indicate whether the frame where the current block is located uses residual scaling technology; a slice-level syntax element identifier, used to indicate whether the slice where the current block is located uses residual scaling technology; a coding tree unit-level syntax element identifier, used to indicate whether the coding tree unit where the current block is located uses residual scaling technology.

[0208] In some embodiments, decoding the code stream and determining the residual scaling parameter of the current block includes: decoding the code stream and determining third indication information; determining the decoded residual scaling parameter according to the third indication information, decoding the code stream and determining the residual scaling parameter of the current block; determining an index value of the decoded residual scaling parameter according to the third indication information, decoding the code stream and determining the index value of the residual scaling parameter; and determining the residual scaling parameter from a residual scaling parameter list according to the index value of the residual scaling parameter.

[0209] In some embodiments, the third indication information may be an index value of a residual scaling parameter, denoted as scaleIdx. When scaleIdx is 0, the decoded residual scaling parameter is determined. When scaleIdx is not 0, the index value of the residual scaling parameter is determined according to scaleIdx.

[0210] In some embodiments, decoding the code stream and determining the residual scaling parameters of the current block include: decoding the code stream and at least two residual scaling parameters of the current block; scaling the sample residuals according to the residual scaling parameters to obtain the scaled residual of the current block, including: classifying and scaling the sample residuals according to the at least two residual scaling parameters to obtain the scaled residual of the current block.

[0211] Furthermore, the method also includes: classifying samples of the current block based on the first edge image to determine the sample type of each sample in the current block; wherein each sample type corresponds to a residual scaling parameter. In other words, the edge image not only serves as additional side information for the loop filtering model, providing the network with sample-level edge information to adjust the learning of the filter strength at the sample level and improve the filtering effect, but also serves as sample classification information to classify and scale sample parameters, improving the accuracy of the scaled residual, thereby improving the reconstructed image quality and decoding efficiency.

[0212] Based on the above embodiments, the decoding method provided in the embodiments of the present application is further illustrated.

[0213] In this embodiment, at the decoder end, the decoder parses or obtains a flag indicating the permission to use neural network loop filtering (NNLF). This flag is a sequence-level flag (sps_nnlf_enable_flag), indicating that the current decoder allows the use of NNLF-based loop filtering. If sps_nnlf_enable_flag is true, the process starts from step 1; otherwise, the process starts from step 3.

[0214] 1. Parse the bitstream to obtain the frame-level usage flag sh_nnlf_flag of the neural network-based loop filtering technology. If this flag is true, parse or set the neural network-based loop filtering usage flag ctb_nnlf_flag of all coding tree units in the current frame based on the flag information; otherwise, the neural network-based loop filtering usage flag ctb_nnlf_flag of all coding tree units in the current frame is false, indicating that all coding tree units in the current frame do not use this technology.

[0215] If sh_nnlf_flag is true, the residual scaling flag scaleFlag of the current frame is parsed; otherwise, scaleFlag defaults to false.

[0216] If scaleFlag is true, the residual scaling parameter index scaleIdx is further parsed; if the parsed scaleIdx indicates that the code stream needs to be further parsed to obtain the residual scaling parameter, the code stream is parsed to obtain the residual scaling parameter scale of the color component of the current frame; otherwise, the preset residual scaling parameter scale is obtained according to the index.

[0217] 2. Obtain the reconstructed sample image of the input network model, use the Sobel operator to calculate the corresponding horizontal gradient Gx and vertical gradient Gy for each sample of the reconstructed sample image, and take the square root of the sum of the squares of Gx and Gy to obtain the gradient amplitude of each sample point, that is, an edge image of the same size as the reconstructed sample image; it should be noted that the edge image is calculated based on the Sobel operator or the Laplace operator, etc., in which the extremely low gradient amplitude contributes little to the enhancement of high-frequency information or is noise, so in some embodiments, a threshold can be pre-set to filter out the noise to obtain an updated edge image. Exemplarily, the mean Gavg of all gradient amplitude samples in the edge image is solved, and the edge image is updated. For all samples in the edge image whose gradient amplitude samples are greater than Gavg, their gradient amplitudes are retained; otherwise, they are set to zero.

[0218] 3. Initialize the network model according to preset parameters;

[0219] If the neural network-based loop filter flag ctb_nnlf_flag for the current coding tree unit is true, the reconstructed samples rec, predicted samples pred, boundary strength BS, mode information IPB, quantization information BaseQP and SliceQP, and edge image of the current coding tree unit area are obtained and input into the network model for inference calculation. After network model inference, the filtered reconstructed samples filteredRec are obtained. If scaleFlag is true, the residual scaling parameter index scaleIdx obtained by parsing is used to obtain the residual scaling factor scale for each color component. The scale is multiplied by the residual between the filtered reconstructed samples filteredRec and the pre-filtered reconstructed samples rec, and the scaled residual is added back to the pre-filtered reconstructed samples rec to obtain the output samples output. If the current frame or current coding tree unit does not use residual scaling technology, the filtered reconstructed samples filteredRec are the output samples output.

[0220] If the neural network-based loop filter usage flag ctb_nnlf_flag of the current coding tree unit is false, the reconstructed sample rec is the output sample output.

[0221] 4. The decoding end continues to operate other loop filtering technologies.

[0222] 5. After executing all loop filtering tools, the final output image is obtained.

[0223] By adopting the above technical solution, at the decoding end, the edge image is input into the loop filtering model as new edge information, which can provide the network with sample-level edge information, adjust the learning of the filter strength at the sample level, improve the filtering effect, and thus improve the reconstructed image quality and decoding performance.

[0224] In yet another embodiment of the present application, based on the same inventive concept as the aforementioned embodiment, FIG10 is a schematic diagram showing the structure of an encoder provided by an embodiment of the present application. As shown in FIG10 , the encoder 100 may include a first determination unit 1001, a first filtering unit 1002, a decision unit 1003, and an encoding unit 1004; wherein:

[0225] A first determining unit 1001 is configured to determine whether a neural network-based loop filtering technology is allowed to be used for a current block;

[0226] The first determining unit 1001 is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0227] The first determining unit 1001 is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0228] a first filtering unit 1002 configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block;

[0229] The decision unit 1003 is configured to perform cost calculation based on the original image and the filtered image of the current block to determine the first generation value;

[0230] The decision unit 1003 is further configured to determine whether the current block uses the neural network-based loop filtering technology according to the first generation value, and set first indication information of the current block;

[0231] The encoding unit 1004 is configured to encode the first indication information and write the obtained encoded bits into the bit stream.

[0232] It can be understood that each functional unit of the encoder also executes the encoding method of any one of the aforementioned embodiments.

[0233] It is understandable that in the embodiments of the present application, a "unit" can be a portion of a circuit, a portion of a processor, a portion of a program or software, etc., and of course it can also be a module, or it can be non-modular. Moreover, the various components in this embodiment can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into a single unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional modules.

[0234] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, or the portion that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media that can store program code, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0235] Therefore, an embodiment of the present application provides a computer-readable storage medium, which is applied to the encoder 100. The computer-readable storage medium stores a computer program, and when the computer program is executed by the first processor, it implements the method of any one of the aforementioned embodiments.

[0236] The present application also provides a computer-readable storage medium that stores a bitstream generated by the encoding method of any of the aforementioned embodiments. The bitstream is generated by bit-encoding information to be encoded; the information to be encoded includes at least one of the following: first indication information, second indication information, third indication information, and residual information, wherein the first indication information indicates whether a neural network-based loop filtering technique is used for the current block, the second indication information indicates whether a residual scaling technique is used for the frame containing the current block, and the third indication information indicates a decoded residual scaling parameter.

[0237] Based on the composition of the encoder 100 and the computer-readable storage medium, refer to Figure 11, which shows a specific hardware structure diagram of the encoder 100 provided in an embodiment of the present application. As shown in Figure 11, the encoder 100 may include: a first communication interface 1101, a first memory 1102 and a first processor 1103; each component is coupled together through a first bus system 1104. It can be understood that the first bus system 1104 is used to achieve connection and communication between these components. In addition to the data bus, the first bus system 1104 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the first bus system 1104 in Figure 11. Among them,

[0238] The first communication interface 1101 is used to receive and send signals when sending and receiving information with other external network elements;

[0239] A first memory 1102 is used to store computer programs that can be run on the first processor 1103;

[0240] The first processor 1103 is configured to, when running a computer program, execute:

[0241] Determining whether the current block allows the use of a neural network-based loop filtering technique;

[0242] Obtaining a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0243] Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0244] Inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block;

[0245] Perform cost calculation based on the original image and filtered image of the current block to determine the first generation value;

[0246] Determining whether a current block uses a neural network-based loop filtering technology according to the first generation value, and setting first indication information of the current block;

[0247] The first indication information is encoded, and the obtained encoded bits are written into a bitstream.

[0248] It is understood that the first memory 1102 in the embodiment of the present application can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct RAM bus random access memory (DRRAM). The first memory 1102 of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0249] The first processor 1103 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by hardware integrated logic circuits or software instructions in the first processor 1103. The above-mentioned first processor 1103 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The various methods, steps, and logic block diagrams disclosed in the embodiments of this application can be implemented or executed. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly implemented as a hardware decoding processor, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium mature in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, or electrically erasable programmable memory, registers, etc. The storage medium is located in the first memory 1102 , and the first processor 1103 reads the information in the first memory 1102 and completes the steps of the above method in combination with its hardware.

[0250] It is understood that the embodiments described herein can be implemented with hardware, software, firmware, middleware, microcode or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSP devices, DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions of the present application or a combination thereof. For software implementation, the technology of the present application can be implemented by a module (such as a process, a function, etc.) that performs the functions of the present application. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0251] Optionally, as another embodiment, the first processor 1103 is further configured to execute any one of the methods in the foregoing embodiments when running a computer program.

[0252] This embodiment provides an encoder in which an edge image is input into a loop filtering model as newly added edge information, thereby providing the network with sample-level edge information, adjusting the learning of the filtering strength at the sample level, improving the filtering effect, and thereby enhancing the reconstructed image quality and coding performance.

[0253] In yet another embodiment of the present application, based on the same inventive concept as the aforementioned embodiment, FIG12 is a schematic diagram showing the structure of a decoder 120 provided in an embodiment of the present application. As shown in FIG12 , the decoder 120 may include: a decoding unit 1201, a second determining unit 1202, and a second filtering unit 1203; wherein:

[0254] The decoding unit 1201 is configured to decode the code stream and determine the first indication information;

[0255] The second determining unit 1202 is configured to determine, according to the first indication information, whether to use a neural network-based loop filtering technology for the current block;

[0256] The second determining unit 1202 is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes the reconstructed samples of the current block;

[0257] The second determining unit 1202 is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0258] The second filtering unit 1203 is configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block.

[0259] It can be understood that each functional unit of the decoder also executes the decoding method of any one of the aforementioned embodiments.

[0260] Based on the composition of the decoder 120 and the computer-readable storage medium, refer to Figure 13, which shows a specific hardware structure diagram of the decoder 120 provided in an embodiment of the present application. As shown in Figure 13, the decoder 120 may include: a second communication interface 1301, a second memory 1302 and a second processor 1303; each component is coupled together through a second bus system 1304. It can be understood that the second bus system 1304 is used to achieve connection and communication between these components. In addition to the data bus, the second bus system 1304 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, various buses are labeled as the second bus system 1304 in Figure 13. Among them,

[0261] The second communication interface 1301 is used to receive and send signals during the process of sending and receiving information between other external network elements;

[0262] The second memory 1302 is used to store computer programs that can be run on the second processor 1303;

[0263] The second processor 1303 is configured to, when running the computer program, execute:

[0264] Decoding the code stream to determine first indication information;

[0265] Determining, according to the first indication information, that a neural network-based loop filtering technology is used for the current block;

[0266] Obtaining a first reconstructed image of the current block; wherein the first reconstructed image includes reconstructed samples of the current block;

[0267] Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample;

[0268] The first reconstructed image and the first edge image are input into a loop filtering model based on a neural network to obtain a filtered image of the current block.

[0269] Optionally, as another embodiment, the second processor 1303 is further configured to execute any one of the methods in the foregoing embodiments when running a computer program.

[0270] It can be understood that the hardware functions of the second memory 1302 and the first memory 1102 are similar, and the hardware functions of the second processor 1303 and the first processor 1103 are similar; they will not be described in detail here.

[0271] This embodiment provides a decoder in which edge images are input into a loop filtering model as newly added edge information, thereby providing the network with sample-level edge information, adjusting the learning of the filtering strength at the sample level, improving the filtering effect, and thereby enhancing the reconstructed image quality and decoding performance.

[0272] In yet another embodiment of the present application, referring to FIG14 , a schematic diagram of the structure of a coding and decoding system provided by an embodiment of the present application is shown. As shown in FIG14 , the coding and decoding system 140 may include an encoder 1401 and a decoder 1402 .

[0273] In the embodiment of the present application, the encoder 1401 may be the encoder described in any one of the aforementioned embodiments, and the decoder 1402 may be the decoder described in any one of the aforementioned embodiments.

[0274] It should be noted that, in this application, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0275] The serial numbers of the above embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0276] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments. The features disclosed in the several product embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new product embodiments. The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined, if they do not conflict, to obtain new method embodiments or device embodiments.

[0277] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims. Industrial Applicability

[0278] In an embodiment of the present application, a coding and decoding method, an encoder, a decoder, and a storage medium are provided. At the coding and decoding end, it is determined whether a neural network-based loop filtering technology is used or allowed to be used for the current block; a first reconstructed image of the current block is obtained; wherein the first reconstructed image includes a reconstructed sample of the current block; a first edge image of the current block is obtained; wherein the first edge image includes edge information of the reconstructed sample; the first reconstructed image and the first edge image are input into a neural network-based loop filtering model to obtain a filtered image of the current block. In this way, inputting the edge image as newly added edge information into the loop filtering model can provide the network with sample-level edge information, adjust the learning of the filter strength at the sample level, improve the filtering effect, and thus improve the quality of the reconstructed image and coding and decoding performance.

Claims

1. A decoding method, applied to a decoder, the method comprising: Decoding the code stream to determine first indication information; Determining, according to the first indication information, that the current block uses a neural network-based loop filtering technology; Acquire a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block; Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample; The first reconstructed image and the first edge image are input into a loop filtering model based on a neural network to obtain a filtered image of the current block.

2. The method according to claim 1, wherein: The edge information includes edge strength.

3. The method according to claim 1 or 2, wherein: The method further comprises: Obtain a second reconstructed image of the frame where the current block is located; Performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; The step of obtaining a first edge image of the current block includes: The first edge image of the current block is obtained from the second edge image.

4. The method according to claim 3, wherein: The obtaining of the second reconstructed image of the frame where the current block is located comprises: Get the initial reconstructed image of the frame where the current block is located; Using the initial reconstructed image as the second reconstructed image; or, Deblocking filtering and / or sample adaptive compensation are performed on the initial reconstructed image to obtain the second reconstructed image.

5. The method according to claim 3, wherein: The method further comprises: Decoding the bitstream to determine the index value of the first edge detection operator; The first edge detection operator is determined from an edge detection operator list according to the index value of the first edge detection operator.

6. The method according to claim 3, wherein: The first edge detection operator is a Sobel operator; The step of performing edge detection on the second reconstructed image based on the first edge detection operator to determine the second edge image includes: Detecting gradient information of the second reconstructed image to determine transverse gradient information and longitudinal gradient information; The second edge image is determined according to the transverse gradient information and the longitudinal gradient information.

7. The method according to claim 6, wherein: The determining the second edge image according to the transverse gradient information and the longitudinal gradient information includes: Determine the absolute value of the gradient of the reconstructed sample according to the transverse gradient of the reconstructed sample in the transverse gradient information and the longitudinal gradient in the longitudinal gradient information; The absolute value of the gradient is used as edge information of the reconstructed sample.

8. The method according to claim 3, wherein: After determining the second edge image, the method further includes: The second edge image is subjected to denoising processing to obtain a denoised second edge image.

9. The method according to claim 8, wherein: The denoising process is performed on the second edge image to obtain a denoised second edge image, comprising: If the edge strength of the first reconstructed sample is greater than or equal to the first threshold, retaining the edge strength of the first reconstructed sample; If the edge strength of the second reconstructed sample is less than the first threshold, the edge strength of the second reconstructed sample is set to zero.

10. The method according to claim 9, wherein: The method further comprises: determining an average value of edge strengths of all reconstructed samples in the second edge image; The first threshold is determined according to the average value.

11. The method according to claim 3, wherein: The first edge image and the second edge image are chrominance component edge images, and the method further includes: Determine a second edge image of a luminance component and a second edge image of a chrominance component; According to the second edge image of the luminance component and the second edge image of the chrominance component, a final second edge image of the chrominance component is determined.

12. The method according to claim 1, wherein: The obtaining of a first reconstructed image of the current block comprises: Get the initial reconstructed image of the frame where the current block is located; The first reconstructed image of the current block is obtained from the initial reconstructed image.

13. The method according to claim 1, wherein: The method further comprises: Obtain at least one of a predicted image, boundary strength information, slice type information, and quantization information of a current block; At least one of the predicted image, the boundary strength information, the slice type information and the quantization information is simultaneously input into the neural network-based loop filtering model to obtain a filtered image of the current block.

14. The method according to claim 1, wherein: The first indication information includes: a first syntax element identifier and a second syntax element identifier; Among them, the first syntax element identifier is used to indicate whether the image sequence where the current block is located allows the use of neural network-based loop filtering technology, and the second syntax element identifier is used to indicate whether the current block uses neural network-based loop filtering technology.

15. The method according to any one of claims 1 to 14, wherein: The neural network-based loop filtering model includes: an input unit, a feature extraction unit and an output unit; The step of inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block includes: The first reconstructed image is input into the input unit, and then passes through the feature extraction unit, and the feature extraction unit inputs the feature information of the outputted first reconstructed image into the output unit; The first edge image is input to the output unit, and the output unit processes the first edge image and the feature map of the first reconstructed image and outputs the filtered image.

16. The method according to any one of claims 1 to 15, wherein: The method further comprises: In the model training stage, edge detection is performed on the training image based on multiple edge detection operators to obtain multiple edge images; Constructing a training sample set according to the training image, the plurality of edge images and the true value image; The training sample set is used to train a loop filter model based on a neural network to obtain a trained loop filter model.

17. A coding method, applied to an encoder, the method comprising: Determining that the current block allows the use of a neural network-based loop filtering technique; Acquire a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block; Acquire a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample; Inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block; Performing cost calculation based on the original image of the current block and the filtered image to determine the first generation value; Determine whether the current block uses a neural network-based loop filtering technology according to the first generation value, and set first indication information of the current block; The first indication information is encoded, and the obtained encoded bits are written into a bit stream.

18. The method according to claim 17, wherein: The edge information includes edge strength.

19. The method according to claim 17 or 18, wherein: The method further comprises: Obtain a second reconstructed image of the frame where the current block is located; Performing edge detection on the second reconstructed image based on the first edge detection operator to determine a second edge image; The step of obtaining a first edge image of the current block includes: The first edge image of the current block is obtained from the second edge image.

20. The method according to claim 19, wherein: The obtaining of the second reconstructed image of the frame where the current block is located comprises: Get the initial reconstructed image of the frame where the current block is located; Using the initial reconstructed image as the second reconstructed image; or, Deblocking filtering and / or sample adaptive compensation are performed on the initial reconstructed image to obtain the second reconstructed image.

21. The method according to claim 19, wherein: The method further comprises: Traversing the edge detection operator list, obtaining a candidate edge detection operator as the first edge detection operator; If the minimum cost value is the first generation value, determine to use the loop filtering technology based on the neural network; Determine the index value of the optimal edge detection operator corresponding to the minimum cost value; The index value of the best edge detection operator is encoded.

22. The method according to claim 19, wherein: The first edge detection operator is a Sobel operator; The step of performing edge detection on the second reconstructed image based on the first edge detection operator to determine the second edge image includes: Detecting gradient information of the second reconstructed image to determine transverse gradient information and longitudinal gradient information; The second edge image is determined according to the transverse gradient information and the longitudinal gradient information.

23. The method according to claim 22, wherein: The determining the second edge image according to the transverse gradient information and the longitudinal gradient information includes: According to the transverse gradient of the reconstructed sample in the transverse gradient information and the longitudinal gradient in the longitudinal gradient information, Determining the absolute value of the gradient of the reconstructed sample; The absolute value of the gradient is used as edge information of the reconstructed sample.

24. The method according to claim 19, wherein: After determining the second edge image, the method further includes: The second edge image is subjected to denoising processing to obtain a denoised second edge image.

25. The method according to claim 24, wherein: The denoising process is performed on the second edge image to obtain a denoised second edge image, comprising: If the edge strength of the first reconstructed sample is greater than or equal to the first threshold, retaining the edge strength of the first reconstructed sample; If the edge strength of the second reconstructed sample is less than the first threshold, the edge strength of the second reconstructed sample is set to zero.

26. The method according to claim 25, wherein: The method further comprises: determining an average value of edge strengths of all reconstructed samples in the second edge image; The first threshold is determined according to the average value.

27. The method of claim 19, wherein: The first edge image and the second edge image are chrominance component edge images, and the method further includes: Determine a second edge image of a luminance component and a second edge image of a chrominance component; According to the second edge image of the luminance component and the second edge image of the chrominance component, a final second edge image of the chrominance component is determined.

28. The method of claim 17, wherein: The obtaining of a first reconstructed image of the current block comprises: Get the initial reconstructed image of the frame where the current block is located; The first reconstructed image of the current block is obtained from the initial reconstructed image.

29. The method of claim 17, wherein: The method further comprises: Obtain at least one of a predicted image, boundary strength information, slice type information, and quantization information of a current block; At least one of the predicted image, the boundary strength information, the slice type information and the quantization information is simultaneously input into the neural network-based loop filtering model to obtain a filtered image of the current block.

30. The method of claim 17, wherein: The first indication information includes: a first syntax element identifier and a second syntax element identifier; Among them, the first syntax element identifier is used to indicate whether the image sequence where the current block is located allows the use of neural network-based loop filtering technology, and the second syntax element identifier is used to indicate whether the current block uses neural network-based loop filtering technology.

31. The method according to any one of claims 17 to 30, wherein: The neural network-based loop filtering model includes: an input unit, a feature extraction unit and an output unit; The step of inputting the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block includes: The first reconstructed image is input into the input unit, and then passes through the feature extraction unit, and the feature extraction unit inputs the feature information of the outputted first reconstructed image into the output unit; The first edge image is input to the output unit, and the output unit processes the first edge image and the feature map of the first reconstructed image and outputs the filtered image.

32. The method according to any one of claims 17 to 31, wherein: The method further comprises: In the model training stage, edge detection is performed on the training image based on multiple edge detection operators to obtain multiple edge images; Constructing a training sample set according to the training image, the plurality of edge images and the true value image; The training sample set is used to train a loop filter model based on a neural network to obtain a trained loop filter model.

33. An encoder, comprising a first determining unit, a first filtering unit, a decision unit and an encoding unit; wherein: The first determining unit is configured to determine whether the current block allows the use of a neural network-based loop filtering technology; The first determining unit is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block; The first determining unit is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample; The first filtering unit is configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block; The decision unit is configured to perform cost calculation based on the original image of the current block and the filtered image to determine the first generation value; The decision unit is further configured to determine whether the current block uses a neural network-based loop filtering technology according to the first generation value, and set first indication information of the current block; The encoding unit is configured to encode the first indication information and write the obtained encoded bits into a bit stream.

34. An encoder comprising a first memory and a first processor; wherein: The first memory is used to store a computer program that can be run on the first processor; The first processor is configured to execute the method according to any one of claims 17 to 32 when running the computer program.

35. A decoder comprising a decoding unit, a second determining unit and a second filtering unit; wherein: The decoding unit is configured to decode the code stream and determine the first indication information; The second determining unit is configured to determine, according to the first indication information, that the current block uses a loop filtering technology based on a neural network; The second determining unit is further configured to obtain a first reconstructed image of the current block; wherein the first reconstructed image includes a reconstructed sample of the current block; The second determining unit is further configured to obtain a first edge image of the current block; wherein the first edge image includes edge information of the reconstructed sample; The second filtering unit is configured to input the first reconstructed image and the first edge image into a loop filtering model based on a neural network to obtain a filtered image of the current block.

36. A decoder comprising a second memory and a second processor; wherein: The second memory is used to store a computer program that can be run on the second processor; The second processor is configured to execute the method according to any one of claims 1 to 16 when running the computer program.

37. A computer-readable storage medium, wherein: The computer-readable storage medium stores a code stream generated by the encoding method according to any one of claims 17 to 32.

38. A computer-readable storage medium, wherein: The computer-readable storage medium stores a computer program, and when the computer program is executed, the method according to any one of claims 1 to 16 is implemented, or the method according to any one of claims 17 to 32 is implemented.

Images