Neural network-based image filtering method and device, electronic device, and computer program

Abstract

This application provides a neural network-based image filtering method, apparatus, device, and storage medium, which can be applied in fields such as image processing, video codec, and live video broadcasting. The method includes the steps of: determining a first data processing method for the information to be filtered from M+1 data processing methods; if the first data processing method is a data rearrangement method, performing data rearrangement on reconstructed image blocks in the information to be filtered based on the first data processing method, and performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain filtered image blocks of the rearranged information to be filtered; and performing reverse data rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing method to obtain filtered image blocks of the reconstructed image blocks. By performing data rearrangement on the information to be filtered, the data distribution characteristics of the rearranged information to be filtered can be made to match or approach those of the training data, thereby enhancing the filtering effect of the neural network filter.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application filed with the China Patent Office on June 21, 2023, bearing application number 2023107519314 and entitled "Neural network-based image filtering method, device, equipment and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The embodiments of the present application relate to the technical field of image encoding and decoding (codec), and in particular to a neural network-based image filtering method and apparatus, electronic device, and computer program. [Background technology]

[0003] With the development of video technology, the amount of data contained in video data has become relatively large. To facilitate the transmission of video data, video devices perform video compression, which allows for more efficient transmission or storage of video data. Video compression requires operations such as inverse quantization and inverse transform to obtain a reconstructed image on both the encoding side (encoding end) and the decoding side (decoding end). Because video compression introduces loss, it is necessary to filter the reconstructed image to reduce the compression loss of the image.

[0004] With the rapid development of neural network technology, neural network filters have been widely applied in video processing. However, in the actual application process, there may be a discrepancy between the actual filtering information of the neural network filter and the data distribution of the training data, which may result in insufficient filtering effect of the neural network filter. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application aim to provide a neural network-based image filtering method and apparatus, an electronic device, and a computer program that can improve the filtering effect of an image. [Means for solving the problem]

[0006] According to a first aspect, the present application provides a neural network based image filtering method, which is applied to a decoding device, and which comprises: determining a first data processing method for the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered including a reconstructed image block to be filtered, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, where M is a positive integer; When the first data processing method is one of the M data rearrangement methods, perform data rearrangement on the reconstructed image block according to the first data processing method to obtain rearranged filtering information; performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and The method includes performing data inverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block.

[0007] According to a second aspect, the present application provides a neural network based image filtering method, adapted for use in an encoding device, the method comprising: determining a first data processing method for the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered including a reconstructed image block, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the information to be filtered according to the first data processing manner to obtain rearranged information to be filtered; performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and The method includes performing data inverse rearrangement on the filtered image blocks of the rearranged information to be filtered according to the first data processing manner to obtain filtered image blocks of the reconstructed image blocks.

[0008] According to a third aspect, the present application provides an image filtering device based on a neural network, adapted for use in a decoding device, the device comprising: a reordering determination unit for determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block to be filtered, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; a rearrangement filtering unit for, when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the reconstructed image block according to the first data processing manner to obtain rearranged information to be filtered, and performing filtering on the rearranged information to be filtered by a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and The apparatus further includes an inverse rearrangement unit for performing data inverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block.

[0009] According to a fourth aspect, the present application provides a neural network based image filtering device, adapted for use in an encoding device, the device comprising: a reordering determination unit for determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; a rearrangement filtering unit for, when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the information to be filtered according to the first data processing manner to obtain rearranged information to be filtered, and performing filtering on the rearranged information to be filtered by a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and The apparatus further includes a reverse rearrangement unit for performing data reverse rearrangement on the filtered image blocks of the rearranged information to be filtered according to the first data processing manner to obtain filtered image blocks of the reconstructed image blocks.

[0010] According to a fifth aspect, there is provided a decoder including a processor and a memory, wherein the memory stores a computer program, and the processor is adapted to execute the method of the first aspect or each implementation thereof by calling and executing the computer program stored in the memory.

[0011] According to a sixth aspect, there is provided an encoder including a processor and a memory, wherein the memory stores a computer program, and the processor is adapted to execute the method of the second aspect or each implementation thereof by calling and executing the computer program stored in the memory.

[0012] According to a seventh aspect, there is provided a chip for implementing the method of any one of the first to second aspects or their respective implementations. Specifically, the chip includes a processor that retrieves and executes a computer program from a memory, thereby causing a device in which the chip is provided to implement the method of any one of the first to second aspects or their respective implementations.

[0013] According to an eighth aspect, there is provided a computer-readable storage medium storing a computer program for causing a computer to perform a method according to any one of the first to second aspects or each implementation thereof.

[0014] According to a ninth aspect, there is provided a computer program product, comprising computer program instructions for causing a computer to perform a method according to any one of the first to second aspects or each implementation thereof.

[0015] According to a tenth aspect, there is provided a computer program which, when executed on a computer, causes the computer to perform the method of any one of the first to second aspects or each implementation thereof. [Effects of the Invention]

[0016] In summary, in this application, a first data processing method for the information to be filtered is determined from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, and the information to be filtered includes reconstructed image blocks to be filtered; when the first data processing method is one of the M data rearrangement methods, data rearrangement is performed on the reconstructed image blocks in the information to be filtered according to the first data processing method to obtain rearranged information to be filtered, and a filtering process is performed on the rearranged information to be filtered using a neural network filter to obtain filtered image blocks of the rearranged information to be filtered; data reverse rearrangement is performed on the filtered image of the rearranged information to be filtered according to the first data processing method to obtain filtered image blocks of the reconstructed image blocks. In other words, in the present application, before filtering the information to be filtered, it is first determined whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to perform data rearrangement on the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered match or approach the data distribution characteristics of the training data, thereby improving the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and enhancing the performance of image encoding and decoding. [Brief explanation of the drawings]

[0017] [Figure 1]1 is an exemplary block diagram of a video codec system according to an embodiment of the present application; [Figure 2] FIG. 1 illustrates an encoding framework provided in an embodiment of the present application. [Figure 3] FIG. 1 illustrates a decoding framework provided in an embodiment of the present application. [Figure 4] FIG. 2 illustrates an encoding unit. [Figure 5] FIG. 1 illustrates the filtering process of a neural network filter. [Figure 6A] FIG. 1 illustrates the difference in data distribution characteristics between the training and test sets. [Figure 6B] FIG. 10 shows the data distribution characteristics of the test set after rotation. [Figure 7] 1 is a flowchart of a neural network-based image filtering method provided in an embodiment of the present application. [Figure 8A] FIG. 2 illustrates a reconstructed image block. [Figure 8B] FIG. 2 illustrates a reconstructed image block. [Figure 8C] FIG. 2 illustrates a reconstructed image block. [Figure 8D] FIG. 2 illustrates a reconstructed image block. [Figure 9] FIG. 10 illustrates an example of a filtered surrounding region of a reconstructed image block. [Figure 10] FIG. 10 is a diagram illustrating a process for determining a first data processing method according to an embodiment of the present application. [Figure 11] 1 is a flowchart of a neural network-based image filtering method provided in an embodiment of the present application. [Figure 12] FIG. 1 is an exemplary block diagram of a neural network-based image filtering device provided in an embodiment of the present application. [Figure 13] FIG. 1 is an exemplary block diagram of a neural network-based image filtering device provided in an embodiment of the present application. [Figure 14]1 is an exemplary block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0018] The following will more clearly and completely explain the technical solutions in the embodiments of this application in conjunction with the drawings in the embodiments of this application. It is clear that the described embodiments are only some embodiments, not all embodiments. All other embodiments that can be obtained by those skilled in the art based on the embodiments of this application without any creative efforts belong to the protection scope of this application.

[0019] It should be noted that the terms "first," "second," etc. in the specification, claims, and drawings of this application are used to distinguish between similar objects and not to describe a particular order or sequence. It should be understood that data used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein may be performed in an order other than that illustrated or described herein. In an embodiment of the present invention, "B corresponding to A" indicates that B is associated with A. In one implementation, B can be determined based on A. However, it should be understood that determining B based on A does not mean determining B based solely on A, and B can also be determined based on A and / or other information. Furthermore, the terms "comprise," "have," and any variations thereof are intended to cover a non-exclusive inclusion. For example, a process, method, system, product, or device comprising a series of steps or units is not necessarily limited to the explicitly listed steps or units, but may also include steps or units not explicitly listed, or other steps or units inherent to such a process, method, product, or device. In this specification, unless otherwise specified, "plurality" refers to two or more.

[0020] The present application may be applied to the fields of image codecs, video codecs, hardware video codecs, dedicated circuit video codecs, real-time video codecs, etc. For example, the scheme of the present application may be combined with a deep learning-based end-to-end image coding standard, such as JPEG AI. Alternatively, the scheme of the present application may be used in conjunction with other proprietary or industry standards, including ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262, ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, and ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including scalable video codec (SVC) and multiview video codec (MVC) extensions. It should be understood that the technical proposal of the present application is not limited to any particular codec standard or technology.

[0021] For ease of understanding, a video codec system according to an embodiment of the present application will be described first in conjunction with FIG.

[0022] FIG. 1 is an exemplary block diagram of a video codec system according to an embodiment of the present application. Note that FIG. 1 is merely an example, and the video codec system according to the embodiment of the present application includes, but is not limited to, the one shown in FIG. 1. As shown in FIG. 1, the video codec system 100 includes an encoding device 110 and a decoding device 120. The encoding device 110 encodes (or compresses) video data to generate a bitstream, and transmits the bitstream to the decoding device 120. The decoding device 120 decodes the bitstream generated by the encoding device 110 to obtain decoded video data.

[0023] The encoding device 110 in the embodiments of the present application may be understood as a device having a video encoding function, and the decoding device 120 may be understood as a device having a video decoding function, i.e., the encoding device 110 and the decoding device 120 in the embodiments of the present application include a wider range of devices, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, in-vehicle computers, etc.

[0024] In some embodiments, encoding device 110 may transmit encoded video data (e.g., a bitstream) to decoding device 120 via channel 130. Channel 130 may include one or more media and / or devices capable of transmitting encoded video data from encoding device 110 to decoding device 120.

[0025] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in a timely manner. In this example, encoding device 110 can modulate the encoded video data according to a communication standard and transmit the modulated video data to decoding device 120. Wherein, the communication media includes a wireless communication medium, such as an RF frequency spectrum; optionally, the communication medium may also include a wired communication medium, such as one or more physical transmission lines.

[0026] In another example, channel 130 includes a storage medium capable of storing the encoded video data of encoding device 110. The storage medium may include various locally accessible data storage media, such as optical disks, DVDs, flash memory, etc. In this example, decoding device 120 may obtain the encoded video data from the storage medium.

[0027] In another example, channel 130 may include a storage server that can store the encoded video data of encoding device 110. In this example, decoding device 120 can download the stored encoded video data from the storage server. Optionally, the storage server can store the encoded video data and transmit the encoded video data to decoding device 120, such as to a web server (e.g., for a website), a File Transfer Protocol (FTP) server, etc.

[0028] In some embodiments, encoding device 110 includes a video encoder 112 and an output interface 113, where output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0029] In some embodiments, encoding device 110 may further include a video source 111 in addition to video encoder 112 and input interface 113 .

[0030] The video source 111 may include at least one of a video collection device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, of which the video input interface is used to receive video data from a video content provider and the computer graphics system is used to generate video data.

[0031] The video encoder 112 encodes video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. A bitstream contains coding information for a picture or a sequence of pictures in the form of a bitstream. The coding information may include coded image data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. An SPS may contain parameters that apply to one or more sequences. A PPS may contain parameters that apply to one or more pictures. A syntax structure refers to a set of one or more syntax elements arranged in a predetermined order in a bitstream.

[0032] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data can also be stored on a storage medium or a storage server for later reading by the decoding device 120.

[0033] In some embodiments, decoding device 120 includes an input interface 121 and a video decoder 122 .

[0034] In some embodiments, the decoding device 120 may further include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0035] The input interface 121 includes a receiver and / or a modem, and can receive encoded video data through a channel 130.

[0036] The video decoder 122 is used to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to a display device 123 .

[0037] The display device 123 displays the decoded video data and may be integrated into the decoding device 120 or located external to the decoding device 120. The display device 123 may include various display devices, for example, a plasma display such as a liquid crystal display (LCD), an organic light emitting diode (OLED) display, or other types of display devices.

[0038] Furthermore, FIG. 1 is merely an example, and the technical solution in the embodiments of the present application is not limited to FIG. 1, for example, the technical solution of the present application can also be applied to one-sided video encoding or one-sided video decoding.

[0039] A video coding framework according to an embodiment of the present application will now be described.

[0040] 2 is an exemplary block diagram of a video encoder according to an embodiment of the present application. As can be appreciated, the video encoder 200 may be used to perform lossy compression on images, or may be used to perform lossless compression on images. The lossless compression may be visually lossless compression or mathematically lossless compression.

[0041] The video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2, or 4:4:4, where Y represents luminance (Luma), Cb (U) represents blue chrominance, and Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) and are used to describe hue and saturation. For example, in color format, 4:2:0 represents four luminance components and two chrominance components (YYYYCbCr) for every four pixels, 4:2:2 represents four luminance components and four chrominance components (YYYYCbCrCbCr) for every four pixels, and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0042] For example, the video encoder 200 reads video data and, for each frame of image in the video data, divides the image into a number of coding tree units (CTUs), which may also be referred to as "tree blocks," "largest coding units (LCUs)," or "coding tree blocks (CTBs)." Each CTU may be associated with a block of pixels of the same size in the image. Each pixel may correspond to one luminance (or luma) sample and two chrominance (or chroma) samples. Thus, each CTU may be associated with one luminance sampling block and two chrominance sampling blocks. The size of a CTU may be, for example, 128x128, 64x64, 32x32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding, which may be rectangular or square blocks. The CU is further divided into a prediction unit (PU) and a transform unit (TU), which may provide more flexibility during processing such as encoding, prediction, transformation, and separation. In one example, the CTU is divided into CUs in a quadtree manner, and the CU is divided into TUs and PUs in a quadtree manner.

[0043] Video encoders and decoders can support various PU sizes. Suppose the size of a CU is 2Nx2N. In this case, the video encoder and decoder can support PUs of 2Nx2N or NxN for intra-frame prediction, and can support symmetric PUs of 2Nx2N, 2NxN, Nx2N, NxN, or similar sizes for inter-frame prediction. The video encoder and decoder can also support asymmetric PUs of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0044] 2, the video encoder 200 may include a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / inverse quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image buffer 270, and an entropy coding unit 280. Note that the video encoder 200 may include more, fewer, or different functional assemblies.

[0045] Optionally, in this application, the current block may also be referred to as a current coding unit (CU), a current prediction unit (PU), etc. A prediction block may also be referred to as a predicted image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstruction block or an image reconstruction image block.

[0046] In some embodiments, prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is a fairly strong correlation between adjacent pixels in a frame of video, video codec techniques use intra-frame prediction methods to remove spatial redundancy between adjacent pixels. Also, because there is a fairly strong similarity between adjacent frames in video, video codec techniques use inter-frame prediction methods to remove temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0047] The inter-frame prediction unit 211 may be used for inter-frame prediction, which may include motion estimation and motion compensation. Motion estimation can find a reference block for a picture block by searching for a reference image in a list of reference images (pictures). Motion estimation can generate an index (index) indicating the reference block and a motion vector indicating the spatial displacement between the picture block and the reference block. Motion estimation can output the reference block index and the motion vector as motion information for the picture block. Motion compensation can obtain prediction information for the picture block based on the motion information for the picture block. Inter-frame prediction can refer to picture information of different frames. Inter-frame prediction uses the motion information to find a reference block from a reference frame and generate a prediction block based on the reference block, thereby eliminating temporal redundancy. The frames used for inter-frame prediction may be P frames and / or B frames, where P frames refer to forward predicted frames and B frames refer to bidirectionally predicted frames. Inter-frame prediction can use the motion information to find a reference block from a reference frame and generate a prediction block based on the reference block. The motion information includes a reference frame list where the reference frame is located, a reference frame index, and a motion vector. The motion vector may be for a whole pixel or a sub-pixel. If the motion vector is for a sub-pixel, interpolation filtering must be used in the reference frame to create the required sub-pixel block. Here, the whole pixel or sub-pixel block in the reference frame found based on the motion vector is called a reference block. Some techniques may directly call the reference block a prediction block, while some techniques may generate a prediction block by reprocessing based on the reference block. Generating a prediction block by reprocessing based on the reference block may be understood as using the reference block as a prediction block, and then reprocessing based on the prediction block to generate a new prediction block.

[0048] The intra-frame prediction unit 212 only refers to information of the image of the same frame and predicts pixel information in the current coding image block to remove spatial redundancy. The frame used for intra-frame prediction may be an I-frame.

[0049] There are multiple prediction modes for intraframe prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has eight angular prediction modes and one non-angular prediction mode, which has been expanded to 33 angular prediction modes and two non-angular prediction modes in H.265 / HEVC. The intraframe prediction modes used in HEVC include planar, DC, and 33 angular modes, for a total of 35 prediction modes. The intraframe modes used in VVC include planar, DC, and 65 angular modes, for a total of 67 prediction modes.

[0050] Furthermore, with the increase in angle modes, intra-frame prediction becomes more accurate and can better meet the needs of the development of high-resolution and super-resolution digital video.

[0051] The residual unit 220 may generate a residual block of a CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, the residual unit 220 may generate a residual block of a CU such that each sample in the residual block has a value equal to the difference between the pixel block of the CU and the corresponding sample in the prediction block of the PU of the CU.

[0052] The transform / quantization unit 230 may quantize the transform coefficients. The transform / quantization unit 230 may quantize the transform coefficients associated with the TUs of a CU based on a quantization parameter (QP) value (QP value) associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU. For example, the residual video signal may undergo a transform operation such as DFT or DCT to convert the signal into a transform domain, which is called a transform coefficient. The signal in the transform domain may then undergo a lossy quantization operation, in which certain information is lost, making the quantized signal more suitable for compressed representation. In some video coding standards, there may be multiple transform options available, and the encoding side may also need to select one of the transforms for the current CU and notify the decoding side. The granularity of quantization is typically determined by the quantization parameter (QP), with larger QP values ​​typically resulting in greater distortion and a lower bitrate since a wider range of coefficients are quantized to the same output; conversely, smaller QP values ​​typically resulting in less distortion and a higher bitrate since a narrower range of coefficients are quantized to the same output.

[0053] The inverse transform / inverse quantization unit 240 may reconstruct residual blocks from the quantized transform coefficients by applying inverse quantization and inverse transform, respectively, to the quantized transform coefficients.

[0054] Reconstruction unit 250 can generate a reconstructed image block associated with a TU by adding the samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210. By reconstructing the sample blocks of each TU of a CU in this manner, video encoder 200 can reconstruct the pixel blocks of the CU.

[0055] The loop filtering unit 260 processes pixels after inverse transformation and inverse quantization to capture distortion information and provide a better reference for subsequent coding pixels. For example, it can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with a CU. As can be seen from the above, a reconstructed decoded image can be obtained by performing inverse quantization, inverse transformation, and prediction compensation on a coded image. The reconstructed image differs from the original image due to the effects of quantization, resulting in distortion. Filtering the reconstructed image, such as by a deblocking filter (DBF), sample adaptive offset (SAO), or adaptive loop filter (ALF), can effectively reduce the distortion caused by quantization. Because the reconstructed image after these filters is used as a reference for subsequent coding images and to predict future signals, the above filtering operation is also referred to as loop filtering or in-coding loop filtering.

[0056] The decoded image buffer 270 can store the reconstructed pixel blocks. The inter-frame prediction unit 211 can perform inter-frame prediction on a PU of another image using a reference image including the reconstructed pixel blocks. Also, the intra-frame prediction unit 212 can perform intra-frame prediction on another PU in the same image as the CU using the reconstructed pixel blocks in the decoded image buffer 270.

[0057] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data. For example, the quantized transform domain signal may be statistically compressed and coded based on the frequency of occurrence of each value, ultimately generating a binary (0 or 1) compressed bitstream. Other information generated by the coding, such as the selected mode and motion vectors, may also need to be entropy coded to reduce the bitrate. In one example, the statistical coding that can be used is a lossless coding scheme, which can effectively reduce the bitrate required to represent a similar signal. Common statistical coding schemes include variable length coding (VLC) or content adaptive binary arithmetic coding (CABAC).

[0058] FIG. 3 is an exemplary block diagram of a video decoder according to an embodiment of the present application.

[0059] 3, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / inverse transform unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image buffer 360. Note that the video decoder 300 may include more, fewer, or different functional assemblies.

[0060] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream and extract syntax elements from the bitstream. As part of the parsed bitstream, the entropy decoding unit 310 may parse entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / inverse transform unit 330, the reconstruction unit 340, and the loop filtering unit 350 may decode video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0061] In some embodiments, the prediction unit 320 includes an intra-frame prediction unit 322 and an inter-frame prediction unit 321 .

[0062] The intra prediction unit 322 may perform intra prediction to generate a prediction block of the PU. The intra prediction unit 322 may generate the prediction block of the PU based on pixel blocks of spatially neighboring PUs using an intra prediction mode. The intra prediction unit 322 may further determine the intra prediction mode of the PU based on one or more syntax elements parsed from the bitstream.

[0063] The inter-frame prediction unit 321 may construct a first reference image list (list 0) and a second reference image list (list 1) based on syntax elements parsed from the bitstream. In addition, if inter-frame predictive coding is used for the PU, the entropy decoding unit 310 may analyze motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a prediction block for the PU based on one or more reference blocks for the PU.

[0064] The inverse quantization / inverse transform unit 330 may inverse quantize (i.e., dequantize) the transform coefficients associated with the TU. The inverse quantization / inverse transform unit 330 may determine the degree of quantization using a QP value associated with the CU of the TU.

[0065] After dequantizing the transform coefficients, the inverse quantization / inverse transform unit 330 may apply one or more inverse transforms to the dequantized transform coefficients to generate a residual block associated with the TU.

[0066] The reconstruction unit 340 may reconstruct pixel blocks of a CU using residual blocks associated with TUs of the CU and prediction blocks of PUs of the CU. For example, the reconstruction unit 340 may obtain reconstructed image blocks by adding samples of the residual blocks to corresponding samples of the prediction blocks to reconstruct pixel blocks of the CU.

[0067] The loop filtering unit 350 may perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with a CU.

[0068] The video decoder 300 can store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 can use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or can transmit the reconstructed image to a display device for display.

[0069] The basic flow of a video codec is as follows: on the encoding side, a frame of image is divided into blocks, and for a current block, a prediction unit 210 generates a predicted block of the current block using intraframe prediction or interframe prediction. A residual unit 220 calculates a residual block, i.e., the difference between the predicted block and the original block of the current block, based on the predicted block and the original block of the current block. The residual block may also be referred to as residual information. The residual block undergoes processes such as transformation and quantization by a transform / quantization unit 230, thereby removing information that is insensitive to the human eye, thereby eliminating visual redundancies. Optionally, the residual block before undergoing transformation and quantization by the transform / quantization unit 230 may be referred to as a time-domain residual block, and the time-domain residual block after undergoing transformation and quantization by the transform / quantization unit 230 may be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 receives the quantized transformation coefficients output from the transform / quantization unit 230, performs entropy coding on the quantized transformation coefficients, and outputs a bitstream. For example, the entropy coding unit 280 can remove character redundancies based on a target context model and probability information of the binary bitstream.

[0070] On the decoding side, the entropy decoding unit 310 can analyze the bitstream to obtain prediction information, a quantization coefficient matrix, etc., for the current block. The prediction unit 320 can generate a prediction block for the current block using intraframe or interframe prediction based on the prediction information. The inverse quantization / inverse transform unit 330 can use the quantization coefficient matrix obtained from the bitstream to perform inverse quantization and inverse transform on the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 can add the prediction block and the residual block to obtain a reconstructed block. The reconstructed block can form a reconstructed image, and the loop filtering unit 350 can perform loop filtering on the reconstructed image based on an image or a block to obtain a decoded image. On the encoding side, similar operations as on the decoding side must be performed to obtain a decoded image. The decoded image may be called a reconstructed image, and the reconstructed image can be used as a reference frame for interframe prediction for subsequent frames.

[0071] In addition, the block division information determined on the encoding side, and mode information and parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., can be carried in the bitstream when necessary. The decoding side analyzes the bitstream and performs analysis based on existing information to determine the same block division information and mode information and parameter information such as prediction, transform, quantization, entropy coding, and loop filtering, etc., as the encoding side, thereby ensuring that the decoded image obtained on the encoding side is the same as the decoded image obtained on the decoding side.

[0072] The above is a basic flow based on a video codec under a block hybrid coding framework, and as technology develops, some modules or steps of the framework or the flow may be optimized, and the present application applies to the basic flow based on a video codec under a block hybrid coding framework, but is not limited to the framework and the flow.

[0073] In a conventional hybrid coding framework, the image of each frame in a video is often first divided into units of a predetermined size, and then the subsequent codec process is performed. As shown in Figure 4, the maximal coding unit (CTU) is the basic coding unit in the hybrid coding framework, and it usually includes two parts: luma Y and chroma UV. Because the characteristics of the U and V components of chroma are relatively close, the coding results of U and V can usually be obtained by sequentially processing U first and then V using the same coding parameters.

[0074] In conventional hybrid coding frameworks, traditional loop filters are used to reduce distortion in the reconstructed image, improving the quality of the reconstructed image and restoring the encoded reconstructed image to the original image. However, traditional loop filters are based on manual design, making it difficult to effectively reduce distortion in the reconstructed image and leaving significant room for optimization. Due to the excellent performance of deep learning tools in image processing, deep learning-based loop filters have increasingly been applied to loop filter modules.

[0075] The main technology involved in this application is a neural network loop filter (NNLF). As shown in Figure 5, an unfiltered image can be input to a trained filter to obtain an enhanced image after filtering.

[0076] During the training process, neural networks typically use a loss function to constrain the filtered image so that it is restored to the original image as closely as possible. The loss function evaluates the difference between the filtered value and the actual value; the larger the loss value, the greater the difference. The goal of training is to reduce the loss. In deep learning-based coding tools, for example, commonly used loss functions include the L1 norm loss function, the L2 norm loss function, and the smooth L1 loss function.

[0077] In the actual training process, neural network filters often adopt data enhancement methods such as rotation, inversion, and downsampling to expand the training set samples and improve the model generalization ability. However, in the actual application or testing process, neural network filters do not perform operations such as inversion, rotation, and downsampling on the input images to be processed, but send them directly to the network.

[0078] In the development process of a neural network filter, the neural network filter is first trained on a training set, and the network parameters are fixed upon completion of training. The neural network filter with the fixed parameters is then used to test on a test set to verify the coding performance of the tool. Ideally, if the data distribution of the test set perfectly matches that of the training set, the neural network filter can achieve optimal coding performance on the test set. However, in actual neural network-based video coding, the image content of the test set is not completely identical to that of the training set, and there will be some differences in the data distribution characteristics between the two. As shown in Figure 6, the data distribution characteristics of the image content in the training set are Data Distribution Characteristic A, Data Distribution Characteristic B, Data Distribution Characteristic C, and Data Distribution Characteristic D, etc., while the data distribution characteristics of the image content in the test set are Data Distribution Characteristic B, Data Distribution Characteristic E, Data Distribution Characteristic F, and Data Distribution Characteristic G, etc., respectively. Since both the training set and the test set contain image content with data distribution characteristic B, the neural network filter can achieve a relatively ideal filtering effect for the information to be filtered of image content with data distribution characteristic B. However, for the information to be filtered of image content with data distribution characteristic E, data distribution characteristic F, and data distribution characteristic G, since the training set does not contain these data distribution characteristics, the neural network filter cannot properly learn these data distribution characteristics, and as a result, the filtering effect of the image content with these data distribution characteristics is not ideal.

[0079] The research of this application has found that the data distribution characteristics of the image content can be changed by performing data rearrangement, such as rotation, flipping, etc., on the image content. For example, after rotating the image content in the test set shown in Figure 6A, the test set shown in Figure 6B can be obtained, and the data distribution characteristics of the test set can be closer to those of the training set compared with the test set shown in Figure 6A.

[0080] Based on this, to solve this technical problem, in the embodiments of the present application, data rearrangement is performed on the information to be filtered input by the neural network filter, so that the data distribution characteristics of the information to be filtered are closer to the data distribution characteristics of the training set, thereby fully exerting the generalization effect of the neural network filter and improving the image filtering effect and the image encoding and decoding performance. Specifically, a first data processing method for the information to be filtered is determined from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial region of the reconstructed image, M is a positive integer; when the first data processing method is one of the M data rearrangement methods, data rearrangement is performed on the reconstructed image block in the information to be filtered according to the first data processing method to obtain rearranged information to be filtered, and a filtering process is performed on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and data reverse rearrangement is performed on the filtered image of the rearranged information to be filtered according to the first data processing method to obtain a filtered image block of the reconstructed image block.In other words, in the present application, before filtering the information to be filtered, it is first determined whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to perform data rearrangement on the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered match or approach the data distribution characteristics of the training data. This can improve the filtering effect of the neural network filter on the rearranged information to be filtered, and thereby improve the image filtering effect and the image encoding and decoding performance.

[0081] The technical solutions of the embodiments of the present application will be described in detail below through several examples, and some of the following examples may be combined, and some of the following examples may omit the description of the same or similar concepts or processes.

[0082] First, the image filtering method provided in the embodiment of this application will be introduced by taking the decoding side as an example.

[0083] 7 is a flowchart of a neural network-based image filtering method provided in an embodiment of the present application. The embodiment of the present application may be applied to the decoder or decoding device shown in FIG. 1 or FIG. 3, or may be applied to other electronic devices. For convenience of explanation, the following description will be given assuming that the execution entity is the decoding side. As shown in FIG. 7, the method in the embodiment of the present application includes the following steps:

[0084] S101: A first data processing method for information to be filtered is determined from among M+1 data processing methods.

[0085] The method in the embodiment of the present application may be applied to video decoding schemes and may also be applied to video post-processing.

[0086] In some embodiments, when applied to a video decoding scheme, the encoding side first divides the current image into coding blocks when encoding the current image, and sequentially performs block coding on the coding blocks as coding units. For example, for a current block to be encoded in the current image, the encoding side first obtains a predicted value of the current block using inter-frame and / or intra-frame prediction. Next, the encoding side obtains a residual value of the current block based on the predicted value of the current block and the current block. The encoding side performs a transform on the residual value of the current block to obtain transform coefficients. In one example, the encoding side does not perform quantization on the transform coefficients of the current block, but directly codes the transform coefficients to obtain a bitstream. In another example, the encoding side quantizes the transform coefficients of the current block to obtain quantized coefficients, and then codes the quantized coefficients to obtain a bitstream.

[0087] In the encoding process, as shown in Figure 2, the encoding side further performs an inverse transform on the transform coefficients to obtain residual values, and then adds the residual values ​​and predicted values ​​to obtain the reconstructed values ​​of the current block. Through the above steps, the reconstructed values ​​of each coding block in the current image can be obtained, and these reconstructed values ​​can form a reconstructed image of the current image. Then, to further improve the quality of the reconstructed image, filtering is performed on the reconstructed image to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoding buffer for prediction of subsequent images.

[0088] As shown in FIG. 3, for each block in the current image waiting to be decoded, e.g., the current block, the decoding side obtains a bitstream and then decodes the bitstream to obtain the transform coefficients of the current block. In one example, when the encoding side quantizes the transform coefficients before encoding, the decoding side decodes the bitstream to obtain the quantized coefficients of the current block and then performs inverse quantization on the quantized coefficients to obtain the transform coefficients of the current image. Next, the decoding side performs inverse transform on the transform coefficients of the current block to obtain the residual values ​​of the current block. Also, the decoding side performs prediction using inter-frame and / or intra-frame prediction to obtain the predicted value of the current block. In this way, the predicted value and the residual value of the current block are added to obtain the reconstructed value of the current block. The decoding side can determine the reconstructed values ​​of each block in the current image by decoding based on the above steps, and these reconstructed values ​​can form a reconstructed image of the current image. Next, to further improve the quality of the reconstructed image, the decoding side filters the reconstructed image to obtain a decoded image of the current image. In one example, the decoding side may store the decoded image in a decoding buffer for prediction of subsequent images. In one example, the decoding side may output the decoded image to a display for display.

[0089] The to-be-filtered information in the embodiment of the present application includes reconstructed image blocks.

[0090] In some embodiments, the image filtering method provided in the embodiments of the present application can be used to filter at least one frame of image in a video, i.e., the above-mentioned current image is an image in a video.

[0091] In some embodiments, the image filtering method provided in the embodiments of the present application can be used to decode a single image, i.e., the current image is a single image, for example, an image generated by an electronic device.

[0092] In the embodiment of the present application, a neural network filter is used at the decoding side to filter the reconstructed image.

[0093] In the embodiments of the present application, the data distribution characteristics may be referred to as data distribution features, and mainly refer to the concentration tendency of the data distribution of an image, the degree of discreteness of the data distribution, etc. When the image contents of two images are different, the data distribution characteristics may also be different. For example, if one image shows Animal A performing Action B and the other image shows Animal A performing Action C, the data distribution characteristics of the two images may be different. Therefore, in the embodiments of the present application, when the image content of the reconstructed image block is different from the image content of the training data, the data distribution characteristics of the reconstructed image block may also be different from the data distribution characteristics of the training data. Thus, when directly using a neural network filter trained based on the training data to filter the reconstructed image block, non-ideal filtering results may occur. Therefore, in the embodiments of the present application, before filtering the information to be filtered using a neural network filter, it is first determined whether data rearrangement is required for the reconstructed image in the information to be filtered.

[0094] In the embodiments of the present application, when filtering is performed on a reconstructed image, the entire reconstructed image may be used as a filtering unit to perform filtering directly, or the reconstructed image may be divided into multiple blocks and each block may be used as a filtering unit to perform filtering. For convenience of explanation, in the embodiments of the present application, a reconstructed image block is used as a filtering unit, and the reconstructed image block may be one reconstructed image (i.e., one frame) or a partial region of the reconstructed image.

[0095] In some embodiments, when the reconstructed image block is a partial region of the reconstructed image, some examples of division of the reconstructed image block include, but are not limited to, the following.

[0096] In one possible implementation, the above-mentioned reconstructed image block includes at least one CTU of the reconstructed image, in other words, the division is performed such that at least one CTU of the reconstructed image becomes one reconstructed image block.

[0097] In some examples, as shown in FIG. 8A, the division is performed such that a reconstructed image block includes one CTU of the reconstructed image, ie, one CTU of the reconstructed image becomes one reconstructed image block.

[0098] In another example, as shown in FIG. 8B, a reconstructed image block includes four CTUs of the reconstructed image, that is, the division is performed such that four CTUs of the reconstructed image become one reconstructed image block.

[0099] In one example, the division may further be performed such that a plurality of CTUs, such as two CTUs, three CTUs, etc., of the reconstructed image constitute one reconstructed image block. The plurality of CTUs may be a plurality of CTUs in the horizontal direction or a plurality of CTUs in the vertical direction. Optionally, the plurality of CTUs may be adjacent, non-adjacent, partially adjacent, or partially non-adjacent.

[0100] In another possible implementation, the reconstructed image blocks to be filtered are predetermined image regions of the reconstructed image, i.e., the division is performed such that one predetermined image region of the reconstructed image becomes one reconstructed image block.

[0101] The embodiments of the present application do not limit the specific shape and size of the predetermined image area.

[0102] In one example, the predetermined image region includes at least one incomplete CTU in the reconstructed image, i.e., the division is performed so that at least one incomplete CTU in the reconstructed image becomes one reconstructed image block. In the example shown in Figure 8C, the division is performed so that one CTU and three incomplete CTUs in the reconstructed image become one reconstructed image block, and for example, as shown in Figure 8D, the division is performed so that four incomplete CTUs in the reconstructed image become one reconstructed image block.

[0103] In some embodiments, the predetermined image region is a fixed region, for example, each time filtering is performed, the predetermined image region is used to obtain the current reconstructed image block to be filtered in the reconstructed image, and the size and shape of the image block input to the neural network filter are the same each time, and all are the predetermined image region.

[0104] In some embodiments, the predetermined image region is variable. For example, during the first filtering, one reconstructed image block to be filtered in the reconstructed image is obtained according to the first predetermined image region, and input to the neural network filter as an input image block for filtering. During the second filtering, one reconstructed image block to be filtered in the reconstructed image is obtained according to the second predetermined image region, and input to the neural network filter as an input image block for filtering. Based on this, an inference is made. In one example of this embodiment, the decoding side can divide the reconstructed image into a number of reconstructed image blocks to be filtered, and the shapes and sizes of the reconstructed image blocks to be filtered can be the same or different, or can be partially the same or partially different.

[0105] The process for determining the first data processing method for the information to be filtered will be described below.

[0106] In an embodiment of the present application, the above-mentioned first data processing method is one of M+1 data processing methods, and the M+1 data processing methods include a method that does not rearrange data and M data rearrangement methods.

[0107] The method of not rearranging data refers to not rearranging the data to be filtered, but directly inputting the filtered data to the neural network filter.

[0108] The data rearrangement method refers to rearranging some or all of the information in the information to be filtered according to the data rearrangement method, and then inputting the data to be filtered into a neural network filter for filtering, where some or all of the information in the information to be filtered includes reconstructed image blocks, in other words, in the embodiments of the present application, when data rearrangement is performed on the information to be filtered, it at least includes data rearrangement on the reconstructed image blocks in the information to be filtered.

[0109] In the embodiments of the present application, the specific type of data rearrangement method is not limited.

[0110] In some embodiments, the M data rearrangement schemes include at least one of the following: a rotation scheme, a horizontal flip and then rotate scheme, a vertical flip and then rotate scheme, a diagonal flip and then rotate scheme, a downsampling and then rotate scheme, and an upsampling and then rotate scheme.

[0111] The rotation methods include N*90° left rotation and N*90° right rotation, where N is a positive integer, for example, rotating information such as a reconstructed image block left or right by 90°, 180°, 270°, etc.

[0112] The methods of performing horizontal flip and then rotating include horizontal flip and then rotating N*90° to the left, and horizontal flip and then rotating N*90° to the right, where N is a positive integer. For example, information such as a reconstructed image block is horizontally flipped, and then rotated 90°, 180°, 270°, etc. to the left or right.

[0113] The vertical flip and then rotation methods include vertical flip and then *90° left rotation, and vertical flip and then N*90° right rotation, where N is a positive integer. For example, information such as a reconstructed image block is vertically flipped, and then rotated 90°, 180°, 270°, etc. left or right.

[0114] The diagonal flip followed by rotation includes diagonal flip followed by N*90° left rotation and diagonal flip followed by N*90° right rotation, where N is a positive integer. For example, information such as a reconstructed image block is diagonally flipped and then rotated 90°, 180°, 270°, etc. to the left or right.

[0115] The downsampling followed by rotation includes downsampling followed by N*90° left rotation and downsampling followed by N*90° right rotation, where N is a positive integer. For example, after downsampling information such as a reconstructed image block, the image block is rotated left or right by 90°, 180°, 270°, etc.

[0116] The upsampling followed by rotation includes upsampling followed by N*90° left rotation and upsampling followed by N*90° right rotation, where N is a positive integer. For example, after upsampling information such as a reconstructed image block, the image is rotated left or right by 90°, 180°, 270°, etc.

[0117] In addition, the M data rearrangement methods in the embodiments of the present application may include at least one of the above-mentioned data rearrangement methods, i.e., the rotation method and the method of rotating after performing any other pre-processing operation, as well as other data rearrangement methods, and the embodiments of the present application are not limited thereto.

[0118] In one example, the M+1 data processing methods in the embodiment of the present application are as shown in Table 1 below.

[0119] [Table 1] In the embodiment of the present application, before filtering is performed on the information to be filtered on the decoding side, a first data processing method for the information to be filtered is first determined from among M+1 data processing methods.

[0120] In the embodiment of the present application, there is no limitation on the specific method for determining the first data processing method for the information to be filtered from among M+1 data processing methods on the decoding side.

[0121] Method 1: When the above-mentioned reconstructed image block is a part of the reconstructed image, a reference image block of the reconstructed image block is determined, and the data processing method of the reference image block is inherited.

[0122] For example, when the prediction method corresponding to the reconstructed image block is inter-frame prediction, a reference image block of the reconstructed image block is determined from the reference image of the current image, and the data processing method of the reference image block is determined as the first data processing method of the reconstructed image block.

[0123] Also, for example, when the prediction method corresponding to the reconstructed image block is intra-frame prediction, a reference image block for the reconstructed image block is determined from the reconstructed image, and the data processing method of the reference image block is determined as the first data processing method for the reconstructed image block.

[0124] Method 2: When the reconstructed image block is a part of the reconstructed image and is not an image block in the first row of the reconstructed image, in other words, when a filtered area is included around the reconstructed image block, the first data processing method can be determined from among M+1 data processing methods based on the filtered area around the reconstructed image block.

[0125] 9, the filtered area around the reconstructed image block includes the areas above and to the left of the reconstructed image block, and may optionally further include the areas above and to the right of the reconstructed image block. Therefore, based on some or all of these areas, one data processing method may be selected from M+1 data processing methods as the first data processing method. Specifically, the decoding side uses these M+1 data processing methods to process the filtered area A, respectively, and selects the data processing method with the smallest filtering cost as the first data processing method. Among them, the filtered area A includes at least one of the areas above, to the left, above and to the right of the reconstructed image block.

[0126] Specifically, the decoding side first does not rearrange the data in the filtered region A, but directly inputs it into the neural network filter for filtering, thereby obtaining a filtered image block 0 without rearranging the data. The filtered image block 0 is compared with the filtered region A to determine the filtering cost 0 corresponding to the method without rearranging the data.

[0127] The embodiments of the present application are not limited to the specific calculation method of the above-mentioned filtering cost. For example, the above-mentioned filtering cost may be a rate-distortion cost (RDO), or may be an approximate cost, such as SSD, STAD, SAD, etc.

[0128] In one possible implementation, for each of the M data rearrangement schemes, the decoding side uses the data rearrangement scheme to perform data rearrangement on the filtered region A, and then inputs the data-rearranged filtered region A into a neural network filter for filtering to obtain a filtered image block under the data rearrangement scheme. The decoding side uses the data rearrangement scheme to perform data inversion on the filtered image block, and then compares it with the filtered region A to determine the corresponding filtering cost under the data rearrangement scheme. Referring to the above example, the decoding side can determine the filtering cost corresponding to each of the M data rearrangement schemes. In this way, the filtering cost corresponding to the scheme without data rearrangement can be compared with the filtering costs corresponding to each of the M data rearrangement schemes, and the one with the smallest filtering cost can be selected as the first data processing scheme.

[0129] For example, as shown in FIG. 10, when the M data rearrangement methods include a 90° left rotation and a 90° right rotation, the decoding side directly inputs the filtered region A into the neural network filter to obtain a filtered image block 0 without data rearrangement. The filtered image block 0 without data rearrangement is compared with the filtered region A to calculate the filtering cost corresponding to the method without data rearrangement. Similarly, the decoding side performs a 90° left rotation on the filtered region A and then inputs it into the neural network filter to obtain a filtered image block rotated 90° to the left. Next, the decoding side performs a 90° right rotation on the filtered image block rotated 90° to the left to obtain a filtered image block with a 90° left rotation method. The filtered image block with a 90° left rotation method is compared with the filtered region A to calculate the filtering cost corresponding to the 90° left rotation method. Similarly, the decoding side rotates the filtered region A by 90° right and then inputs the rotated region into a neural network filter to obtain a filtered image block after 90° right rotation. Next, the filtered image block after 90° right rotation is rotated by 90° left to obtain a filtered image block under the 90° right rotation scheme. The filtered image block under the 90° right rotation scheme is compared with the filtered region A to calculate the filtering cost corresponding to the 90° right rotation scheme. The decoding side selects the first data processing scheme from the non-rearrangement scheme, the 90° right rotation scheme, and the 90° left rotation scheme, which has the smallest filtering cost.

[0130] In another possible implementation, the decoding side calculates the filtering costs for M data rearrangement schemes in a predetermined order. Specifically, the decoding side first performs data rearrangement on the filtered region A using a first data rearrangement scheme among the M data rearrangement schemes, and then inputs the data-rearranged filtered region A into a neural network filter for filtering to obtain a first filtered image block. Next, the decoding side performs reverse data rearrangement on the first filtered image block using the first data rearrangement scheme to obtain a filtered image block 1 under the first data rearrangement scheme. The decoding side compares the filtered image block 1 with the filtered region A to determine a filtering cost 1 corresponding to the first data rearrangement scheme. Next, the decoding side compares the filtering cost 1 with the above-mentioned filtering cost 0. If the filtering cost 1 is smaller than the filtering cost 0, the decoding side determines the first data rearrangement scheme as the first data rearrangement scheme and does not try subsequent data rearrangement schemes. If filtering cost 1 is greater than or equal to filtering cost 0, the decoding side uses a second data rearrangement method among the M data rearrangement methods to rearrange data on the filtered area A, calculates a filtering cost 2 corresponding to the second data rearrangement method, and compares the filtering cost 2 with filtering cost 0. If filtering cost 2 is less than filtering cost 0, the decoding side determines the second data rearrangement method as the first data processing method. If filtering cost 2 is greater than or equal to filtering cost 0, the decoding side uses a third data rearrangement method among the M data rearrangement methods to rearrange data on the filtered area A, and by analogy, checks all M data rearrangement methods and ends. If the filtering costs corresponding to all M data rearrangement methods are greater than or equal to filtering cost 0 corresponding to the method without data rearrangement, the decoding side determines the method without data rearrangement as the first data processing method.

[0131] Method 3: The encoding side determines the first data processing method and then indicates the first data processing method to the decoding side. At this time, the above-mentioned S101 includes the following step S101-A.

[0132] S101-A: On the decoding side, the bitstream is decoded to determine a first data processing method from among M+1 data processing methods.

[0133] In the embodiment of the present application, the encoding side directly instructs the decoding side to use the first data processing method, and the decoding side decodes the bitstream to obtain the first data processing method, which does not require additional calculations, thereby reducing the computational complexity on the decoding side and improving decoding performance.

[0134] In some embodiments, the first data processing method is one of the M+1 data processing methods with the smallest filtering cost, for example, the encoding side determines the filtering cost corresponding to each of the M+1 data processing methods, and then selects the one of the M+1 data processing methods with the smallest filtering cost as the first data processing method.

[0135] In some embodiments, the first data processing method is any one of the M data rearrangement methods whose filtering cost is smaller than that of the method without data rearrangement. For example, the encoding side first determines the filtering cost corresponding to the method without data rearrangement, then selects the first data rearrangement method among the M data rearrangement methods according to a predetermined order, determines the filtering cost corresponding to the selected data rearrangement method, and when the filtering cost corresponding to the selected data rearrangement method is smaller than that of the method without data rearrangement, stops trying other data rearrangement methods, and directly selects the selected data rearrangement method as the first data processing method, thereby improving the image filtering effect, reducing the amount of data processing, and improving encoding performance.

[0136] In the embodiment of the present application, the implementation manner of the above S101-A includes, but is not limited to, the following several manners:

[0137] Method 1: The above-mentioned S101-A includes the following S101-A-a1 and S101-A-a2.

[0138] S101-A-a1: At the decoding side, the bitstream is decoded to obtain a first flag; and S101-A-a2: On the decoding side, a first data processing method is obtained from among M+1 data processing methods based on the first flag.

[0139] In method 1, the encoding side writes a first flag into the bitstream, and the first flag is used to indicate whether data rearrangement should be performed on the information to be filtered when filtering is performed. For example, when the encoding side determines that data rearrangement should be performed on the information to be filtered when filtering is performed, the encoding side sets the value of the first flag to yes (e.g., 1) and then writes it into the bitstream. When the encoding side determines that data rearrangement should not be performed on the information to be filtered when filtering is performed, the encoding side sets the value of the first flag to no (e.g., 0) and then writes it into the bitstream.

[0140] In this way, the decoding side can decode the bitstream to obtain the first flag, and then determine the first data processing method based on the first flag.

[0141] For example, if the first flag indicates that data rearrangement is not to be performed on the information to be filtered when filtering is performed (for example, a value of the first flag of 0 indicates that data rearrangement is not to be performed on the information to be filtered when filtering is performed), the decoding side determines a method of not rearranging data as the first data processing method.

[0142] Furthermore, for example, when the first flag indicates that data rearrangement is to be performed on the information to be filtered when filtering is performed (for example, the value of the first flag being 1 indicates that data rearrangement is to be performed on the information to be filtered when filtering is performed), the decoding side continues to decode the bitstream to obtain the second flag, which is used to indicate the index of the data rearrangement method for the information to be filtered. The decoding side can determine the first data processing method from among the M data rearrangement methods based on the second flag.

[0143] In the embodiment of the present application, the specific expression formats of the first flag and the second flag are not limited.

[0144] In one example, the correspondence between the values ​​of the first flag and the second flag and the data processing method is as shown in Table 2 below.

[0145] [Table 2] As shown in Table 2, the decoding side first decodes the bitstream to obtain a first flag. When the value of the first flag is 0, it indicates that data rearrangement is not required for the information to be filtered when filtering, and in this case, the decoding side determines a method of not rearranging data as the first data processing method. When the value of the first flag is 1, it indicates that data rearrangement is required for the information to be filtered when filtering, and in this case, the bitstream is further decoded to obtain a second flag, and the first data processing method is determined based on the value of the second flag. For example, when the value of the second flag is 00, it indicates that the first data processing method is a 90° left rotation method.

[0146] In this method 1, the encoding side specifies the first data processing method by the first flag and the second flag.

[0147] Method 2: In this method, the encoding side may replace the first and second flags with one flag, i.e., the third flag. In other words, the third flag can indicate both whether to perform data rearrangement when filtering the information to be filtered, and the specific data rearrangement method. In this case, the above-mentioned S101-A includes the following S101-A-b1 and S101-A-b2.

[0148] S101-A-b1: At the decoding side, the bitstream is decoded to obtain a third flag, and the third flag is used to indicate the first data processing method; and S101-A-b2: On the decoding side, a first data processing method is determined from among M+1 data processing methods based on the third flag.

[0149] In method 2, the encoding side directly indicates the index of the first data processing method to the decoding side using the third flag. Specifically, the encoding side determines the first data processing method for the information to be filtered from among M+1 data processing methods, and then indicates the first data processing method to the decoding side using the third flag. For example, when the first data processing method is a method that does not rearrange data, the encoding side sets the third flag to a first numerical value (e.g., 0); when the first data processing method is a data rearrangement method, the encoding side sets the third flag to another numerical value; for example, when the first data processing method is a 90° left rotation, the value of the third flag is set to 100.

[0150] In this way, the decoding side decodes the bitstream to obtain the third flag, and can then determine the first data processing method from among M+1 data processing methods based on the value of the third flag.

[0151] In the embodiment of the present application, the specific expression format of the third flag is not limited.

[0152] In one example, the correspondence between the third flag value and the data processing method is as shown in Table 3 below.

[0153] [Table 3] As shown in Table 3, the decoding side first decodes the bitstream to obtain the third flag. When the value of the third flag is 0, this indicates that data rearrangement is not required for the information to be filtered when filtering is performed. In this case, the decoding side determines a method of not rearranging data as the first data processing method. When the value of the third flag is 100, this indicates that data rearrangement is required for the information to be filtered when filtering is performed, and the first data processing method is a method of rotating 90 degrees left. When the value of the third flag is 110, this indicates that the first data processing method is a method of performing horizontal flipping and then rotating 90 degrees left. In this way, the decoding side can obtain the first data processing method by looking up Table 3 above based on the value of the third flag.

[0154] When the reconstructed image block is a single reconstructed image, the first flag, the second flag, and the third flag may be picture-level flags. When the reconstructed image block is a partial area of ​​a single reconstructed image, the first flag, the second flag, and the third flag may be block-level flags.

[0155] The above has described a specific process for determining the first data processing method for information to be filtered on the decoding side. After determining the first data processing method on the decoding side, the decoding side executes the following step S102.

[0156] S102: When the first data processing method is one of M data rearrangement methods, perform data rearrangement on the reconstructed image block based on the first data processing method to obtain rearranged information to be filtered, and perform filtering processing on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered.

[0157] The decoding side determines a first data processing method based on the above steps, and when the first data processing method is a method that does not rearrange data, the decoding side does not perform data rearrangement on the information to be filtered, but performs filtering processing on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0158] When the above-mentioned first data processing method is one of M data rearrangement methods, it means that before the information to be filtered is input into the neural network filter, data rearrangement needs to be performed first.

[0159] In some embodiments, the to-be-filtered information includes only reconstructed image blocks.

[0160] In some embodiments, the to-be-filtered information includes, in addition to the reconstructed image block, at least one of a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, a frame type (slice type) of the reconstructed image block, and quantization parameter information. Optionally, the quantization parameter information includes a slice quantization parameter (QP) and a base quantization parameter (QP).

[0161] In the embodiment of the present application, when the first data processing method is the data rearrangement method, the decoding side performs data rearrangement on the information to be filtered, and at least the following several methods are included.

[0162] First method: the decoding side performs data rearrangement only on the reconstructed image block in the filtering information, and does not perform data rearrangement on other information in the filtering information. For example, the filtering information includes information such as a reconstructed image block, a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, the frame type of the reconstructed image block, and quantization parameter information. The decoding side adopts a first data processing method to perform data rearrangement on the reconstructed image block, but does not perform data rearrangement on information such as a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, the frame type of the reconstructed image block, and quantization parameter information. Since the predicted image block corresponding to the reconstructed image block, the boundary intensity image block, the frame type of the reconstructed image block, and quantization parameter information are all auxiliary information of the reconstructed image block, data rearrangement does not need to be performed on this information, which reduces the data rearrangement load on the decoding side and improves the decoding efficiency on the decoding side.

[0163] Second method: On the decoding side, data rearrangement is performed on the reconstructed image block based on the first data processing method, and data rearrangement is performed on at least one of the predicted image block, the boundary strength image block, the frame type, and the quantization parameter information based on the first data processing method, thereby obtaining rearranged filtering data.

[0164] In one example of the second method, the decoding side performs data rearrangement on all information in the information to be filtered based on a first data processing method, for example, the first data processing method is a 90° left rotation, and the information to be filtered includes information such as a reconstructed image block, a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type, and quantization parameter information, in this case, the decoding side performs a 90° left rotation on all information such as the reconstructed image block, the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type, and the quantization parameter information, to obtain rearranged filtering data.

[0165] In one example of the second method, the decoding side performs data rearrangement on the reconstructed image block based on the first data processing method, performs data rearrangement on the predicted image block based on the first data processing method, and does not rearrange information such as boundary strength image block, frame type, and quantization parameter information, thereby obtaining rearranged filtering data.

[0166] In one example of the second method, the decoding side performs data rearrangement on the reconstructed image block based on the first data processing method, performs data rearrangement on the boundary intensity image block based on the first data processing method, and does not rearrange information such as the predicted image block, frame type, and quantization parameter information, thereby obtaining rearranged filtering data.

[0167] Third method: On the decoding side, data rearrangement is performed on the reconstructed image block based on the first data processing method, and data rearrangement is performed on at least one of the predicted image block and the boundary intensity image block based on the first data processing method, and data rearrangement on at least one of the frame type and quantization parameter information is skipped to obtain rearranged filtering data.

[0168] In one example of the third method, when the filtering waiting information includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, frame type and quantization parameter information, the decoding side performs data rearrangement for the reconstructed image block based on the first data processing method, performs data rearrangement for the predicted image block and boundary strength image block based on the first data processing method, and skips data rearrangement for the frame type and quantization parameter information (i.e., does not perform data rearrangement), thereby obtaining rearranged filtered data.

[0169] In one example of the third method, when the filtering waiting information includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type, and quantization parameter information, the decoding side performs data rearrangement for the reconstructed image block, the predicted image block, the boundary strength image block, and the frame type based on the first data processing method, and skips data rearrangement for the quantization parameter information (i.e., does not perform data rearrangement), thereby obtaining the rearranged filtering data.

[0170] In one example of the third method, when the filtering waiting information includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type, and quantization parameter information, the decoding side performs data rearrangement for the reconstructed image block, the predicted image block, the boundary strength image block, and the quantization parameter information based on the first data processing method, and skips data rearrangement for the frame type (i.e., does not perform data rearrangement), thereby obtaining rearranged filtered data.

[0171] In one example of the third method, when the filtering waiting information includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type, and quantization parameter information, the decoding side performs data rearrangement for the reconstructed image block and the predicted image block based on the first data processing method, and skips data rearrangement for the boundary strength image block, the quantization parameter information, and the frame type (i.e., does not perform data rearrangement), thereby obtaining the rearranged filtering data.

[0172] The decoding side adopts a first data processing method to perform data rearrangement on the reconstructed image blocks in the information to be filtered, or performs data rearrangement on the reconstructed image blocks and other information in the filtering information, thereby obtaining rearranged filtering information. Next, the decoding side inputs the rearranged filtering information into a neural network filter, and the neural network filter performs filtering processing on the rearranged information to be filtered, thereby obtaining the filtered image blocks in the rearranged information to be filtered.

[0173] In an embodiment of the present application, before filtering the information to be filtered on the decoding side, it is first determined whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, a first data processing method is used to perform data rearrangement on information such as reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered coincide with or approach the data distribution characteristics of the training data. This can improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and the performance of image encoding and decoding.

[0174] S103: Perform data reverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block.

[0175] According to the above steps, the decoding side performs data rearrangement on information such as the reconstructed image block in the information to be filtered, so that the neural network filter filters the rearranged information to be filtered, and what is obtained is a filtered image of the rearranged information to be filtered, and the data distribution characteristics of the filtered image of the rearranged information to be filtered are inconsistent with the data distribution characteristics of the information to be filtered. Therefore, in the embodiment of the present application, the decoding side uses a first data processing method to perform data rearrangement on the reconstructed image block in the information to be filtered, so as to obtain a filtered image of the rearranged information to be filtered, and then further uses the first data processing method to perform data reverse rearrangement on the filtered image of the rearranged information to be filtered, so as to obtain a filtered image block of the reconstructed image block.

[0176] For example, the first data processing method described above is a 90° left rotation. In this case, the decoding side needs to perform a 90° right rotation on the filtered image of the rearranged filtering-waiting information to obtain the filtered image block of the reconstructed image block.

[0177] Also, for example, the first data processing method described above is a 90° right rotation, and in this case, on the decoding side, it is necessary to perform a 90° left rotation on the filtered image of the rearranged filtering-waiting information to obtain the filtered image block of the reconstructed image block.

[0178] In some embodiments, the reconstructed image block includes at least one of a Y component reconstructed image block, a U component reconstructed image block, and a V component reconstructed image block. In other words, the adaptive data rearrangement scheme in the embodiments of the present application can be applied to at least one of the luma component Y and the chroma components U and V.

[0179] The filtering process of one reconstructed image block in a reconstructed image has been introduced above, but for the filtering process of other reconstructed image blocks in the reconstructed image that are waiting to be filtered, the filtering process of the reconstructed image block described above can be referred to, and ultimately a reconstructed image after filtering is obtained.

[0180] In some embodiments, the image filtering method in the embodiments of the present application can be used in a video decoder, for example, by using loop filtering to store the filtered reconstructed image in a decoding buffer for subsequent image decoding, and the filter output affects the video decoding. Because the reconstructed image provided in the embodiments of the present application has high filtering quality, the video decoding effect can be improved when subsequent decoding is performed based on the reconstructed image with relatively good quality.

[0181] In some embodiments, the image filtering method in the embodiments of the present application can be used in video post-processing products, for example, applied on the display side, i.e., the filtered reconstructed image is directly displayed without being stored in a decoding buffer, and the output of the filter does not affect video decoding.

[0182] In some embodiments, the image decoding method in the embodiments of the present application can further be used in any one module that uses a neural network in neural network-based video coding, such as neural network super-resolution, neural network intra-frame prediction, etc., and only needs to replace the reconstructed image block with the corresponding image block.

[0183] In the neural network-based image filtering method provided in the embodiments of the present application, a first data processing method for the information to be filtered is determined from M+1 data processing methods at the decoding side, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered includes reconstructed image blocks to be filtered, the reconstructed image block is a reconstructed image or a partial region of the reconstructed image, M is a positive integer; when the first data processing method is one of the M data rearrangement methods, data rearrangement is performed on the reconstructed image blocks in the information to be filtered according to the first data processing method to obtain rearranged information to be filtered, and a filtering process is performed on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; data reverse rearrangement is performed on the filtered image of the rearranged information to be filtered according to the first data processing method to obtain a filtered image block of the reconstructed image block. In other words, in the present application, before filtering the information to be filtered, it is first determined whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to perform data rearrangement on the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered match or approach the data distribution characteristics of the training data, thereby improving the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and image decoding performance.

[0184] The image filtering method in the embodiment of the present application has been described above by taking the decoding side as an example. Below, the image filtering method in the embodiment of the present application will be introduced by taking the encoding side as an example.

[0185] 11 is a flowchart of a neural network-based image filtering method provided in an embodiment of the present application. The embodiment of the present application can be applied to the encoder shown in FIG. 1 or FIG. 2 or other electronic devices. For convenience of explanation, the encoder will be taken as an example for explanation. As shown in FIG. 11, the method in the embodiment of the present application includes the following steps:

[0186] S201: A first data processing method for information to be filtered is determined from among M+1 data processing methods.

[0187] In an embodiment of the present application, when encoding a current image, the encoding side divides the current image into coding blocks and performs sequential block coding using the coding blocks as coding units. For example, for a current block in the current image to be encoded, a predicted value of the current block is first obtained using inter-frame and / or intra-frame prediction. Next, a residual value of the current block is obtained based on the predicted value of the current block and the current block. The encoding side performs a transform on the residual value of the current block to obtain transform coefficients. In one example, the encoding side does not quantize the transform coefficients of the current block, but directly codes the transform coefficients to obtain a bitstream. In another example, the encoding side quantizes the transform coefficients of the current block to obtain quantized coefficients, and then codes the quantized coefficients to obtain a bitstream.

[0188] In the encoding process, as shown in Figure 2, the encoding side further inversely transforms the transform coefficients to obtain residual values, and then adds the residual values ​​and the predicted values ​​to obtain the reconstructed values ​​of the current block. Through the above steps, the reconstructed values ​​of each coding block in the current image can be obtained, and these reconstructed values ​​constitute a reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, filtering is performed on the reconstructed image to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoding buffer for prediction of subsequent images.

[0189] In some embodiments, the image filtering method provided in the embodiments of the present application can be used to filter at least one frame of image in a video, i.e., the above-mentioned current image is an image in a video.

[0190] In some embodiments, the image filtering method provided in the embodiments of the present application can be used to perform decoding on a single image, i.e., the current image is a single image, for example, an image generated by an electronic device.

[0191] In the embodiment of the present application, a neural network filter is used on the encoding side to perform filtering on the reconstructed image.

[0192] In the embodiments of the present application, the data distribution characteristics may also be referred to as data distribution features, and mainly refer to the concentration tendency of the data distribution of an image, the degree of discreteness of the data distribution, etc. When the contents of two images are different, the data distribution characteristics are usually different as well. For example, if one image shows animal A performing action B and the other image shows animal A performing action C, the data distribution characteristics of the two images may be different. Therefore, in the embodiments of the present application, when the image content of the information to be filtered is different from the image content of the training data, the data distribution characteristics of the information to be filtered and the data distribution characteristics of the training data may also be different. Thus, when a neural network filter obtained by directly training using the training data directly filters the information to be filtered, the filtering effect may not be ideal. Therefore, in the embodiments of the present application, before using a neural network filter to filter the information to be filtered, it is first determined whether data rearrangement is required for the reconstructed image in the information to be filtered.

[0193] The to-be-filtered information in the embodiment of the present application includes reconstructed image blocks.

[0194] In the embodiments of the present application, when filtering is performed on a reconstructed image on the encoding side, the entire reconstructed image may be directly used as a filtering unit for filtering, or the component image may be divided into multiple blocks and each block may be used as a filtering unit for filtering. For convenience of explanation, in the embodiments of the present application, a reconstructed image block is used as one filtering unit, and the reconstructed image block may be one reconstructed image (i.e., one frame) or a partial region of the reconstructed image.

[0195] In some embodiments, when the reconstructed image block is a partial region of the reconstructed image, some examples of division of the reconstructed image block include, but are not limited to, the following.

[0196] In one possible implementation, the above-mentioned reconstructed image block includes at least one CTU of the reconstructed image, in other words, the encoding side performs division so that at least one CTU of the reconstructed image becomes one reconstructed image block.

[0197] In some examples, as shown in FIG. 8A, a reconstructed image block includes one CTU of the reconstructed image, that is, on the encoding side, division is performed so that one CTU of the reconstructed image becomes one reconstructed image block.

[0198] In another example, as shown in FIG. 8B, a reconstructed image block includes four CTUs of the reconstructed image, that is, on the encoding side, division is performed so that four CTUs of the reconstructed image become one reconstructed image block.

[0199] In one example, the encoding side may further divide the reconstructed image into two, three, or more CTUs to form one reconstructed image block. The CTUs may be horizontal or vertical. Optionally, the CTUs may be adjacent, non-adjacent, partially adjacent, or partially non-adjacent.

[0200] In another possible implementation, the reconstructed image block to be filtered is a predetermined image region of the reconstructed image, i.e., one predetermined image region of the reconstructed image is one reconstructed image block.

[0201] The embodiments of the present application do not limit the specific shape and size of the predetermined image area.

[0202] In one example, the predetermined image region includes at least one incomplete CTU of the reconstructed image, i.e., the division is performed so that at least one incomplete CTU of the reconstructed image becomes one reconstructed image block. In the example shown in Figure 8C, the encoding side performs division so that one CTU and three incomplete CTUs of the reconstructed image become one reconstructed image block, and for example, as shown in Figure 8D, the encoding side performs division so that four incomplete CTUs of the reconstructed image become one reconstructed image block.

[0203] In some embodiments, the predetermined image region is a fixed region, for example, the encoding side obtains the current reconstructed image block to be filtered in the reconstructed image according to the predetermined image region each time filtering is performed, and the size and shape of the image block input to the neural network filter each time are the same, and they are all the predetermined image region.

[0204] In some embodiments, the predetermined image region is variable. For example, in the encoding side, during the first filtering, one reconstructed image block to be filtered in the reconstructed image is obtained according to a first predetermined image region, and input to a block neural network filter as an input image for filtering. During the second filtering, one reconstructed image block to be filtered in the reconstructed image is obtained according to a second predetermined image region, and input to a block neural network filter as an input image for filtering. Based on this, an inference is made. In one example of this embodiment, in the decoding side, the reconstructed image can be divided into several reconstructed image blocks to be filtered, and the shapes and sizes of the several reconstructed image blocks to be filtered can be the same or different, or can be partially the same or partially different.

[0205] The process for determining the first data processing method for the information to be filtered will be described below.

[0206] In an embodiment of the present application, the above-mentioned first data processing method is one of M+1 data processing methods, and the M+1 data processing methods include a method of not rearranging data and M data rearrangement methods.

[0207] The method of not rearranging data refers to not rearranging the data to be filtered, but directly inputting the filtered data to the neural network filter.

[0208] The data rearrangement method refers to rearranging some or all of the information in the information to be filtered according to the data rearrangement method, and then inputting the data to be filtered into a neural network filter for filtering, where some or all of the information in the information to be filtered includes reconstructed image blocks, in other words, in the embodiments of the present application, when data rearrangement is performed on the information to be filtered, it at least includes data rearrangement on the reconstructed image blocks in the information to be filtered.

[0209] In the embodiments of the present application, the specific type of data rearrangement method is not limited.

[0210] In some embodiments, the M data rearrangement schemes include at least one of the following: a rotation scheme, a horizontal flip and then rotate scheme, a vertical flip and then rotate scheme, a diagonal flip and then rotate scheme, a downsampling and then rotate scheme, and an upsampling and then rotate scheme.

[0211] The rotation methods include N*90° left rotation and N*90° right rotation, where N is a positive integer.

[0212] The method of performing horizontal flip and then rotation includes a method of performing horizontal flip and then rotating left by N*90° and a method of performing horizontal flip and then rotating right by N*90°, where N is a positive integer.

[0213] The vertical flip followed by rotation includes vertical flip followed by *90° rotation to the left and vertical flip followed by N*90° rotation to the right, where N is a positive integer.

[0214] The diagonal flip followed by rotation includes diagonal flip followed by N*90° left rotation and diagonal flip followed by N*90° right rotation, where N is a positive integer.

[0215] The downsampling followed by rotation includes downsampling followed by N*90° left rotation and downsampling followed by N*90° right rotation, where N is a positive integer.

[0216] The upsampling followed by rotation includes upsampling followed by N*90° left rotation and upsampling followed by N*90° right rotation, where N is a positive integer.

[0217] In addition, the M data rearrangement methods in the embodiments of the present application may include at least one of the above-mentioned data rearrangement methods, i.e., the rotation method and the method of performing rotation after performing any other pre-processing operation, as well as other data rearrangement methods, and the embodiments of the present application are not limited thereto.

[0218] In one example, the M+1 data processing methods in the embodiment of the present application are as shown in Table 1.

[0219] In the embodiment of the present application, before filtering is performed on the information to be filtered on the encoding side, a first data processing method for the information to be filtered is first determined from among M+1 data processing methods.

[0220] In the embodiment of the present application, there is no limitation on the specific method for determining the first data processing method for the information to be filtered from among M+1 data processing methods on the encoding side.

[0221] Method 1: When the above-mentioned reconstructed image block is a part of the reconstructed image, the encoding side determines a reference image block for the reconstructed image block, and inherits the data processing method of the reference image block.

[0222] For example, when the prediction method corresponding to the reconstructed image block is inter-frame prediction, the encoding side determines a reference image block for the reconstructed image block from the reference image of the current image, and then determines the data processing method of the reference image block as the first data processing method for the reconstructed image block.

[0223] Also, for example, when the prediction method corresponding to the reconstructed image block is intra-frame prediction, the encoding side determines a reference image block for the reconstructed image block from the reconstructed image, and then determines the data processing method of the reference image block as the first data processing method for the reconstructed image block.

[0224] Method 2: The above-mentioned S201 includes the following steps S201-A1 and S201-A2.

[0225] S201-A1: The encoding side determines filtering costs corresponding to each of the M+1 data processing methods; and S201-A2: On the encoding side, one of the M+1 data processing methods with the smallest filtering cost is determined as the first data processing method.

[0226] In the embodiment of the present application, determining a filtering cost corresponding to the no-data-rearrangement mode among the M+1 data processing modes at the encoding side specifically involves performing a filtering process on the information to be filtered using a neural network filter to obtain a filtered image block under the no-data-rearrangement mode, and then determining a filtering cost corresponding to the no-data-rearrangement mode based on the filtered image block under the no-data-rearrangement mode and the original image block of the reconstructed image block.

[0227] The embodiments of the present application are not limited to a specific calculation method of the above-mentioned filtering cost. For example, the above-mentioned filtering cost may be a rate-distortion cost (RDO), or may be an approximate cost, such as SSD, STAD, SAD, etc.

[0228] In the embodiments of the present application, the method for determining the filtering cost corresponding to each of the M data rearrangement methods on the encoding side is basically the same. For convenience of explanation, the method for determining the filtering cost corresponding to the i-th data rearrangement method will be taken as an example.

[0229] At this time, determining the filtering cost corresponding to the data rearrangement method in the above-mentioned S201-A1 includes the following steps S201-A1-a1 to S201-A1-a3.

[0230] S201-A1-a1: for the i-th data rearrangement method among the M data rearrangement methods, the encoding side performs data rearrangement on the reconstructed image based on the i-th data processing method, and uses a neural network filter to perform filtering on the information to be filtered after rearrangement based on the i-th data rearrangement method, thereby obtaining the i-th filtered image block, where i is a positive integer not greater than M; S201-A1-a2: At the encoding side, the i-th filtered image block is obtained by performing data inverse rearrangement on the i-th filtered image block according to the i-th data rearrangement scheme; and S201-A1-a3: In the encoding side, a filtering cost corresponding to the ith data rearrangement scheme is determined based on the original image blocks of the filtered image block and the reconstructed image block under the ith data rearrangement scheme.

[0231] In an embodiment of the present application, for the i-th data rearrangement scheme among M data rearrangement schemes, data rearrangement is performed on a reconstructed image block using the i-th data rearrangement scheme to obtain information to be filtered after rearrangement using the i-th data rearrangement scheme. Next, the information to be filtered after rearrangement using the i-th data rearrangement scheme is input to a neural network filter for filtering to obtain one filtered image block, which is referred to as the i-th filtered image block. After that, data reverse rearrangement is performed on the i-th filtered image block based on the i-th data rearrangement scheme to obtain a filtered image block under the i-th data rearrangement scheme. Finally, a filtering cost corresponding to the i-th data rearrangement scheme is determined based on the filtered image block under the i-th data rearrangement scheme and the original image block of the reconstructed image block.

[0232] For example, as shown in FIG. 10, the M data rearrangement methods include 90° left rotation and 90° right rotation. In this case, the encoding side directly inputs the information to be filtered into a neural network filter to obtain a filtered image block 0 without data rearrangement. This filtered image block 0 without data rearrangement is compared with the original image block of the reconstructed image to calculate a filtering cost corresponding to the method without data rearrangement. Similarly, the encoding side performs a 90° left rotation on information such as the reconstructed image block in the information to be filtered, and then inputs it into the neural network filter to obtain a filtered image block with a 90° left rotation. Next, the encoding side performs a 90° right rotation on the filtered image block with a 90° left rotation to obtain a filtered image block with a 90° left rotation method. The encoding side compares the filtered image block with the 90° left rotation method with the original image block of the reconstructed image to calculate a filtering cost corresponding to the 90° left rotation method. Similarly, the encoding side rotates information such as a reconstructed image block in the information to be filtered by 90° right and then inputs the information into a neural network filter to obtain a 90° right-rotated filtered image block. Next, the encoding side rotates the 90° right-rotated filtered image block by 90° left to obtain a filtered image block using the 90° right rotation method. The encoding side compares the filtered image block using the 90° right rotation method with the original image block of the reconstructed image to calculate a filtering cost corresponding to the 90° right rotation method. The encoding side selects the data processing method with the smallest filtering cost from the method without rearrangement of data, the 90° right rotation method, and the 90° left rotation method as the first data processing method.

[0233] Method 3: The above-mentioned S201 includes the following steps S201-B1 and S201-B2.

[0234] S201-B1: In the encoding side, among the M+1 data processing methods, a filtering cost corresponding to a method that does not rearrange data is determined; and S201-B2: On the encoding side, from among M data rearrangement methods in a predetermined order, the data rearrangement method with the smallest first filtering cost, which corresponds to a method that does not rearrange data, is determined as the first data processing method.

[0235] Among them, the process of determining the filtering cost corresponding to the method that does not rearrange data among the M+1 data processing methods on the encoding side can be referred to the description of the above steps, and detailed description thereof will be omitted here.

[0236] In an embodiment of the present application, the encoding side first determines a filtering cost corresponding to a method without data rearrangement, and then calculates the filtering costs for M data rearrangement methods in a predetermined order. Specifically, first, a first data rearrangement method among the M data rearrangement methods is used to perform data rearrangement on information such as a reconstructed image block in the information to be filtered, and then the information is input to a neural network filter for filtering to obtain a first filtered image block. Next, the first data rearrangement method is used to perform reverse data rearrangement on the first filtered image block to obtain a filtered image block 1 under the first data rearrangement method. The filtered image block 1 is compared with the original image block of the reconstructed image block to determine a filtering cost 1 corresponding to the first data rearrangement method. Next, the filtering cost 1 is compared with the filtering cost corresponding to the method without data rearrangement. If the filtering cost 1 is smaller than the filtering cost corresponding to the method without data rearrangement, the first data rearrangement method is determined as the first data rearrangement method, and subsequent data rearrangement methods are not tried. If filtering cost 1 is equal to or greater than the filtering cost corresponding to the method without data rearrangement, a second data rearrangement method among the M data rearrangement methods is used to perform data rearrangement on information such as reconstructed image blocks in the information to be filtered, a filtering cost 2 corresponding to the second data rearrangement method is calculated, and filtering cost 2 is compared with the filtering cost corresponding to the method without data rearrangement, and if filtering cost 2 is smaller than the filtering cost corresponding to the method without data rearrangement, the second data rearrangement method is determined as the first data processing method. If filtering cost 2 is equal to or greater than the filtering cost corresponding to the method without data rearrangement, a third data rearrangement method among the M data rearrangement methods is used to perform data rearrangement on information such as reconstructed image blocks in the information to be filtered, and based on this, all M data rearrangement methods are checked and the process is terminated.

[0237] In some embodiments, when the filtering costs corresponding to the M data rearrangement methods are all equal to or greater than the filtering costs corresponding to the method that does not rearrange the data, the encoding side determines the method that does not rearrange the data as the first data processing method.

[0238] In the above-mentioned methods 2 and 3, rearranging information such as reconstructed image blocks in the information to be filtered based on the i-th data rearrangement method on the encoding side to obtain rearranged filtering information includes at least the following methods:

[0239] Method 1: When the to-be-filtered information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type of the reconstructed image block, and quantization parameter information, in addition to the reconstructed diagram block, the encoding side may perform data rearrangement only for the reconstructed diagram block, and may not perform data rearrangement for other information.

[0240] Method 2: When the filtering waiting information further includes, in addition to the reconstructed image block, at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type of the reconstructed image block, and quantization parameter information, the encoding side performs data rearrangement on the reconstructed image block according to the i-th data processing method, and also performs data rearrangement on at least one of the predicted image block, the boundary strength image block, the frame type, and quantization parameter information according to the i-th data processing method, thereby obtaining rearranged filtering data.

[0241] Method 3: When the filtering waiting information further includes, in addition to the reconstructed image block, at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type of the reconstructed image block, and quantization parameter information, the encoding side performs data rearrangement for the reconstructed image block based on the i-th data processing method, performs data rearrangement for at least one of the predicted image block and the boundary strength image block based on the i-th data processing method, and skips data rearrangement for at least one of the frame type and quantization parameter information, thereby obtaining rearranged filtering data.

[0242] On the encoding side, a first data processing method can be determined from among M+1 data processing methods based on the above-mentioned methods 1, 2, and 3.

[0243] In some embodiments, after determining the first data processing method on the encoding side, the following steps are further included:

[0244] Step A: Indicate a first data processing method in the bitstream.

[0245] In the embodiment of the present application, after the encoding side determines the first data processing method based on the above steps, the first data processing method is directly indicated to the decoding side, so that the decoding side can obtain the first data processing method by decoding the bitstream without requiring additional calculation, thereby reducing the calculation complexity of the decoding side and improving the decoding performance.

[0246] In the embodiment of the present application, the implementation manner of the above step A includes, but is not limited to, several manners as follows:

[0247] Method 1: The above-mentioned step A includes step A-a1 and step A-a2.

[0248] Step A-a1: determining a first flag based on a first data processing method at the encoding side; and Step A-a2: The encoding side writes a first flag into the bitstream, and the first flag is used to instruct whether data rearrangement is to be performed on the information to be filtered when filtering is performed.

[0249] In method 1, the encoding side first determines the value of the first flag based on the first data processing method. For example, if the encoding side determines that data rearrangement is required for the information to be filtered when filtering, that is, when the first data processing method is a data rearrangement method, the encoding side sets the value of the first flag to yes (e.g., 1) and then writes it into the bitstream. If the encoding side determines that data rearrangement is not required for the information to be filtered when filtering, that is, when the first data processing method is a method that does not rearrange data, the encoding side sets the value of the first flag to no (e.g., 0) and then writes it into the bitstream. In this way, the decoding side decodes the bitstream to obtain the first flag, and then determines the first data processing method based on the first flag.

[0250] In some embodiments, when the first flag indicates that data rearrangement is to be performed on the information to be filtered when filtering, the encoding side must further write a second flag into the bitstream, and the second flag is used to indicate the data rearrangement method of the information to be filtered.

[0251] In the embodiment of the present application, the specific expression formats of the first flag and the second flag are not limited.

[0252] In one example, the correspondence between the values ​​of the first flag and the second flag and the data processing method is as shown in Table 2.

[0253] As shown in Table 2, the encoding side first determines the value of the first flag based on the first data processing method. For example, when the first data processing method is a method that does not rearrange data, the value of the first flag is set to 0. When the first data processing method is a data rearrangement method, the value of the first flag is set to 1, and a second flag is further written into the bitstream. For example, when the first data processing method is a method that rotates left by 90 degrees, the value of the second flag is set to 00 and then written into the bitstream.

[0254] In this method 1, the encoding side specifies the first data processing method by the first flag and the second flag.

[0255] Method 2: In this method, the first and second flags can be replaced by one flag, i.e., the third flag, at the encoding side. In other words, the third flag can indicate both whether to perform data rearrangement when filtering the information to be filtered, and the specific data rearrangement method. In this case, the above-mentioned step A includes step A-b1 and step A-b2.

[0256] Step A-b1: At the encoding side, a third flag is determined based on a first data processing method; and Step A-b2: The encoding side writes a third flag into the bitstream, and the third flag is used to indicate the first data processing method.

[0257] In method 2, the encoding side directly indicates the index of the first data processing method to the decoding side using the third flag. Specifically, the encoding side determines the first data processing method for the information to be filtered from among M+1 data processing methods, and then indicates the first data processing method to the decoding side using the third flag. For example, when the first data processing method is a method that does not rearrange data, the encoding side sets the value of the third flag to a first numerical value (e.g., 0); when the first data processing method is a data rearrangement method, the encoding side sets the value of the third flag to another numerical value; for example, when the first data processing method is a 90° left rotation, the value of the third flag is set to 100. In this way, the decoding side decodes the bitstream to obtain the third flag, and then determines the first data processing method from among M+1 data processing methods based on the value of the third flag.

[0258] In the embodiment of the present application, the specific expression format of the third flag is not limited.

[0259] In one example, the correspondence between the third flag value and the data processing method is as shown in Table 3.

[0260] When the reconstructed image block is a single reconstructed image, the first flag, the second flag, and the third flag may be picture-level flags. When the reconstructed image block is a partial area of ​​a single reconstructed image, the first flag, the second flag, and the third flag may be block-level flags.

[0261] The above is a specific process for determining the first data processing method for information to be filtered on the encoding side. After determining the first data processing method on the encoding side, the encoding side executes the following step S202.

[0262] S202: When the first data processing method is one of M data rearrangement methods, perform data rearrangement on the reconstructed image block based on the first data processing method to obtain rearranged information to be filtered, and perform filtering processing on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered.

[0263] The encoding side determines the first data processing method through the above steps, and when the first data processing method is a method that does not rearrange data, the encoding side does not rearrange data on the information to be filtered, but performs filtering processing on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0264] When the above-mentioned first data processing method is one of M data rearrangement methods, it means that before the information to be filtered is input into the neural network filter, data rearrangement needs to be performed first.

[0265] In some embodiments, the to-be-filtered information includes only reconstructed image blocks.

[0266] In some embodiments, the to-be-filtered information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type (slice type) of the reconstructed image block, and quantization parameter information. Optionally, the quantization parameter information includes a slice quantization parameter (QP) and a base quantization parameter (QP).

[0267] In the embodiment of the present application, when the first data processing method is a data rearrangement method, the encoding side performs data rearrangement on the information to be filtered, and at least the following several methods are included:

[0268] First method: The encoding side performs data rearrangement only on the reconstructed image block in the information to be filtered, and does not perform data rearrangement on other information in the information to be filtered. For example, the information to be filtered includes information such as a reconstructed image block, a predicted image block corresponding to the reconstructed image block, a boundary strength image block, the frame type of the reconstructed image block, and quantization parameter information, and the encoding side adopts a first data processing method to perform data rearrangement on the reconstructed image block, but does not perform data rearrangement on information such as a predicted image block corresponding to the reconstructed image block, a boundary strength image block, the frame type of the reconstructed image block, and quantization parameter information. Because the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type of the reconstructed image block, and quantization parameter information are all auxiliary information of the reconstructed image block, there is no need to perform data rearrangement on this information, which reduces the data rearrangement load on the encoding side and improves the coding efficiency on the encoding side.

[0269] Second method: Perform data rearrangement on the reconstructed image block based on the first data processing method, and perform data rearrangement on at least one of the predicted image block, the boundary strength image block, the frame type, and the quantization parameter information on the encoding side based on the first data processing method, thereby obtaining rearranged filtering data.

[0270] Third method: Data rearrangement is performed on the reconstructed image block based on the first data processing method, and data rearrangement is performed on at least one of the predicted image block and the boundary intensity image block on the encoding side based on the first data processing method, and data rearrangement of at least one of the frame type and quantization parameter information is skipped, thereby obtaining rearranged filtering data.

[0271] The encoding side adopts a first data processing method to perform data rearrangement on the reconstructed image blocks in the information to be filtered, or to perform data rearrangement on the reconstructed image blocks and other information in the filtering information, thereby obtaining rearranged filtering information, and then inputting the rearranged filtering information into a neural network filter, which performs filtering processing on the rearranged information to be filtered, thereby obtaining the filtered image blocks in the rearranged information to be filtered.

[0272] In an embodiment of the present application, before filtering the information to be filtered on the encoding side, it is first determined whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, a first data processing method is used to rearrange information such as reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered match or approach those of the training data. This improves the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and image encoding performance.

[0273] S203: Perform data reverse rearrangement on the rearranged filtered image of the information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block.

[0274] According to the above steps, the encoding side performs data rearrangement on information such as the reconstructed image blocks in the information to be filtered, so that the neural network filter filters the rearranged information to be filtered, and what is obtained is a filtered image of the rearranged information to be filtered, and the data distribution characteristics of the filtered image of the rearranged information to be filtered are inconsistent with the data distribution characteristics of the information to be filtered. Therefore, in the embodiments of the present application, the encoding side must use a first data processing method to perform data rearrangement on the reconstructed image blocks etc. in the information to be filtered to obtain the filtered image of the rearranged information to be filtered, and then further use the first data processing method to perform data reverse rearrangement on the filtered image of the rearranged information to be filtered to obtain the filtered image block of the reconstructed image block.

[0275] For example, if the first data processing method described above is a 90° left rotation, then on the encoding side, the filtered image of the rearranged information waiting to be filtered must be rotated 90° right to obtain the filtered image block of the reconstructed image block.

[0276] Also, for example, if the first data processing method described above is a 90° right rotation, then the encoding side needs to perform a 90° left rotation on the filtered image of the rearranged filtering waiting information to obtain the filtered image block of the reconstructed image block.

[0277] In some embodiments, the reconstructed image block includes at least one of a Y component reconstructed image block, a U component reconstructed image block, and a V component reconstructed image block. In other words, the adaptive data rearrangement scheme in the embodiments of the present application can be applied to at least one of the luma component Y and the chroma components U and V.

[0278] In some embodiments, the image filtering method in the embodiments of the present application can be used in a video encoder, for example, in loop filtering, and the filtered reconstructed image is stored in a decoding buffer for encoding the image of the next frame, and at this time, the output of the filter can affect the video encoding.

[0279] In some embodiments, the image filtering method in the embodiments of the present application can be used in video post-processing products, for example, on the display side, i.e., the reconstructed image after filtering can be directly displayed without being stored in a decoding buffer, and the output of the filter does not affect the video encoding.

[0280] In some embodiments, the image encoding method in the embodiments of the present application can further be used in any one module that uses a neural network in neural network-based video encoding, such as neural network super-resolution, neural network intra-frame prediction, etc., and can simply replace the reconstructed image block with the corresponding image block.

[0281] In the image filtering method provided in the embodiments of the present application, the encoding side determines a first data processing method for the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered includes reconstructed image blocks to be filtered, the reconstructed image block is a reconstructed image or a partial region of the reconstructed image, M is a positive integer; when the first data processing method is one of the M data rearrangement methods, the encoding side performs data rearrangement on the reconstructed image blocks in the information to be filtered according to the first data processing method to obtain rearranged information to be filtered, and also performs filtering on the rearranged information to be filtered using a neural network filter to obtain filtered image blocks of the rearranged information to be filtered; the encoding side performs data reverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing method to obtain filtered image blocks of the reconstructed image blocks. In other words, in the present application, before filtering the information to be filtered, the encoding side first determines whether data rearrangement is required for the information to be filtered. If it is determined that data rearrangement is required, the encoding side uses the first data processing method to perform data rearrangement on the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered match or approach the data distribution characteristics of the training data. This improves the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and image encoding performance.

[0282] Although the above describes in detail the most suitable implementation modes of the present application in conjunction with the drawings, the present application is not limited to the specific details of the above implementation modes, and any modifications may be made to the technical solutions of the present application within the technical scope of the present application, and all such modifications fall within the scope of protection of the present application. For example, the specific technical features described in the above specific implementation modes may be combined in any suitable manner if there is no contradiction, and in order to avoid unnecessary repetition, this application will omit a comprehensive description of each possible combination. For example, the different implementation modes of the present application may also be combined in any manner, and as long as they do not contradict the idea of ​​the present application, all of them shall be considered as the disclosure content of the present application.

[0283] It can be further understood that in the embodiments of the methods of the present application, the magnitude of the sequence numbers of the above-mentioned processes does not mean the execution order, and the execution order of each process should be determined by its function and internal logic, and should not limit the implementation process of the embodiments of the present application.

[0284] An embodiment of the method of the present application has been described in detail above in conjunction with FIGS. 7 to 11. Below, an embodiment of the apparatus of the present application will be introduced in detail in conjunction with FIGS.

[0285] 12 is an exemplary block diagram of a neural network-based image filtering device provided in one embodiment of the present application, which can be applied to a decoding device.

[0286] As shown in FIG. 12, the image filtering device 10 may include:

[0287] a reordering determination unit 11: determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block to be filtered, the reconstructed image block being a reconstructed image or a partial region of the reconstructed image, where M is a positive integer; a reordering filtering unit 12: when the first data processing manner is one of the M data reordering manners, performing data reordering on the reconstructed image block according to the first data processing manner to obtain reordered information to be filtered, and performing filtering on the reordered information to be filtered using a neural network filter to obtain a filtered image block of the reordered information to be filtered; and Inverse rearrangement unit 13: performs data inverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain the filtered image block of the reconstructed image block.

[0288] In some embodiments, when the first data processing method is a method of not rearranging the data, the rearrangement filtering unit 12 further performs a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0289] In some embodiments, the first data processing method is one of the M+1 data processing methods that has the smallest filtering cost.

[0290] In some embodiments, the re-sequencing unit 11 specifically determines the first data processing manner from among the M+1 data processing manners by decoding the bitstream.

[0291] In some embodiments, the reordering determination unit 11 specifically decodes the bitstream to obtain a first flag, which is used to indicate whether to perform data reordering on the information to be filtered when filtering; and obtains the first data processing method from among the M+1 data processing methods based on the first flag.

[0292] In some embodiments, the re-ordering determination unit 11 specifically determines a method of not re-ordering the data as the first data processing method when the first flag indicates that data re-ordering is not performed on the information to be filtered when filtering; and when the first flag indicates that data re-ordering is performed on the information to be filtered when filtering, decodes the bitstream to obtain a second flag, and determines the first data processing method from among the M data re-ordering methods based on the second flag, where the second flag is used to indicate an index of the data re-ordering method of the information to be filtered.

[0293] In some embodiments, the re-sequencing unit 11 specifically decodes the bitstream to obtain a third flag, which is used to indicate the first data processing method; and determines the first data processing method from among the M+1 data processing methods based on the third flag.

[0294] In some embodiments, the M data rearrangement methods include at least one of the following: a rotation method, a horizontal flip and then rotate method, a vertical flip and then rotate method, a diagonal flip and then rotate method, a downsampling and then rotate method, and an upsampling and then rotate method.

[0295] In some embodiments, the rotation scheme comprises any one of N*90° left rotation and N*90° right rotation, where N is a positive integer.

[0296] In some embodiments, when the first data processing method is a method of not rearranging the data, the rearrangement filtering unit 12 further performs a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0297] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block of a Y component, a reconstructed image block of a U component, and a reconstructed image block of a V component.

[0298] In some embodiments, the filtering wait information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, a frame type of the reconstructed image block, and quantization parameter information.

[0299] In some embodiments, the rearrangement filtering unit 12 specifically performs data rearrangement on the reconstructed image block based on the first data processing method, and performs data rearrangement on at least one of the predicted image block, the boundary intensity image block, the frame type, and the quantization parameter information based on the first data processing method, to obtain the rearranged filtered data.

[0300] It should be understood that the device embodiment corresponds to the method embodiment, and similar descriptions can be referred to the method embodiment. To avoid repetition, detailed descriptions thereof will be omitted here. Specifically, the device shown in Figure 12 can implement the method embodiment shown in Figure 7 above, and the above and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the decoder, and detailed descriptions thereof will be omitted here for brevity.

[0301] 13 is an exemplary block diagram of a neural network-based image filtering device provided in one embodiment of the present application, which can be applied to an encoding device.

[0302] As shown in FIG. 13, the image filtering device 20 may include:

[0303] a reordering determination unit 21: determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block, the reconstructed image block being a reconstructed image or a partial region of the reconstructed image, where M is a positive integer; a reordering filtering unit 22: when the first data processing manner is one of the M data reordering manners, performing data reordering on the information to be filtered according to the first data processing manner to obtain rearranged information to be filtered, and performing filtering on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and Inverse rearrangement unit 23: performs data inverse rearrangement on the filtered image block of the rearranged information to be filtered according to the first data processing manner to obtain the filtered image block of the reconstructed image block.

[0304] In some embodiments, when the first data processing method is a method of not rearranging the data, the rearrangement filtering unit 22 further performs a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0305] In some embodiments, the re-sequencing unit 21 specifically determines a filtering cost corresponding to each of the M+1 data processing methods; and determines one data processing method among the M+1 data processing methods with the smallest filtering cost as the first data processing method.

[0306] In some embodiments, the reordering determination unit 21 specifically: for an i-th data reordering scheme among the M data reordering schemes, perform data reordering on the reconstructed image based on the i-th data processing scheme, and use the neural network filter to perform filtering processing on the information to be filtered after reordering by the i-th data reordering scheme, to obtain an i-th filtered image block, where i is a positive integer less than or equal to M; perform data reverse reordering on the i-th filtered image block based on the i-th data reordering scheme, to obtain a filtered image block under the i-th data reordering scheme; and determine a filtering cost corresponding to the i-th data reordering scheme based on the filtered image block under the i-th data reordering scheme and the original image block of the reconstructed image block.

[0307] In some embodiments, the reordering determination unit 21 specifically performs a filtering process on the information to be filtered using the neural network filter to obtain a filtered image block without rearranging the data; and determines a filtering cost corresponding to the method without rearranging the data based on the filtered image block without rearranging the data and the original image block of the reconstructed image block.

[0308] In some embodiments, the reordering filtering unit 22 further indicates the first data processing scheme in the bitstream.

[0309] In some embodiments, the reordering filtering unit 22 specifically determines a first flag based on the first data processing method; and writes the first flag into the bitstream, where the first flag is used to indicate whether to perform data reordering when filtering the information to be filtered.

[0310] In some embodiments, when the first flag indicates that data rearrangement is to be performed on the information to be filtered when filtering, the rearrangement filtering unit 22 further writes a second flag into the bitstream, which is used to indicate the data rearrangement method of the information to be filtered.

[0311] In some embodiments, the reordering filtering unit 22 specifically determines the third flag based on the first data processing method; and writes the third flag into the bitstream, where the third flag is used to indicate the first data processing method.

[0312] In some embodiments, the M data rearrangement methods include at least one of the following: a rotation method, a horizontal flip and then rotate method, a vertical flip and then rotate method, a diagonal flip and then rotate method, a downsampling and then rotate method, and an upsampling and then rotate method.

[0313] In some embodiments, the rotation scheme comprises any one of N*90° left rotation and N*90° right rotation, where N is a positive integer.

[0314] In some embodiments, when the first data processing method is a method of not rearranging the data, the rearrangement filtering unit 22 further performs a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

[0315] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block of a Y component, a reconstructed image block of a U component, and a reconstructed image block of a V component.

[0316] In some embodiments, the filtering wait information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, a frame type of the reconstructed image block, and quantization parameter information.

[0317] In some embodiments, the rearrangement filtering unit 22 specifically performs data rearrangement on the reconstructed image block based on the first data processing method, and performs data rearrangement on at least one of the predicted image block, the boundary intensity image block, the frame type, and the quantization parameter information based on the first data processing method, to obtain the rearranged filtered data.

[0318] It should be understood that the device embodiment corresponds to the method embodiment, and similar descriptions can be referred to the method embodiment. To avoid repetition, detailed descriptions thereof will be omitted here. Specifically, the device shown in Figure 13 can implement the method embodiment shown in Figure 11 above, and the above and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the encoder, and detailed descriptions thereof will be omitted here for brevity.

[0319] The above describes the apparatus in the embodiments of the present application from the perspective of functional modules, in conjunction with the drawings. It can be understood that the functional modules may be implemented in the form of hardware, software, or a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application may be completed by a hardware integrated logic circuit and / or software instructions in a processor. Furthermore, the method steps disclosed in the embodiments of the present application may be directly implemented as a hardware decoding processor or may be executed as a combination of hardware and software modules in a decoding processor. Optionally, the software modules may be located in a storage medium well-established in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory, and the processor can read information in the memory and complete the steps of the above method embodiments in combination with the hardware.

[0320] 14 is an exemplary block diagram of an electronic device provided in an embodiment of the present application. The electronic device in FIG. 14 may be the encoder, decoder, or other image filtering device described above.

[0321] As shown in FIG. 14, the electronic equipment 30 may include:

[0322] Memory 31 and processor 32: The memory 31 is used to store a computer program 33 and transmit the program 33 to the processor 32. In other words, the processor 32 can call up and run the computer program 33 from the memory 31 to perform the method in the embodiment of the present application.

[0323] For example, the processor 32 may perform the steps in the method 200 described above based on instructions in the computer program 33 .

[0324] In some embodiments of the present application, the processor 32 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), other programmable logic devices, discrete gates, transistor logic devices, discrete hardware components, etc.

[0325] In some embodiments of the present application, storage 31 may include, but is not limited to, volatile and / or nonvolatile memory. Nonvolatile memory includes read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory includes random access memory (RAM), which acts as external cache memory. Many forms of RAM are available, including, by way of example and not limitation, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM, DR RAM).

[0326] In some embodiments of the present application, the computer program 33 can be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to implement the methods provided in the present application. The one or more modules can be a series of computer program instruction segments that can complete a specific function, and the instruction segments are intended to explain the processes that the computer program 33 executes in the electronic device.

[0327] As shown in FIG. 14, the electronic device 30 may further include: Transceiver 34: The transceiver 34 is connected to the processor 32 or the memory 31.

[0328] The processor 32 controls the transceiver 34 to communicate with other devices, specifically, to transmit information or data to other devices or receive information or data from other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include an antenna, and the number of antennas may be one or more.

[0329] It should be understood that each assembly in the electronic device 30 is connected via a bus system, which includes a data bus as well as a power bus, a control bus and a status signal bus.

[0330] According to one aspect of the present application, a computer storage medium is provided having a computer program stored therein, which, when executed by a computer, causes the computer to perform the method in the above-described method embodiments. Alternatively, embodiments of the present application may further provide a computer program product including instructions, which, when executed by a computer, cause the computer to perform the method in the above-described method embodiments.

[0331] According to another aspect of the present application, there is provided a computer program product or a computer program comprising computer instructions stored in a computer-readable storage medium, the computer instructions being read by a processor of a computing device from the computer-readable storage medium and executed by the processor to cause the computing device to perform the method in the method embodiments described above.

[0332] In other words, when implemented in software, the software may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed by a computer, the flowcharts or functions in the embodiments of the present application can be generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored on a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optics, digital subscriber line (DSL)) or wireless (e.g., infrared, radio, microwave, etc.) methods. The computer-readable storage medium may be any available medium accessible by a computer, or a data storage device, such as a server or data center, that includes one or more available media. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape, etc.), an optical medium (e.g., a digital video disc (DVD)), or a semiconductor medium (e.g., a solid state disk (SSD)).

[0333] Furthermore, modules described as separate components may or may not be physically separated, and components shown as modules may or may not be physical modules. That is, they may be located in one location or distributed across multiple network elements. To implement the technical solutions of the present embodiments, some or all of the modules may be selected according to actual needs. For example, each functional module in each embodiment of the present application may be integrated into one processing module, each module may exist physically separately, or two or more modules may be integrated into one module.

[0334] Although the preferred embodiment of the present application has been described above, the present application is not limited to this embodiment, and any modification to the present application falls within the technical scope of the present application as long as it does not depart from the spirit of the present application.

Claims

1. 1. A neural network-based image filtering method implemented by a decoding device, comprising: determining a first data processing method of the to-be-filtered information from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the to-be-filtered information including reconstructed image blocks to be filtered, where M is a positive integer; when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the reconstructed image block according to the first data processing manner to obtain rearranged filtering information; performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and an image filtering method including performing data inverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block;

2. 2. The image filtering method according to claim 1, When the first data processing method is a method that does not rearrange the data, the image filtering method further includes a step of performing a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

3. 2. The image filtering method according to claim 1, The image filtering method, wherein the first data processing method is one of the M+1 data processing methods with the smallest filtering cost.

4. 2. The image filtering method according to claim 1, The step of determining a first data processing method for the information to be filtered from among the M+1 data processing methods includes: a step of decoding the bitstream to obtain a first flag, the first flag being used to indicate whether data rearrangement is to be performed on the information to be filtered when filtering; determining a method of not rearranging the data as the first data processing method when the first flag indicates that data rearrangement is not to be performed when filtering the information to be filtered; and When the first flag indicates that data rearrangement should be performed on the information to be filtered during filtering, decoding the bitstream to obtain a second flag, and determining the first data processing method from among the M data rearrangement methods based on the second flag, wherein the second flag is used to indicate an index of the data rearrangement method of the information to be filtered.

5. 2. The image filtering method according to claim 1, The step of determining a first data processing method for the information to be filtered from among the M+1 data processing methods includes: decoding the bitstream to obtain a third flag, the third flag being used to indicate the first data processing method; and An image filtering method comprising: determining the first data processing method from among the M+1 data processing methods based on the third flag.

6. 2. The image filtering method according to claim 1, The M data rearrangement methods include at least one of a rotation method, a horizontal flip and then rotation method, a vertical flip and then rotation method, a diagonal flip and then rotation method, a downsampling and then rotation method, and an upsampling and then rotation method.

7. 7. An image filtering method according to claim 6, comprising: The image filtering method, wherein the rotation method includes any one of N*90° left rotation and N*90° right rotation, where N is a positive integer.

8. 2. The image filtering method according to claim 1, The image filtering method, wherein the filtering wait information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, a frame type of the reconstructed image block, and quantization parameter information.

9. 9. An image filtering method according to claim 8, comprising: performing data rearrangement on the reconstructed image block based on the first data processing method to obtain the rearranged filtering information, performing data rearrangement on the reconstructed image block based on the first data processing scheme; and performing data rearrangement on at least one of the predicted image block, the boundary intensity image block, the frame type, and the quantization parameter information based on the first data processing scheme, thereby obtaining the rearranged filtered data.

10. 1. A neural network-based image filtering method implemented by an encoding device, comprising: determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not rearranging data and M data rearrangement methods, the information to be filtered including a reconstructed image block, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the information to be filtered according to the first data processing manner to obtain rearranged information to be filtered; performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and an image filtering method including performing data inverse rearrangement on the filtered image blocks of the rearranged information to be filtered according to the first data processing manner to obtain filtered image blocks of the reconstructed image blocks.

11. 11. An image filtering method according to claim 10, comprising: When the first data processing method is a method that does not rearrange the data, the image filtering method further includes a step of performing a filtering process on the information to be filtered using the neural network filter to obtain a filtered image of the reconstructed image block.

12. 11. An image filtering method according to claim 10, comprising: The step of determining a first data processing method for the information to be filtered from among the M+1 data processing methods includes: determining a filtering cost corresponding to each of the M+1 data processing schemes; and An image filtering method comprising the step of determining, as the first data processing method, one data processing method having a minimum filtering cost from among the M+1 data processing methods.

13. 11. An image filtering method according to claim 10, comprising: The step of determining a filtering cost corresponding to each of the M data rearrangement schemes includes: a step of performing data rearrangement on the reconstructed image based on the i-th data processing method for an i-th data rearrangement method among the M data rearrangement methods, and performing filtering processing on the information to be filtered after rearrangement by the i-th data rearrangement method using the neural network filter to obtain an i-th filtered image block, where i is a positive integer equal to or less than M; performing data inverse rearrangement on the i-th filtered image block based on the i-th data rearrangement scheme to obtain a filtered image block under the i-th data rearrangement scheme; and The image filtering method includes determining a filtering cost corresponding to the ith data rearrangement scheme based on the filtered image block under the ith data rearrangement scheme and the original image block of the reconstructed image block.

14. 13. An image filtering method according to claim 12, comprising: The step of determining a filtering cost corresponding to a method of not rearranging the data includes: performing a filtering process on the information to be filtered using the neural network filter to obtain a filtered image block without rearranging the data; and An image filtering method comprising: determining a filtering cost corresponding to the method without rearranging the data based on a filtered image block under the method without rearranging the data and an original image block of the reconstructed image block.

15. 11. An image filtering method according to claim 10, comprising: writing a first flag into the bitstream, the first flag being used to indicate whether data rearrangement should be performed on the information to be filtered when filtering; and an image filtering method, further comprising the step of: writing a second flag into the bitstream when the first flag indicates that data rearrangement should be performed on the information to be filtered during filtering, the second flag being used to indicate a data rearrangement method for the information to be filtered.

16. 11. An image filtering method according to claim 10, comprising: The M data rearrangement methods include at least one of a rotation method, a horizontal flip and then rotation method, a vertical flip and then rotation method, a diagonal flip and then rotation method, a downsampling and then rotation method, and an upsampling and then rotation method.

17. 11. An image filtering method according to claim 10, comprising: The image filtering method, wherein the filtering wait information further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary intensity image block, a frame type of the reconstructed image block, and quantization parameter information.

18. 18. An image filtering method according to claim 17, comprising: performing data rearrangement on the reconstructed image block based on the first data processing method to obtain the rearranged filtering information, performing data rearrangement on the reconstructed image block based on the first data processing scheme; and performing data rearrangement on at least one of the predicted image block, the boundary intensity image block, the frame type, and the quantization parameter information based on the first data processing scheme, thereby obtaining the rearranged filtered data.

19. 1. A neural network-based image filtering device applied to a decoding device, comprising: a reordering determination unit for determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block to be filtered, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; a rearrangement filtering unit for, when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the reconstructed image block according to the first data processing manner to obtain rearranged information to be filtered, and performing a filtering process on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and an inverse rearrangement unit for performing data inverse rearrangement on the filtered image of the rearranged information to be filtered according to the first data processing manner to obtain a filtered image block of the reconstructed image block;

20. 1. A neural network-based image filtering device applied to an encoding device, comprising: a reordering determination unit for determining a first data processing method of the information to be filtered from M+1 data processing methods, the M+1 data processing methods including a method of not reordering data and M data reordering methods, the information to be filtered including a reconstructed image block, the reconstructed image block being one reconstructed image or a partial region of the reconstructed image, and the M being a positive integer; a rearrangement filtering unit for, when the first data processing manner is one of the M data rearrangement manners, performing data rearrangement on the information to be filtered according to the first data processing manner to obtain rearranged information to be filtered, and performing filtering on the rearranged information to be filtered using a neural network filter to obtain a filtered image block of the rearranged information to be filtered; and an inverse rearrangement unit for performing data inverse rearrangement on the filtered image blocks of the rearranged information to be filtered according to the first data processing manner to obtain filtered image blocks of the reconstructed image blocks;

21. An electronic device, a processor; and a memory coupled to the processor; The storage device stores a computer program, 19. An electronic device, wherein the processor is configured to execute a computer program to implement the image filtering method according to any one of claims 1 to 18.

22. A program for causing a computer to execute the image filtering method according to any one of claims 1 to 18.

Images