Image filtering method and device based on neural network, equipment and storage medium

Abstract

The invention provides an image filtering method and device based on a neural network, equipment and a storage medium, which can be applied to the fields of image processing, video coding and decoding, video live broadcast and the like, and comprises the following steps: determining a first data processing mode of to-be-filtered information from M + 1 data processing modes; if the first data processing mode is a data rearrangement mode, based on the first data processing mode, carrying out data rearrangement on the reconstructed image block in the to-be-filtered information, and carrying out filtering processing on the rearranged to-be-filtered information through a neural network filter to obtain a filtered image block of the rearranged to-be-filtered information; and based on the first data processing mode, performing data anti-rearrangement on the rearranged filtered image of the to-be-filtered information to obtain a filtered image block of the reconstructed image block. According to the method, the to-be-filtered information is subjected to data rearrangement, so that the data distribution characteristics of the rearranged to-be-filtered information are consistent with or close to the data distribution characteristics of the training data, and the filtering effect of the neural network filter is improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image coding and decoding technology, and in particular to a neural network-based image filtering method, device, equipment and storage medium. Background Art

[0002] With the development of video technology, the amount of data included in video data is large. In order to facilitate the transmission of video data, video devices implement video compression technology to make video data more efficiently transmitted or stored. In video compression, both the encoding end and the decoding end need to obtain reconstructed images through operations such as inverse quantization and inverse transformation. Since loss is introduced in video compression, the reconstructed image is filtered to reduce the compression loss of the image.

[0003] With the rapid development of neural network technology, neural network filters have been widely used in video processing. However, in practical applications, there may be differences between the actual information to be filtered by the neural network filter and the data distribution of the training data, resulting in unsatisfactory filtering effects of the neural network filter. Summary of the invention

[0004] The present application provides a neural network-based image filtering method, device, equipment and storage medium to improve the filtering effect of the image.

[0005] In a first aspect, the present application provides an image filtering method based on a neural network, comprising:

[0006] Determine a first data processing method for information to be filtered from M+1 data processing methods, wherein the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0007] If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data is rearranged on the reconstructed image block to obtain rearranged information to be filtered, and filtering is performed on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0008] Based on the first data processing method, reversely rearrange the data of the rearranged filtered image of the information to be filtered to obtain a filtered image block of the reconstructed image block; and

[0009] If the first data processing method is the data non-rearrangement method, the information to be filtered is filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

[0010] In a second aspect, the present application provides an image filtering method based on a neural network, comprising:

[0011] Determine a first data processing method for the information to be filtered from M+1 data processing methods, wherein the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0012] If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data rearrangement is performed on the information to be filtered to obtain rearranged information to be filtered, and filtering is performed on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0013] Based on the first data processing method, reversely rearrange the data of the rearranged filtered image block of the information to be filtered to obtain the filtered image block of the reconstructed image block; and

[0014] If the first data processing method is the data non-rearrangement method, the information to be filtered is filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

[0015] In a third aspect, the present application provides an image filtering device based on a neural network, comprising:

[0016] a rearrangement determination unit, configured to determine a first data processing mode of the information to be filtered from M+1 data processing modes, wherein the M+1 data processing modes include a mode of not rearranging data and M data rearrangement modes, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0017] a rearrangement filtering unit, configured to, if the first data processing mode is one of the M data rearrangement modes, perform data rearrangement on the reconstructed image block based on the first data processing mode to obtain rearranged information to be filtered, and perform filtering processing on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0018] an anti-rearrangement unit, configured to perform data anti-rearrangement on the rearranged filtered image of the information to be filtered based on the first data processing method, so as to obtain a filtered image block of the reconstructed image block; and

[0019] The rearrangement filtering unit is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-rearrangement method.

[0020] In a fourth aspect, the present application provides an image filtering device based on a neural network, comprising:

[0021] a rearrangement determination unit, configured to determine a first data processing mode of the information to be filtered from M+1 data processing modes, wherein the M+1 data processing modes include a mode of not rearranging data and M data rearrangement modes, the information to be filtered includes a reconstructed image block, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0022] a rearrangement filtering unit, configured to, if the first data processing mode is one of the M data rearrangement modes, rearrange the information to be filtered based on the first data processing mode to obtain rearranged information to be filtered, and filter the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0023] an anti-rearrangement unit, configured to perform data anti-rearrangement on the rearranged filtered image block of the to-be-filtered information based on the first data processing method, to obtain a filtered image block of the reconstructed image block; and

[0024] The rearrangement filtering unit is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-rearrangement method.

[0025] In a fifth aspect, a decoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the first aspect or its implementations.

[0026] In a sixth aspect, an encoder is provided, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program stored in the memory to execute the method in the second aspect or its implementation manners.

[0027] In a seventh aspect, a chip is provided for implementing the method in any one of the first to second aspects or their respective implementations. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device equipped with the chip executes the method in any one of the first to second aspects or their respective implementations.

[0028] In an eighth aspect, a computer-readable storage medium is provided for storing a computer program, wherein the computer program enables a computer to execute the method in any one of the first to second aspects above or in each of their implementations.

[0029] In a ninth aspect, a computer program product is provided, comprising computer program instructions, wherein the computer program instructions enable a computer to execute the method in any one of the first to second aspects above or in each of their implementations.

[0030] In a tenth aspect, a computer program is provided, which, when executed on a computer, enables the computer to execute the method in any one of the first to second aspects or in each of their implementations.

[0031] In summary, the present application determines a first data processing method for the information to be filtered from M+1 data processing methods, and the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, and the information to be filtered includes a reconstructed image block to be filtered; if the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data is rearranged on the reconstructed image block in the information to be filtered to obtain the rearranged information to be filtered, and the rearranged information to be filtered is filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; based on the first data processing method, data is reversely rearranged on the filtered image of the rearranged information to be filtered to obtain a filtered image block of the reconstructed image block. That is to say, before filtering the information to be filtered, the present application first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image encoding and decoding performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 A schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application;

[0034] Figure 2 A schematic diagram of a coding framework provided in an embodiment of the present application;

[0035] Figure 3 is a schematic diagram of a decoding framework provided in an embodiment of the present application;

[0036] Figure 4 is a schematic diagram of a coding unit;

[0037] Figure 5 A schematic diagram of the filtering process of a neural network filter;

[0038] Fig. 6A Schematic diagram of the difference in data distribution characteristics between the training set and the test set;

[0039] Figure 6B Schematic diagram of data distribution characteristics of the rotated test set;

[0040] Figure 7 A flowchart of a neural network-based image filtering method provided in one embodiment of the present application;

[0041] FIG. 8A to FIG. 8D Schematic diagram of reconstructed image blocks;

[0042] Fig. 9 A schematic diagram of a filtered area around a reconstructed image block;

[0043] Fig.10 A schematic diagram of a process for determining a first data processing method according to an embodiment of the present application;

[0044] Fig.11 A flowchart of a neural network-based image filtering method provided in one embodiment of the present application;

[0045] Fig.12 is a schematic block diagram of an image filtering device based on a neural network provided in one embodiment of the present application;

[0046] Fig.13 is a schematic block diagram of an image filtering device based on a neural network provided in one embodiment of the present application;

[0047] Fig.14 It is a schematic block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0048] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0049] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In an embodiment of the present invention, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined according to A. However, it should also be understood that determining B according to A does not mean that B is determined only according to A, but B can also be determined according to A and / or other information. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or server that includes a series of steps or units does not have to be limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices. In the description of the present application, unless otherwise specified, "multiple" refers to two or more than two.

[0050] The present application can be applied to the field of image coding and decoding, the field of video coding and decoding, the field of hardware video coding and decoding, the field of dedicated circuit video coding and decoding, the field of real-time video coding and decoding, etc. For example, the scheme of the present application can be combined with the standard of end-to-end image coding based on deep learning, such as JPEG AI. Alternatively, the scheme of the present application can be combined with other exclusive or industry standards and operate, and the standards include ITU-TH.261, ISO / IECMPEG-1Visual, ITU-TH.262 or ISO / IECMPEG-2Visual, ITU-TH.263, ISO / IECMPEG-4Visual, ITU-TH.264 (also known as ISO / IECMPEG-4AVC), including scalable video coding (SVC) and multi-view video coding (MVC) extensions. It should be understood that the technology of the present application is not limited to any specific coding standard or technology.

[0051] To facilitate understanding, first combine Figure 1 The video encoding and decoding system involved in the embodiments of the present application is introduced.

[0052] Figure 1 This is a schematic block diagram of a video encoding and decoding system involved in an embodiment of the present application. It should be noted that: Figure 1 This is just an example. The video encoding and decoding system of the embodiment of the present application includes but is not limited to Figure 1 As shown. Figure 1 As shown, the video encoding and decoding system 100 includes an encoding device 110 and a decoding device 120. The encoding device is used to encode (which can be understood as compressing) video data to generate a code stream, and transmit the code stream to the decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0053] The encoding device 110 of the embodiment of the present application can be understood as a device with a video encoding function, and the decoding device 120 can be understood as a device with a video decoding function, that is, the embodiment of the present application includes a wider range of devices for the encoding device 110 and the decoding device 120, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle-mounted computers, etc.

[0054] In some embodiments, the encoding device 110 may transmit the encoded video data (eg, a code stream) to the decoding device 120 via the channel 130. The channel 130 may include one or more media and / or devices capable of transmitting the encoded video data from the encoding device 110 to the decoding device 120.

[0055] In one example, the channel 130 includes one or more communication media that enable the encoding device 110 to transmit the encoded video data directly to the decoding device 120 in real time. In this example, the encoding device 110 can modulate the encoded video data according to the communication standard and transmit the modulated video data to the decoding device 120. The communication medium includes a wireless communication medium, such as a radio frequency spectrum, and optionally, the communication medium may also include a wired communication medium, such as one or more physical transmission lines.

[0056] In another example, the channel 130 includes a storage medium, which can store the video data encoded by the encoding device 110. The storage medium includes a variety of locally accessible data storage media, such as optical disks, DVDs, flash memories, etc. In this example, the decoding device 120 can obtain the encoded video data from the storage medium.

[0057] In another example, the channel 130 may include a storage server that can store the video data encoded by the encoding device 110. In this example, the decoding device 120 can download the stored encoded video data from the storage server. Alternatively, the storage server can store the encoded video data and transmit the encoded video data to the decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0058] In some embodiments, the encoding device 110 includes a video encoder 112 and an output interface 113. The output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0059] In some embodiments, the encoding device 110 may further include a video source 111 in addition to the video encoder 112 and the input interface 113 .

[0060] The video source 111 may include at least one of a video acquisition device (eg, a video camera), a video archive, a video input interface, and a computer graphics system, wherein the video input interface is used to receive video data from a video content provider, and the computer graphics system is used to generate video data.

[0061] The video encoder 112 encodes the video data from the video source 111 to generate a code stream. The video data may include one or more pictures or a sequence of pictures. The code stream contains the coding information of the picture or the sequence of pictures in the form of a bit stream. The coding information may include the coded picture data and associated data. The associated data may include a sequence parameter set (SPS for short), a picture parameter set (PPS for short) and other syntax structures. The SPS may contain parameters applied to one or more sequences. The PPS may contain parameters applied to one or more pictures. The syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the code stream.

[0062] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored in a storage medium or a storage server for subsequent reading by the decoding device 120.

[0063] In some embodiments, the decoding device 120 includes an input interface 121 and a video decoder 122 .

[0064] In some embodiments, the decoding device 120 may include a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0065] The input interface 121 includes a receiver and / or a modem. The input interface 121 can receive the encoded video data through the channel 130 .

[0066] The video decoder 122 is used to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to the display device 123 .

[0067] The display device 123 displays the decoded video data. The display device 123 may be integrated with the decoding device 120 or external to the decoding device 120. The display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0068] also, Figure 1 This is only an example, and the technical solution of the embodiment of the present application is not limited to Figure 1 For example, the technology of the present application can also be applied to single-sided video encoding or single-sided video decoding.

[0069] The following is an introduction to the video encoding framework involved in the embodiments of the present application.

[0070] Figure 2 2 is a schematic block diagram of a video encoder involved in an embodiment of the present application. It should be understood that the video encoder 200 can be used to perform lossy compression on an image, or can be used to perform lossless compression on an image. The lossless compression can be visually lossless compression or mathematically lossless compression.

[0071] The video encoder 200 can be applied to image data in luminance and chrominance (YCbCr, YUV) format. For example, the YUV ratio can be 4:2:0, 4:2:2 or 4:4:4, Y represents brightness (Luma), Cb (U) represents blue chrominance, Cr (V) represents red chrominance, and U and V represent chrominance (Chroma) for describing color and saturation. For example, in color format, 4:2:0 means that every 4 pixels have 4 luminance components and 2 chrominance components (YYYYCbCr), 4:2:2 means that every 4 pixels have 4 luminance components and 4 chrominance components (YYYYCbCrCbCr), and 4:4:4 means full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0072] For example, the video encoder 200 reads video data, and for each frame of the video data, divides the frame into a number of coding tree units (CTUs). In some examples, CTUs may be referred to as "tree blocks", "largest coding units" (LCU) or "coding tree blocks" (CTB). Each CTU may be associated with a pixel block of equal size within the image. Each pixel may correspond to a luminance (luminance or luma) sample and two chrominance (chrominance or chroma) samples. Therefore, each CTU may be associated with a luminance sample block and two chrominance sample blocks. The size of a CTU is, for example, 128×128, 64×64, 32×32, etc. A CTU may be further divided into a number of coding units (CUs) for encoding, and a CU may be a rectangular block or a square block. CU can be further divided into prediction unit (PU) and transform unit (TU), so that coding, prediction and transform are separated and more flexible in processing. In one example, CTU is divided into CU in quadtree mode, and CU is divided into TU and PU in quadtree mode.

[0073] The video encoder and video decoder may support various PU sizes. Assuming that the size of a particular CU is 2N×2N, the video encoder and video decoder may support PU sizes of 2N×2N or N×N for intra-frame prediction, and support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N or similar sizes for inter-frame prediction. The video encoder and video decoder may also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0074] In some embodiments, Figure 2 As shown, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filter unit 260, a decoded image buffer 270, and an entropy coding unit 280. It should be noted that the video encoder 200 may include more, fewer, or different functional components.

[0075] Optionally, in the present application, the current block may be referred to as a current coding unit (CU) or a current prediction unit (PU), etc. A prediction block may also be referred to as a prediction image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstructed block or an image reconstructed image block.

[0076] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Since there is a strong correlation between adjacent pixels in a frame of a video, the intra-frame prediction method is used in the video coding and decoding technology to eliminate the spatial redundancy between adjacent pixels. Since there is a strong similarity between adjacent frames in a video, the inter-frame prediction method is used in the video coding and decoding technology to eliminate the temporal redundancy between adjacent frames, thereby improving the coding efficiency.

[0077] The inter-frame prediction unit 211 can be used for inter-frame prediction, which may include motion estimation and motion compensation. Motion estimation can search the reference pictures in the list of reference pictures to find the reference block of the image block to be encoded. Motion estimation can generate an index indicating the reference block and a motion vector indicating the spatial displacement between the image block to be encoded and the reference block. Motion estimation can output the index of the reference block and the motion vector as the motion information of the image block to be encoded. Motion compensation can obtain the prediction information of the image block to be encoded based on the motion information of the image block to be encoded. Inter-frame prediction can refer to the image information of different frames. Inter-frame prediction uses motion information to find the reference block from the reference frame, and generates a prediction block based on the reference block to eliminate temporal redundancy; the frame used for inter-frame prediction can be a P frame and / or a B frame, P frame refers to a forward prediction frame, and B frame refers to a bidirectional prediction frame. Inter-frame prediction uses motion information to find the reference block from the reference frame, and generates a prediction block based on the reference block. The motion information includes the reference frame list where the reference frame is located, the reference frame index, and the motion vector. The motion vector can be an integer pixel or a sub-pixel. If the motion vector is a sub-pixel, then it is necessary to use interpolation filtering in the reference frame to make the required sub-pixel block. Here, the integer pixel or sub-pixel block in the reference frame found according to the motion vector is called a reference block. Some technologies will directly use the reference block as a prediction block, while some technologies will generate a prediction block based on the reference block. Generating a prediction block based on the reference block can also be understood as using the reference block as a prediction block and then processing it based on the prediction block to generate a new prediction block.

[0078] The intra-frame prediction unit 212 only refers to the information of the same frame image to predict the pixel information in the current coded image block to eliminate spatial redundancy. The frame used for intra-frame prediction may be an I frame.

[0079] There are multiple prediction modes for intra-frame prediction. Taking the H series of international digital video coding standards as an example, the H.264 / AVC standard has 8 angle prediction modes and 1 non-angle prediction mode, and H.265 / HEVC is extended to 33 angle prediction modes and 2 non-angle prediction modes. The intra-frame prediction modes used by HEVC are Planar, DC, and 33 angle modes, for a total of 35 prediction modes. The intra-frame modes used by VVC are Planar, DC, and 65 angle modes, for a total of 67 prediction modes.

[0080] It should be noted that with the increase of angle modes, intra-frame prediction will be more accurate and more in line with the needs of the development of high-definition and ultra-high-definition digital videos.

[0081] The residual unit 220 may generate a residual block of the CU based on the pixel blocks of the CU and the prediction blocks of the PUs of the CU. For example, the residual unit 220 may generate a residual block of the CU so that each sample in the residual block has a value equal to the difference between the following two: a sample in the pixel blocks of the CU and a corresponding sample in the prediction blocks of the PUs of the CU.

[0082] The transform / quantization unit 230 can quantize the transform coefficients. The transform / quantization unit 230 can quantize the transform coefficients associated with the TU of the CU based on the quantization parameter (QP) value associated with the CU. The video encoder 200 can adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU. Exemplarily, the residual video signal is converted into a transform domain through transformation operations such as DFT and DCT, which are called transform coefficients. The signal in the transform domain is further subjected to a lossy quantization operation, which loses certain information, making the quantized signal conducive to compressed expression. In some video coding standards, there may be more than one transform method to choose from, so the encoder also needs to select one of the transforms for the current encoded CU and inform the decoder. The degree of quantization fineness is usually determined by the quantization parameter (QP). A larger QP value means that coefficients with a larger value range will be quantized to the same output, which usually results in greater distortion and a lower bit rate. Conversely, a smaller QP value means that coefficients with a smaller value range will be quantized to the same output, which usually results in less distortion and a corresponding higher bit rate.

[0083] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform to the quantized transform coefficients, respectively, to reconstruct a residual block from the quantized transform coefficients.

[0084] The reconstruction unit 250 may add the samples of the reconstructed residual block to the corresponding samples of one or more prediction blocks generated by the prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of the CU in this manner, the video encoder 200 may reconstruct the pixel blocks of the CU.

[0085] The loop filter unit 260 is used to process the pixels after inverse transformation and inverse quantization, compensate for the distortion information, and provide a better reference for the subsequent encoded pixels. For example, a deblocking filter operation can be performed to reduce the blocking effect of the pixel blocks associated with the CU. As can be seen from the above, the encoded image can be reconstructed and decoded after inverse quantization, inverse transformation and prediction compensation. Compared with the original image, the reconstructed image has some information different from the original image due to the influence of quantization, resulting in distortion. Filtering the reconstructed image, such as deblocking filter DBF (deblocking filter), SAO (sample adaptive offset) or ALF (adaptive loop filter) and other filters, can effectively reduce the degree of distortion caused by quantization. Since these filtered reconstructed images will be used as references for subsequent encoded images and used to predict future signals, the above filtering operation is also called loop filtering, and filtering operations within the encoding loop.

[0086] The decoded image buffer 270 may store the reconstructed pixel blocks. The inter prediction unit 211 may use the reference image containing the reconstructed pixel blocks to perform inter prediction on PUs of other images. In addition, the intra prediction unit 212 may use the reconstructed pixel blocks in the decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.

[0087] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy coded data. Exemplarily, the quantized transform domain signal will be statistically compressed and encoded according to the frequency of occurrence of each value, and finally a binary (0 or 1) compressed code stream will be output. At the same time, the encoding generates other information, such as the selected mode, motion vector, etc., which also needs to be entropy encoded to reduce the bit rate. In one example, statistical coding can be used as a lossless coding method that can effectively reduce the bit rate required to express the same signal. Common statistical coding methods include variable length coding (VLC) or context-based binary arithmetic coding (CABAC, Content Adaptive Binary Arithmetic Coding).

[0088] Figure 3 It is a schematic block diagram of a video decoder involved in an embodiment of the present application.

[0089] like Figure 3As shown, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transformation unit 330, a reconstruction unit 340, a loop filter unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.

[0090] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse the syntax elements in the bitstream that have been entropy encoded. The prediction unit 320, the inverse quantization / transformation unit 330, the reconstruction unit 340, and the loop filter unit 350 may decode the video data according to the syntax elements extracted from the bitstream, that is, generate decoded video data.

[0091] In some embodiments, the prediction unit 320 includes an intra prediction unit 322 and an inter prediction unit 321 .

[0092] The intra prediction unit 322 may perform intra prediction to generate a prediction block for the PU. The intra prediction unit 322 may use an intra prediction mode to generate a prediction block for the PU based on pixel blocks of spatially neighboring PUs. The intra prediction unit 322 may also determine the intra prediction mode of the PU according to one or more syntax elements parsed from the code stream.

[0093] The inter prediction unit 321 may construct a first reference image list (list 0) and a second reference image list (list 1) according to the syntax elements parsed from the code stream. In addition, if the PU is encoded using inter prediction, the entropy decoding unit 310 may parse the motion information of the PU. The inter prediction unit 321 may determine one or more reference blocks of the PU according to the motion information of the PU. The inter prediction unit 321 may generate a prediction block of the PU according to one or more reference blocks of the PU.

[0094] The inverse quantization / transform unit 330 may inversely quantize (ie, dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use the QP value associated with the CU of the TU to determine the degree of quantization.

[0095] After inverse quantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.

[0096] The reconstruction unit 340 uses the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 may add samples of the residual block to corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0097] The loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking effects of pixel blocks associated with a CU.

[0098] The video decoder 300 may store the reconstructed image of the CU in the decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0099] The basic process of video encoding and decoding is as follows: at the encoding end, a frame of image is divided into blocks, and for the current block, the prediction unit 210 uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block. The residual unit 220 can calculate the residual block based on the original block of the prediction block and the current block, that is, the difference between the original block of the prediction block and the current block, and the residual block can also be called residual information. The residual block can remove information that is not sensitive to the human eye through the transformation and quantization process of the transformation / quantization unit 230 to eliminate visual redundancy. Optionally, the residual block before transformation and quantization by the transformation / quantization unit 230 can be called a time domain residual block, and the time domain residual block after transformation and quantization by the transformation / quantization unit 230 can be called a frequency residual block or a frequency domain residual block. The entropy coding unit 280 receives the quantized change coefficient output by the change quantization unit 230, and can entropy encode the quantized change coefficient and output a bit stream. For example, the entropy coding unit 280 can eliminate character redundancy according to the target context model and the probability information of the binary bit stream.

[0100] At the decoding end, the entropy decoding unit 310 can parse the code stream to obtain the prediction information, quantization coefficient matrix, etc. of the current block. The prediction unit 320 uses intra-frame prediction or inter-frame prediction to generate a prediction block of the current block based on the prediction information. The inverse quantization / transformation unit 330 uses the quantization coefficient matrix obtained from the code stream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or on the block to obtain a decoded image. The encoding end also requires similar operations as the decoding end to obtain a decoded image. The decoded image can also be called a reconstructed image, and the reconstructed image can be used as a reference frame for inter-frame prediction for subsequent frames.

[0101] It should be noted that the block division information determined by the encoder, as well as the mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., are carried in the bitstream when necessary. The decoder parses the bitstream and determines the same block division information, prediction, transformation, quantization, entropy coding, loop filtering, etc. mode information or parameter information as the encoder by analyzing the existing information, thereby ensuring that the decoded image obtained by the encoder is the same as the decoded image obtained by the decoder.

[0102] The above is the basic process of the video codec under the block-based hybrid coding framework. With the development of technology, some modules or steps of the framework or process may be optimized. The present application is applicable to the basic process of the video codec under the block-based hybrid coding framework, but is not limited to the framework and process.

[0103] In the existing hybrid coding framework, each frame of a video is often divided into units of a certain size before the subsequent encoding and decoding process. Figure 4 As shown in the figure, the largest coding unit (CTU) is the basic coding unit in the hybrid coding framework, which usually contains two parts: luminance Y and chrominance UV. Since the U component and V component in chrominance have similar characteristics, they are usually processed in the order of U first and then V using the same coding parameters, and the corresponding coding results of U and V are obtained.

[0104] The existing hybrid coding framework uses traditional loop filters to suppress the distortion of reconstructed images, improve the quality of reconstructed images, and expects to restore the encoded reconstructed images to the original images. However, traditional loop filters are based on manual design and are difficult to effectively reduce the distortion of reconstructed images, leaving a large room for optimization. Due to the excellent performance of deep learning tools in image processing, deep learning-based loop filters are applied to the loop filter module.

[0105] The main technology involved in this application is the filter based on neural network loop filter (NNLF). Figure 5 As shown, by inputting the image to be filtered before filtering into the trained filter, the enhanced image after filtering can be obtained.

[0106] During the training process, neural networks usually use loss functions to constrain the filtered image so that it can be restored to the original image as much as possible. The loss function measures the difference between the filtered value and the true value. The larger the loss value, the greater the difference, and the goal of training is to reduce the loss. For deep learning-based encoding tools, exemplary and commonly used loss functions are: L1 norm loss function, L2 norm loss function and smooth L1 loss function.

[0107] In the actual training process, neural network filters often use data enhancement methods such as rotation, flipping and downsampling to expand the training set samples and improve the generalization ability of the model. However, in the actual application or testing process, the neural network filter does not flip, rotate or downsample the input image to be processed, but directly sends it to the network.

[0108] In the development process of the neural network filter, the neural network filter is first trained on the training set, and the network parameters are fixed when the training is completed. Then, the neural network filter with fixed parameters is used to test on the test set to verify the coding performance of the tool. Ideally, if the data distribution of the test set is exactly the same as that of the training set, the neural network filter can achieve the best coding performance on the test set. However, in actual neural network-based video coding, the image content of the test set is not exactly the same as that of the training set, and there are certain differences in the data distribution characteristics between the two. As shown in Figure 6, the data distribution characteristics of the image content included in the training set are: data distribution characteristics A, data distribution characteristics B, data distribution characteristics C, and data distribution characteristics D, etc., while the data distribution characteristics of the image content included in the test set are: data distribution characteristics B, data distribution characteristics E, data distribution characteristics F, and data distribution characteristics G, etc. Since both the training set and the test set include image content with data distribution characteristic B, the neural network filter can achieve a relatively good filtering effect for the to-be-filtered information of the image content with data distribution characteristic B. However, for the to-be-filtered information of the image content with data distribution characteristic E, data distribution characteristic F, and data distribution characteristic G, since these data distribution characteristics are not included in the training set, the neural network filter does not learn these data distribution characteristics very well, which leads to unsatisfactory filtering effects for the image content with these data distribution characteristics.

[0109] The present application has found that rearranging the image content, such as rotating and flipping, can change the data distribution characteristics of the image content. Fig. 6A After rotating the image content in the test set in Figure 6B The test set shown is compared to Fig. 6A The test set shown has data distribution characteristics that are closer to those of the training set.

[0110] Based on this, in order to solve the technical problem, the embodiment of the present application rearranges the data of the information to be filtered input by the neural network filter, and it is expected that the data distribution characteristics of the information to be filtered are closer to the data distribution characteristics of the training set, so as to give full play to the generalization effect of the neural network filter, thereby improving the image filtering effect and image encoding and decoding performance. Specifically, from M+1 data processing methods, a first data processing method of the information to be filtered is determined, and the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods. The information to be filtered includes a reconstructed image block to be filtered, and the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer; if the first data processing method is one of the M data rearrangement methods, based on the first data processing method, the reconstructed image block in the information to be filtered is rearranged to obtain the rearranged information to be filtered, and the rearranged information to be filtered is filtered through the neural network filter to obtain the rearranged filtered image block of the information to be filtered; based on the first data processing method, the rearranged filtered image of the information to be filtered is reversely rearranged to obtain the filtered image block of the reconstructed image block. That is to say, before filtering the information to be filtered, the present application first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image encoding and decoding performance.

[0111] The technical solutions of the embodiments of the present application are described in detail below through some embodiments. The following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments.

[0112] First, taking the decoding end as an example, the image filtering method provided in the embodiment of the present application is introduced.

[0113] Figure 7 A flowchart of a neural network-based image filtering method provided in an embodiment of the present application. The present application embodiment is applied to Figure 1 or Figure 3 The decoder or decoding device shown in the figure can also be applied to other electronic devices. For the convenience of description, the following description is made by taking the execution subject as a decoding end as an example. Figure 7 As shown, the method of the embodiment of the present application includes:

[0114] S101 . Determine a first data processing method for information to be filtered from M+1 data processing methods.

[0115] The method of the embodiment of the present application can be applied to a video decoding solution or to video post-processing.

[0116] In some embodiments, when applied to a video decoding scheme, the encoder first divides the current image into coding blocks when encoding the current image, and performs block-by-block encoding using the coding blocks as coding units. For example, for the current block to be encoded in the current image, first obtain the prediction value of the current block through inter-frame and / or intra-frame prediction. Then, based on the prediction value of the current block and the current block, obtain the residual value of the current block. The encoder transforms the residual value of the current block to obtain a transformation coefficient. In one example, the encoder does not quantize the transformation coefficient of the current block, but directly encodes the transformation coefficient to obtain a code stream. In another example, the encoder quantizes the transformation coefficient of the current block to obtain a quantization coefficient, and then encodes the quantization coefficient to obtain a code stream.

[0117] During the encoding process, Figure 2 As shown, the encoding end also performs an inverse transformation on the transform coefficients to obtain a residual value, and adds the residual value to the predicted value to obtain a reconstruction value of the current block. Based on the above steps, the reconstruction value of each coding block in the current image can be obtained, and these reconstruction values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoding cache for prediction of subsequent images.

[0118] like Figure 3 As shown, for each block to be decoded in the current image, such as the current block, after the decoder obtains the code stream, it decodes the code stream to obtain the transform coefficient of the current block. In one example, if the encoder quantizes the transform coefficient and then encodes it, the decoder decodes the code stream, obtains the quantization coefficient of the current block, and then dequantizes the quantization coefficient to obtain the transform coefficient of the current image. Then, the decoder performs a decompression on the transform coefficient of the current block to obtain the residual value of the current block. At the same time, the decoder uses an inter-frame and / or intra-frame prediction method to predict the predicted value of the current block. In this way, the predicted value and the residual value of the current block are added to obtain the reconstructed value of the current block. Based on the above steps, the decoder can decode and determine the reconstructed value of each block to be decoded in the current image, and these reconstructed values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the decoder filters the reconstructed image to obtain the decoded image of the current image. In one example, the decoder can store the decoded image in a decoding buffer for prediction of subsequent images. In one example, the decoder can output the decoded image to a display device for display.

[0119] The information to be filtered in the embodiment of the present application includes a reconstructed image block.

[0120] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to filter at least one frame of image in a video. That is, the current image is an image in the video.

[0121] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to decode a single image, that is, the current image is a single image, such as an image generated by an electronic device.

[0122] In an embodiment of the present application, the decoding end uses a neural network filter to filter the reconstructed image.

[0123] In the embodiment of the present application, the data distribution characteristics may also be referred to as data distribution features, which mainly refer to the data distribution concentration trend of the image, the degree of data distribution dispersion, etc. When the image contents of two images are different, their data distribution characteristics may also be different. For example, one image is animal A doing action B, and the other image is animal A doing action C. The data distribution characteristics of these two images may be different. Therefore, in the embodiment of the present application, if the image content of the reconstructed image block is different from the image content of the training data, the data distribution characteristics of the reconstructed image block may also be different from the data distribution characteristics of the training data. In this way, if the neural network filter obtained by training the training data is directly used to filter the reconstructed image block, the filtering effect may be unsatisfactory. Therefore, in the embodiment of the present application, before using the neural network filter to filter the information to be filtered, it is first determined whether it is necessary to rearrange the data of the reconstructed image in the information to be filtered.

[0124] When filtering the reconstructed image, the embodiment of the present application can directly filter the entire reconstructed image as a filtering unit, or divide the reconstructed image into multiple blocks, and filter each block as a filtering unit. For ease of description, the embodiment of the present application uses a reconstructed image block as a filtering unit, and the reconstructed image block can be a reconstructed image (i.e., a frame) or a partial area of ​​a reconstructed image.

[0125] In some embodiments, if the reconstructed image block is a partial area of ​​the reconstructed image, the division of the reconstructed image block includes but is not limited to the following examples:

[0126] In a possible implementation, the reconstructed image block includes at least one CTU of the reconstructed image. That is, at least one CTU of the reconstructed image is divided into a reconstructed image block.

[0127] In some examples, such as Fig. 8A As shown, the reconstructed image block includes a CTU of the reconstructed image, that is, a CTU of the reconstructed image is divided into a reconstructed image block.

[0128] In another example, Figure 8B As shown, the reconstructed image block includes 4 CTUs of the reconstructed image, that is, the 4 CTUs of the reconstructed image are divided into one reconstructed image block.

[0129] In an example, multiple CTUs such as 2 CTUs or 3 CTUs of the reconstructed image may be divided into a reconstructed image block. The multiple CTUs may be multiple CTUs in the horizontal direction or multiple CTUs in the vertical direction. Optionally, the multiple CTUs may be adjacent, or non-adjacent, or partially adjacent and partially non-adjacent.

[0130] In another possible implementation, the reconstructed image block to be filtered is a preset image region of the reconstructed image, that is, a preset image region of the reconstructed image is used as a reconstructed image block.

[0131] The embodiment of the present application does not limit the specific shape and size of the preset image area.

[0132] In one example, the preset image area includes at least one incomplete number of CTUs of the reconstructed image, that is, the at least one incomplete number of CTUs of the reconstructed image is divided into a reconstructed image block. Figure 8C For example, a CTU of the reconstructed image and three defective CTUs are divided into a reconstructed image block. Fig.8D As shown, the four incomplete CTUs of the reconstructed image are divided into a reconstructed image block.

[0133] In some embodiments, the preset image area is a fixed area. For example, at each filtering, the reconstructed image block to be filtered in the reconstructed image is obtained according to the preset image area. At this time, the size and shape of the image block input to the neural network filter each time are the same, and they are all preset image areas.

[0134] In some embodiments, the above-mentioned preset image area is a variable value. For example, during the first filtering, a reconstructed image block to be filtered in the reconstructed image is obtained according to the first preset image area, and is input into the neural network filter as an input image block for filtering. During the second filtering, a reconstructed image block to be filtered in the reconstructed image is obtained according to the second preset image area, and is input into the neural network filter as an input image block for filtering, and so on. In an example of this embodiment, the decoding end can divide the reconstructed image into several reconstructed image blocks to be filtered, and the shapes and sizes of these several reconstructed image blocks to be filtered can be the same, or different, or partially the same and partially different.

[0135] The following is an introduction to the process of determining the first data processing method for the information to be filtered.

[0136] In the embodiment of the present application, the first data processing method mentioned above is one of M+1 data processing methods, and the M+1 data processing methods include a data non-rearrangement method and M data re-arrangement methods.

[0137] The data non-rearrangement method refers to not re-arranging the filtered information, and directly using the filtered data as the input of the neural network filter.

[0138] The data rearrangement method refers to the data rearrangement method according to which part or all of the information to be filtered is input into the neural network filter for filtering. Part or all of the information to be filtered includes the reconstructed image block, that is, in the embodiment of the present application, when the data is rearranged for the information to be filtered, at least the data rearrangement of the reconstructed image block in the information to be filtered is included.

[0139] The embodiments of the present application do not limit the specific type of data rearrangement method.

[0140] In some embodiments, the M data rearrangement methods include at least one of the following: rotation method, horizontal flipping and then rotating method, vertical flipping and then rotating method, diagonal flipping and then rotating method, downsampling and then rotating method, and upsampling and then rotating method.

[0141] The rotation method includes left rotation of N*90° and right rotation of N*90°, where N is an integer positive number. For example, the reconstructed image block and other information are rotated 90°, 180°, 270°, etc. to the left or right.

[0142] The method of horizontal flipping and then rotating includes horizontal flipping and then rotating to the left by N*90°, and horizontal flipping and then rotating to the right by N*90°, where N is an integer positive number. For example, the information such as the reconstructed image block is horizontally flipped first, and then rotated to the left or right by 90°, 180°, 270°, etc.

[0143] The method of first vertically flipping and then rotating includes first vertically flipping and then rotating to the left by N*90°, and first vertically flipping and then rotating to the right by N*90°, where N is an integer positive number. For example, the information such as the reconstructed image block is first vertically flipped and then rotated to the left or right by 90°, 180°, 270°, etc.

[0144] The method of first flipping diagonally and then rotating includes first flipping diagonally and then rotating to the left by N*90°, and first flipping diagonally and then rotating to the right by N*90°, where N is an integer positive number. For example, the information such as the reconstructed image block is first flipped diagonally and then rotated to the left or right by 90°, 180°, 270°, etc.

[0145] The downsampling and then rotating method includes downsampling first and then rotating to the left by N*90°, downsampling first and then rotating to the right by N*90°, where N is an integer positive number. For example, the reconstructed image block and other information are downsampled first and then rotated to the left or right by 90°, 180°, 270°, etc.

[0146] The upsampling and then rotating method includes upsampling first and then rotating to the left by N*90°, and upsampling first and then rotating to the right by N*90°, where N is a positive number. For example, the reconstructed image block and other information are first upsampled and then rotated to the left or right by 90°, 180°, 270°, etc.

[0147] It should be noted that the M data rearrangement methods in the embodiment of the present application may include other data rearrangement methods in addition to the at least one data rearrangement method mentioned above, namely, rotation and performing any other preprocessing operations before rotation, and the embodiment of the present application does not limit this.

[0148] In one example, M+1 data processing methods of the embodiment of the present application are shown in Table 1:

[0149] Table 1

[0150] 0 Data not rearranged 1 Rotate 90° left 2 Rotate right 90° 3 Flip horizontally and then rotate 90° to the left 4 Flip horizontally and then rotate right 90° …… ……

[0151] In the embodiment of the present application, before filtering the information to be filtered, the decoding end first determines a first data processing method for the information to be filtered from M+1 data processing methods.

[0152] The embodiment of the present application does not limit the specific manner in which the decoding end determines the first data processing manner for the information to be filtered from the M+1 data processing manners.

[0153] Method 1: if the reconstructed image block is a partial area of ​​the reconstructed image, a reference image block of the reconstructed image block is determined, and then the data processing method of the reference image block is inherited.

[0154] For example, if the prediction method corresponding to the reconstructed image block is inter-frame prediction, the reference image block of the reconstructed image block is determined in the reference image of the current image, and then the data processing method of the reference image block is determined as the first data processing method of the reconstructed image block.

[0155] For another example, if the prediction method corresponding to the reconstructed image block is intra-frame prediction, a reference image block of the reconstructed image block is determined in the reconstructed image, and then the data processing method of the reference image block is determined as the first data processing method of the reconstructed image block.

[0156] In the second method, if the reconstructed image block is a partial area of ​​the reconstructed image and is not the first row of the image block of the reconstructed image, that is, the area around the reconstructed image block includes a filtered area, then the first data processing method can be determined from the M+1 data processing methods through the filtered area around the reconstructed image block.

[0157] For example, Fig. 9 As shown, the filtered area around the reconstructed image block includes the upper and left areas of the reconstructed image block, and optionally may also include the upper left and upper right areas of the reconstructed image block. Therefore, based on part or all of these areas, a data processing method can be selected from M+1 data processing methods as the first data processing method. Specifically, the M+1 data processing methods are used to process the filtered area A respectively, so as to select a data processing method with the minimum filtering cost as the first data processing method. The filtered area A includes at least one of the upper, left, upper left and upper right areas of the reconstructed image block.

[0158] Specifically, first, the filtered area A is not rearranged, and is directly input into the neural network filter for filtering, to obtain a filtered image block 0 without data rearrangement. The filtered image block 0 is compared with the filtered area A to determine the filtering cost 0 corresponding to the data without data rearrangement.

[0159] The embodiment of the present application does not limit the specific calculation method of the above-mentioned filtering cost. For example, the above-mentioned filtering cost can be a rate-distortion cost (RDO) or an approximate cost, such as SSD, STAD or SAD.

[0160] In a possible implementation, for each of the M data rearrangement methods, the data rearrangement method is used to rearrange the data of the filtered area A, and the filtered area A after the data rearrangement is input into the neural network filter for filtering to obtain a filtered image block under the data rearrangement method. After the filtered image block is reversely converted using the data rearrangement method, it is compared with the filtered area A to determine the corresponding filtering cost under the data rearrangement method. Referring to the above example, the filtering cost corresponding to each of the M data rearrangement methods can be determined. In this way, the filtering cost corresponding to the above data non-rearrangement method is compared with the filtering costs corresponding to the M data rearrangement methods, and the one with the smallest filtering cost is selected as the first data processing method.

[0161] For example, Fig.10As shown, assuming that the M data rearrangement methods include rotating 90° to the left and rotating 90° to the right, the decoder directly inputs the filtered area A into the neural network filter to obtain the filtered image block 0 in the data non-rearrangement method. Compare the filtered image block 0 in the data non-rearrangement method with the filtered area A, and calculate the filtering cost corresponding to the data non-rearrangement method. Similarly, the decoder rotates the filtered area A to the left by 90° and inputs it into the neural network filter to obtain the filtered image block rotated 90° to the left. Then, the filtered image block rotated 90° to the left is rotated 90° to the right to obtain the filtered image block rotated 90° to the left. Compare the filtered image block rotated 90° to the left with the filtered area A, and calculate the filtering cost corresponding to the left rotation of 90°. Similarly, the decoder rotates the filtered area A to the right by 90° and inputs it into the neural network filter to obtain the filtered image block rotated 90° to the right. Then, the filtered image block rotated 90° to the right is rotated 90° to the left to obtain the filtered image block rotated 90° to the right. The filtered image block in the 90° right rotation mode is compared with the filtered area A, and the filtering cost corresponding to the 90° right rotation mode is calculated. The decoding end determines the data processing mode with the smallest filtering cost among the data non-rearrangement mode, the 90° right rotation mode, and the 90° left rotation mode as the first data processing mode.

[0162] In another possible implementation, the decoding end calculates the filtering cost for the M data rearrangement methods in a preset order. Specifically, firstly, the first data rearrangement method among the M data rearrangement methods is used to rearrange the data of the filtered area A, and the filtered area A after the data rearrangement is input into the neural network filter for filtering to obtain a first filtered image block, and then the first data rearrangement method is used to reversely rearrange the data of the first filtered image block to obtain a filtered image block 1 under the first data rearrangement method. The filtered image block 1 is compared with the filtered area A to determine the corresponding filtering cost 1 of the first data rearrangement method. Then, the filtering cost 1 is compared with the above-mentioned filtering cost 0. If the filtering cost 1 is less than the filtering cost 0, the first data rearrangement method is determined as the first data rearrangement method, and no subsequent data rearrangement methods are attempted. If the filter cost 1 is greater than or equal to the filter cost 0, the second data rearrangement method among the M data rearrangement methods is used to rearrange the data of the filtered area A, and the filter cost 2 corresponding to the second data rearrangement method is calculated, and the filter cost 2 is compared with the filter cost 0. If the filter cost 2 is less than the filter cost 0, the second data rearrangement method is determined as the first data processing method. If the filter cost 2 is greater than or equal to the filter cost 0, the third data rearrangement method among the M data rearrangement methods is used to rearrange the data of the filtered area A, and so on, until all the M data rearrangement methods are traversed. If the filter costs corresponding to the M data rearrangement methods are all greater than or equal to the filter cost 0 corresponding to the data non-rearrangement method, the data non-rearrangement method is determined as the first data processing method.

[0163] In a third method, after the encoding end determines the first data processing method, the encoding end indicates the first data processing method to the decoding end.

[0164] At this time, the above S101 includes the following step S101-A:

[0165] S101-A. Determine a first data processing method from M+1 data processing methods by decoding a bit stream.

[0166] In an embodiment of the present application, the encoding end directly indicates the first data processing method to the decoding end. The decoding end can obtain the first data processing method by decoding the code stream without performing other additional calculations, thereby reducing the calculation complexity of the decoding end and improving the decoding performance.

[0167] In some embodiments, the first data processing method is a data processing method with the smallest filtering cost among the M+1 data processing methods. For example, the encoding end determines the filtering cost corresponding to each of the M+1 data processing methods, and then selects a data processing method with the smallest filtering cost as the first data processing method.

[0168] In some embodiments, the first data processing method is any one of the M data rearrangement methods whose filtering cost is less than the filtering cost of the data non-rearrangement method. For example, the encoding end first determines the filtering cost corresponding to the data non-rearrangement method, and then determines the filtering cost corresponding to the selected data rearrangement method starting from the first data rearrangement method of the M data rearrangement methods in a preset order. If the filtering cost corresponding to the data rearrangement method is less than the filtering cost corresponding to the data non-rearrangement method, then stop trying other data rearrangement methods, and directly use the data rearrangement method as the first data processing method, thereby improving the image filtering effect while reducing the data processing amount and improving the encoding performance.

[0169] In the embodiment of the present application, the implementation methods of the above S101-A include but are not limited to the following:

[0170] Mode 1, the above S101-A includes S101-A-a1 and S101-A-a2:

[0171] S101-A-a1, decoding the code stream to obtain a first flag;

[0172] S101-A-a2. Based on the first flag, obtain a first data processing method from M+1 data processing methods.

[0173] In the first mode, the encoder writes a first flag in the code stream, and the first flag is used to indicate whether the information to be filtered needs to be rearranged during filtering. For example, if the encoder determines that the information to be filtered needs to be rearranged during filtering, the value of the first flag is set to yes (for example, 1) and then written into the code stream. If the encoder determines that the information to be filtered does not need to be rearranged during filtering, the value of the first flag is set to no (for example, 0) and then written into the code stream.

[0174] In this way, the decoding end decodes the code stream to obtain a flag, and then determines the first data processing method based on the first flag.

[0175] For example, if the first flag indicates that the information to be filtered does not need to be rearranged during filtering, for example, the value of the first flag is 0 indicating that the information to be filtered does not need to be rearranged during filtering, the decoding end determines the data non-rearrangement mode as the first data processing mode.

[0176] For another example, if the first flag indicates that the information to be filtered is to be rearranged during filtering, for example, the value of the first flag is 1, indicating that the information to be filtered is to be rearranged during filtering, then the decoding end continues to decode the code stream to obtain a second flag, and the second flag is used to indicate the index of the data rearrangement method of the information to be filtered. The decoding end can determine the first data processing method from the M data rearrangement methods based on the second flag.

[0177] The embodiment of the present application does not limit the specific forms of expression of the first mark and the second mark.

[0178] In an example, the corresponding relationship between the values ​​of the first flag and the second flag and the data processing method is shown in Table 2:

[0179] Table 2

[0180]

[0181] As shown in Table 2, the decoding end first decodes the code stream to obtain the first flag. If the value of the first flag is 0, it means that the information to be filtered does not need data rearrangement during filtering, and the decoding end determines the data non-rearrangement mode as the first data processing mode. If the value of the first flag is 1, it means that the information to be filtered needs data rearrangement during filtering, and further decodes the code stream to obtain the second flag, and determines the first data processing mode based on the value of the second flag. For example, if the value of the second flag is 00, it means that the first data processing mode is a 90° left rotation mode.

[0182] In the method 1, the encoding end indicates the first data processing method through the first flag and the second flag.

[0183] Mode 2: In this mode, the encoding end can replace the first flag and the second flag with a flag, namely, a third flag. That is, the third flag can indicate whether the information to be filtered is to be rearranged during filtering, and can also indicate a specific data rearrangement method. At this time, the above S101-A includes S101-A-b1 and S101-A-b2:

[0184] S101-A-b1, decoding the code stream to obtain a third flag, where the third flag is used to indicate a first data processing mode;

[0185] S101-A-b2. Based on the third flag, determine a first data processing method from M+1 data processing methods.

[0186] In this mode 2, the encoder directly indicates the index of the first data processing mode to the decoder through the third flag. Specifically, after the encoder determines the first data processing mode of the information to be filtered from the M+1 data processing modes, the encoder indicates the first data processing mode to the encoder through the third flag. For example, if the first data processing mode is a data non-rearrangement mode, the encoder sets the third flag to a first value (for example, 0); if the first data processing mode is a data rearrangement mode, the third flag is set to another value. For example, when the first data processing mode is a 90° rotation to the left, the value of the third flag is set to 100.

[0187] In this way, the decoding end obtains the third flag by decoding the code stream, and then determines the first data processing mode from among M+1 data processing modes based on the value of the third flag.

[0188] The embodiment of the present application does not limit the specific form of the third mark.

[0189] In an example, the correspondence between the third flag value and the data processing method is shown in Table 3:

[0190] Table 3

[0191]

[0192] As shown in Table 3, the decoding end first decodes the code stream to obtain the third flag. If the value of the third flag is 0, it means that the information to be filtered does not need data rearrangement during filtering, and then the decoding end determines the data non-rearrangement mode as the first data processing mode. If the value of the third flag is 100, it means that the information to be filtered needs data rearrangement during filtering, and the first data processing mode is to rotate 90° to the left. If the value of the third flag is 110, it means that the first data processing mode is to flip horizontally first and then rotate 90° to the left. In this way, the decoding end can query the above Table 3 through the value of the third flag to obtain the first data processing mode.

[0193] It should be noted that if the reconstructed image block is a reconstructed image, the first flag, the second flag and the third flag may be picture-level flags. If the reconstructed image block is a partial area of ​​a reconstructed image, the first flag, the second flag and the third flag may be block-level flags.

[0194] The above describes the specific process of the decoding end determining the first data processing method for the information to be filtered. After the decoding end determines the first data processing method, the following step S102 is executed.

[0195] S102. If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data is rearranged on the reconstructed image block to obtain rearranged information to be filtered, and the rearranged information to be filtered is filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered.

[0196] Based on the above steps, the decoding end determines a first data processing method. If the first data processing method is a data non-rearrangement method, the decoding end does not rearrange the data on the information to be filtered, but instead filters the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

[0197] If the first data processing method is one of the M data rearrangement methods, it means that data rearrangement is required first before the information to be filtered is input into the neural network filter.

[0198] In some embodiments, the information to be filtered only includes the reconstructed image blocks.

[0199] In some embodiments, in addition to the reconstructed image block, the information to be filtered also includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type (slice type) of the reconstructed image block, and quantization parameter information. Optionally, the quantization parameter information includes a slice QP (slice level quantization parameter) and a base QP (basic quantization parameter).

[0200] In the embodiment of the present application, if the first data processing method is a data rearrangement method, the decoding end may rearrange the data of the information to be filtered in at least the following ways:

[0201] In the first method, the decoding end only rearranges the data of the reconstructed image block in the information to be filtered, and does not rearrange the data for other information in the information to be filtered. For example, the information to be filtered includes the reconstructed image block, the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block, and other information. The decoding end adopts the first data processing method to rearrange the data of the reconstructed image block, and does not rearrange the data for the predicted image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block corresponding to the reconstructed image block. Since the predicted image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block corresponding to the reconstructed image block are all auxiliary information of the reconstructed image block, data rearrangement of this information is not required, which can reduce the data rearrangement pressure of the decoding end, thereby improving the decoding efficiency of the decoding end.

[0202] The second method is to rearrange the data of the reconstructed image block based on the first data processing method, and to rearrange the data of at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information based on the first data processing method to obtain the rearranged filtered data.

[0203] In an example of the second method, the decoding end rearranges all information in the information to be filtered based on the first data processing method. For example, the first data processing method is to rotate 90° to the left, and the information to be filtered includes information such as the reconstructed image block, the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type and the quantization parameter information. Then, the decoding end rotates the reconstructed image block, the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type and the quantization parameter information to the left by 90° to obtain the rearranged filtered data.

[0204] In an example of the second method, the decoding end rearranges the data of the reconstructed image block based on the first data processing method, rearranges the data of the predicted image block based on the first data processing method, and does not rearrange information such as the boundary strength image block, frame type and quantization parameter information, to obtain the rearranged filtered data.

[0205] In an example of the second method, the decoding end rearranges the data of the reconstructed image block based on the first data processing method, rearranges the data of the boundary strength image block based on the first data processing method, and does not rearrange the information such as the predicted image block, frame type and quantization parameter information, to obtain the rearranged filtered data.

[0206] The third method is to rearrange the data of the reconstructed image block based on the first data processing method, and to rearrange the data of at least one of the predicted image block and the boundary strength image block based on the first data processing method, and skip the data rearrangement of at least one of the frame type and quantization parameter information to obtain the rearranged filtered data.

[0207] In an example of the third method, if the information to be filtered includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information, the decoding end rearranges the data of the reconstructed image block based on the first data processing method, rearranges the data of the predicted image block and the boundary strength image block based on the first data processing method, and skips data rearrangement for the frame type and quantization parameter information (i.e., no data rearrangement is performed) to obtain the rearranged filtered data.

[0208] In an example of the third method, if the information to be filtered includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information, the decoding end rearranges the data of the reconstructed image block, the predicted image block, the boundary strength image block and the frame type based on the first data processing method, and skips data rearrangement for the quantization parameter information (i.e., no data rearrangement is performed), to obtain the rearranged filtered data.

[0209] In an example of the third method, if the information to be filtered includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information, the decoding end rearranges the reconstructed image block, the predicted image block, the boundary strength image block and the quantization parameter information based on the first data processing method, skips data rearrangement for the frame type (i.e., does not perform data rearrangement), and obtains the rearranged filtered data.

[0210] In an example of the third method, if the information to be filtered includes a reconstructed image block, a predicted image block of the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information, the decoding end rearranges the data of the reconstructed image block and the predicted image block based on the first data processing method, and skips data rearrangement (i.e., does not perform data rearrangement) for the boundary strength image block, the quantization parameter information and the frame type, to obtain the rearranged filtered data.

[0211] The decoding end adopts the first data processing method to rearrange the reconstructed image blocks in the information to be filtered, or rearrange the reconstructed image blocks and other information in the filtering information to obtain the rearranged filtering information. Then, the rearranged filtering information is input into the neural network filter, and the rearranged information to be filtered is filtered by the neural network filter to obtain the rearranged filtered image blocks of the information to be filtered.

[0212] In an embodiment of the present application, before filtering the information to be filtered, the decoding end first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks and other information in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image encoding and decoding performance.

[0213] S103 . Based on the first data processing method, reversely rearrange the data of the rearranged filtered image of the information to be filtered to obtain a filtered image block of a reconstructed image block.

[0214] Due to the above steps, the reconstructed image blocks and other information in the information to be filtered are rearranged, so that the neural network filter filters the rearranged information to be filtered, and the result is a filtered image of the rearranged information to be filtered. The data distribution characteristics of the filtered image of the rearranged information to be filtered are inconsistent with the data distribution characteristics of the information to be filtered. Therefore, in an embodiment of the present application, after the decoding end uses the first data processing method to rearrange the reconstructed image blocks and other information in the information to be filtered, and obtains the rearranged filtered image of the information to be filtered, it is also necessary to use the first data processing method to reversely rearrange the data of the rearranged filtered image of the information to be filtered, and obtain the filtered image block of the reconstructed image block.

[0215] For example, the first data processing method is to rotate 90° to the left. At this time, the decoding end needs to rotate the rearranged filtered image of the to-be-filtered information to the right by 90° to obtain a filtered image block of the reconstructed image block.

[0216] For another example, the first data processing method is to rotate right by 90°. At this time, the decoding end needs to rotate the filtered image of the rearranged information to be filtered by 90° to the left to obtain a filtered image block of the reconstructed image block.

[0217] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block under a Y component, a reconstructed image block under a U component, and a reconstructed image block under a V component. That is, the adaptive data rearrangement method of the embodiment of the present application can be applied to at least one of the luminance component Y and the chrominance components U and V.

[0218] The above introduces the filtering process of a reconstructed image block in the reconstructed image. The filtering process of other reconstructed image blocks to be filtered in the reconstructed image can refer to the filtering process of the above reconstructed image blocks, and finally a filtered reconstructed image is obtained.

[0219] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video decoder, for example, in loop filtering, where the filtered reconstructed image is stored in a decoding cache for subsequent image decoding, and the filter output affects the video decoding. Since the embodiment of the present application provides improved filtering quality of the reconstructed image, the decoding effect of the video can be improved when subsequent decoding is performed based on the reconstructed image with better quality.

[0220] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video post-processing product. For example, it can be applied to a display end, that is, the filtered reconstructed image is directly displayed without being placed in a decoding cache, and the filter output does not affect video decoding.

[0221] In some embodiments, the image decoding method of the embodiments of the present application can also be used in any module that uses a neural network in neural network-based video encoding, such as a neural network super-resolution module, a neural network intra-frame prediction module, etc., by simply replacing the reconstructed image block with the corresponding image block.

[0222] The neural network-based image filtering method provided in the embodiment of the present application, the decoding end determines a first data processing method for information to be filtered from M+1 data processing methods, the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​a reconstructed image, and M is a positive integer; if the first data processing method is one of the M data rearrangement methods, based on the first data processing method, the reconstructed image block in the information to be filtered is rearranged to obtain the rearranged information to be filtered, and the rearranged information to be filtered is filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; based on the first data processing method, the filtered image of the rearranged information to be filtered is reversely rearranged to obtain a filtered image block of the reconstructed image block. That is to say, before filtering the information to be filtered, the present application first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image decoding performance.

[0223] The above describes the image filtering method of the embodiment of the present application by taking the decoding end as an example. The following describes the image filtering method of the embodiment of the present application by taking the encoding end as an example.

[0224] Fig.11 A flowchart of a neural network-based image filtering method provided in an embodiment of the present application. The present application embodiment is applied to Figure 1 or Figure 2 The encoder or other electronic device shown in the figure is used as an example for convenience of description. Fig.11 As shown, the method of the embodiment of the present application includes:

[0225] S201 . Determine a first data processing method for information to be filtered from M+1 data processing methods.

[0226] In an embodiment of the present application, when encoding the current image, the encoding end divides the current image into coding blocks, and performs block-by-block encoding using the coding blocks as encoding units. For example, for the current block to be encoded in the current image, first, a prediction value of the current block is obtained through inter-frame and / or intra-frame prediction. Then, based on the prediction value of the current block and the current block, a residual value of the current block is obtained. The encoding end transforms the residual value of the current block to obtain a transformation coefficient. In one example, the encoding end does not quantize the transformation coefficient of the current block, but directly encodes the transformation coefficient to obtain a code stream. In another example, the encoding end quantizes the transformation coefficient of the current block to obtain a quantization coefficient, and then encodes the quantization coefficient to obtain a code stream.

[0227] During the encoding process, Figure 2 As shown, the encoding end also performs an inverse transformation on the transform coefficients to obtain a residual value, and adds the residual value to the predicted value to obtain a reconstruction value of the current block. Based on the above steps, the reconstruction value of each coding block in the current image can be obtained, and these reconstruction values ​​constitute the reconstructed image of the current image. Then, in order to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image of the current image. In one example, the decoded image can be stored in a decoding cache for prediction of subsequent images.

[0228] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to filter at least one frame of image in a video. That is, the current image is an image in the video.

[0229] In some embodiments, the image filtering method proposed in the embodiments of the present application can be used to decode a single image, that is, the current image is a single image, such as an image generated by an electronic device.

[0230] In an embodiment of the present application, the encoding end uses a neural network filter to filter the reconstructed image.

[0231] In the embodiment of the present application, the data distribution characteristics may also be referred to as data distribution features, which mainly refer to the data distribution concentration trend of the image, the degree of data distribution dispersion, etc. When the contents of two images are different, their data distribution characteristics are usually different. One image is animal A doing action B, and the other image is animal A doing action C. The data distribution characteristics of these two images may be different. Therefore, in the embodiment of the present application, if the image content of the information to be filtered is different from the image content of the training data, the data distribution characteristics of the information to be filtered may be different from the data distribution characteristics of the training data. In this way, if the neural network filter trained directly using the training data is used to directly filter the information to be filtered, it may result in an unsatisfactory filtering effect. Therefore, in the embodiment of the present application, before using the neural network filter to filter the information to be filtered, it is first determined whether it is necessary to rearrange the data of the reconstructed image in the information to be filtered.

[0232] The information to be filtered in the embodiment of the present application includes a reconstructed image block.

[0233] When filtering the reconstructed image, the embodiment of the present application can directly filter the entire reconstructed image as a filtering unit, or divide the reconstructed image into multiple blocks, and filter each block as a filtering unit. For ease of description, the embodiment of the present application uses a reconstructed image block as a filtering unit, and the reconstructed image block can be a reconstructed image (i.e., a frame) or a partial area of ​​a reconstructed image.

[0234] In some embodiments, if the reconstructed image block is a partial area of ​​the reconstructed image, the division of the reconstructed image block includes but is not limited to the following examples:

[0235] In a possible implementation, the reconstructed image block includes at least one CTU of the reconstructed image. That is, at least one CTU of the reconstructed image is divided into a reconstructed image block.

[0236] In some examples, such as Fig. 8A As shown, the reconstructed image block includes a CTU of the reconstructed image, that is, a CTU of the reconstructed image is divided into a reconstructed image block.

[0237] In another example, Figure 8B As shown, the reconstructed image block includes 4 CTUs of the reconstructed image, that is, the 4 CTUs of the reconstructed image are divided into one reconstructed image block.

[0238] In an example, multiple CTUs such as 2 CTUs or 3 CTUs of the reconstructed image may be divided into a reconstructed image block. The multiple CTUs may be multiple CTUs in the horizontal direction or multiple CTUs in the vertical direction. Optionally, the multiple CTUs may be adjacent, or non-adjacent, or partially adjacent and partially non-adjacent.

[0239] In another possible implementation, the reconstructed image block to be filtered is a preset image region of the reconstructed image, that is, a preset image region of the reconstructed image is used as a reconstructed image block.

[0240] The embodiment of the present application does not limit the specific shape and size of the preset image area.

[0241] In one example, the preset image area includes at least one incomplete number of CTUs of the reconstructed image, that is, the at least one incomplete number of CTUs of the reconstructed image is divided into a reconstructed image block. Figure 8C For example, a CTU of the reconstructed image and three defective CTUs are divided into a reconstructed image block. Fig.8D As shown, the four incomplete CTUs of the reconstructed image are divided into a reconstructed image block.

[0242] In some embodiments, the preset image area is a fixed area. For example, at each filtering, the reconstructed image block to be filtered in the reconstructed image is obtained according to the preset image area. At this time, the size and shape of the image block input to the neural network filter each time are the same, and they are all preset image areas.

[0243] In some embodiments, the above-mentioned preset image area is a variable value. For example, during the first filtering, a reconstructed image block to be filtered in the reconstructed image is obtained according to the first preset image area, and is input into the neural network filter as an input image block for filtering. During the second filtering, a reconstructed image block to be filtered in the reconstructed image is obtained according to the second preset image area, and is input into the neural network filter as an input image block for filtering, and so on. In an example of this embodiment, the decoding end can divide the reconstructed image into several reconstructed image blocks to be filtered, and the shapes and sizes of these several reconstructed image blocks to be filtered can be the same, or different, or partially the same and partially different.

[0244] The following is an introduction to the process of determining the first data processing method for the information to be filtered.

[0245] In the embodiment of the present application, the first data processing method mentioned above is one of M+1 data processing methods, and the M+1 data processing methods include a data non-rearrangement method and M data re-arrangement methods.

[0246] The data non-rearrangement method refers to not re-arranging the filtered information, and directly using the filtered data as the input of the neural network filter.

[0247] The data rearrangement method refers to the data rearrangement method according to which part or all of the information to be filtered is input into the neural network filter for filtering. Part or all of the information to be filtered includes the reconstructed image block, that is, in the embodiment of the present application, when the data is rearranged for the information to be filtered, at least the data rearrangement of the reconstructed image block in the information to be filtered is included.

[0248] The embodiments of the present application do not limit the specific type of data rearrangement method.

[0249] In some embodiments, the M data rearrangement methods include at least one of the following: rotation method, horizontal flipping and then rotating method, vertical flipping and then rotating method, diagonal flipping and then rotating method, downsampling and then rotating method, and upsampling and then rotating method.

[0250] The rotation modes include left rotation of N*90° and right rotation of N*90°, where N is a positive number.

[0251] The method of flipping horizontally first and then rotating includes flipping horizontally first and then rotating to the left by N*90 degrees, and flipping horizontally first and then rotating to the right by N*90 degrees, where N is an integer positive number.

[0252] The method of first flipping vertically and then rotating includes first flipping vertically and then rotating to the left by N*90 degrees, and first flipping vertically and then rotating to the right by N*90 degrees, wherein N is an integer positive number.

[0253] The diagonal flipping and then rotating method includes diagonal flipping and then rotating to the left by N*90 degrees, and diagonal flipping and then rotating to the right by N*90 degrees, wherein N is an integer positive number.

[0254] The downsampling and then rotating method includes downsampling and then rotating to the left by N*90°, and downsampling and then rotating to the right by N*90°, where N is an integer positive number.

[0255] The upsampling-then-rotating method includes upsampling-then-rotating to the left by N*90° and upsampling-then-rotating to the right by N*90°, where N is an integer positive number.

[0256] It should be noted that the M data rearrangement methods in the embodiment of the present application may include other data rearrangement methods in addition to the at least one data rearrangement method mentioned above, namely, rotation and performing any other preprocessing operations before rotation, and the embodiment of the present application does not limit this.

[0257] In one example, M+1 data processing methods of an embodiment of the present application are shown in Table 1.

[0258] In the embodiment of the present application, before filtering the information to be filtered, the encoding end first determines a first data processing method for the information to be filtered from M+1 data processing methods.

[0259] The embodiment of the present application does not limit the specific manner in which the encoding end determines the first data processing manner for the information to be filtered from the M+1 data processing manners.

[0260] Method 1: if the reconstructed image block is a partial area of ​​the reconstructed image, a reference image block of the reconstructed image block is determined, and then the data processing method of the reference image block is inherited.

[0261] For example, if the prediction method corresponding to the reconstructed image block is inter-frame prediction, the reference image block of the reconstructed image block is determined in the reference image of the current image, and then the data processing method of the reference image block is determined as the first data processing method of the reconstructed image block.

[0262] For another example, if the prediction method corresponding to the reconstructed image block is intra-frame prediction, a reference image block of the reconstructed image block is determined in the reconstructed image, and then the data processing method of the reference image block is determined as the first data processing method of the reconstructed image block.

[0263] Mode 2, the above S201 includes the following steps S201-A1 and S201-A2:

[0264] S201-A1, determining filtering costs corresponding to M+1 data processing methods respectively;

[0265] S201-A2: Determine a data processing method with the smallest filtering cost among the M+1 data processing methods as the first data processing method.

[0266] In an embodiment of the present application, the encoding end determines the filtering cost corresponding to the data non-rearrangement method among the M+1 data processing methods. Specifically, the filtering information to be filtered is filtered through a neural network filter to obtain a filtered image block under the data non-rearrangement method; based on the filtered image block under the data non-rearrangement method and the original image block of the reconstructed image block, the filtering cost corresponding to the data non-rearrangement method is determined.

[0267] The embodiment of the present application does not limit the specific calculation method of the above-mentioned filtering cost. For example, the above-mentioned filtering cost can be a rate-distortion cost (RDO) or an approximate cost, such as SSD, STAD or SAD.

[0268] In the embodiment of the present application, the method of determining the filtering cost corresponding to each of the M data rearrangement methods is basically the same. For the convenience of description, the method of determining the filtering cost corresponding to the i-th data rearrangement method is taken as an example.

[0269] At this time, the filtering cost corresponding to the data rearrangement method is determined in the above S201-A1, including the following steps S201-A1-a1 to S201-A1-a3:

[0270] S201-A1-a1, for the i-th data rearrangement method among the M data rearrangement methods, based on the i-th data processing method, data is rearranged on the reconstructed image, and filtering is performed on the information to be filtered after rearrangement by the i-th data rearrangement method through a neural network filter to obtain the i-th filtered image block, where i is a positive integer less than or equal to M;

[0271] S201-A1-a2, based on the i-th data rearrangement method, reversely rearrange the data of the i-th filtered image block to obtain the filtered image block under the i-th data rearrangement method;

[0272] S201-A1-a3. Determine the filtering cost corresponding to the i-th data rearrangement method based on the filtered image block under the i-th data rearrangement method and the original image block of the reconstructed image block.

[0273] In an embodiment of the present application, for the i-th data rearrangement method among the M data rearrangement methods, the i-th data rearrangement method is used to rearrange data on the reconstructed image block to obtain the information to be filtered after the rearrangement by the i-th data rearrangement method. Next, the information to be filtered after the rearrangement by the i-th data rearrangement method is input into the neural network filter for filtering to obtain a filtered image block, recorded as the i-th filtered image block. Then, based on the i-th data rearrangement method, the i-th filtered image block is reversely rearranged to obtain the filtered image block under the i-th data rearrangement method. Finally, based on the filtered image block under the i-th data rearrangement method and the original image block of the reconstructed image block, the filtering cost corresponding to the i-th data rearrangement method is determined.

[0274] For example, Fig.10As shown, assuming that the M data rearrangement methods include rotating 90° to the left and rotating 90° to the right, the encoder directly inputs the information to be filtered into the neural network filter to obtain the filtered image block 0 in the data non-rearrangement method. The filtered image block 0 in the data non-rearrangement method is compared with the original image block of the reconstructed image, and the filtering cost corresponding to the data non-rearrangement method is calculated. Similarly, the encoder rotates the reconstructed image block and other information in the information to be filtered by 90° to the left and inputs it into the neural network filter to obtain the filtered image block rotated 90° to the left. Then, the filtered image block rotated 90° to the right is rotated 90° to obtain the filtered image block in the method of rotating 90° to the left. The filtered image block in the method of rotating 90° to the left is compared with the original image block of the reconstructed image, and the filtering cost corresponding to the method of rotating 90° to the left is calculated. Similarly, the encoder rotates the reconstructed image block and other information in the information to be filtered by 90° to the right and inputs it into the neural network filter to obtain the filtered image block rotated 90° to the right. Next, the filtered image block rotated 90° to the left is rotated 90° to the right to obtain a filtered image block rotated 90° to the right. The filtered image block rotated 90° to the right is compared with the original image block of the reconstructed image to calculate the filtering cost corresponding to the right rotation of 90°. The encoding end determines the data processing method with the smallest filtering cost among the data non-rearrangement method, the right rotation of 90° method, and the left rotation of 90° method as the first data processing method.

[0275] Mode 3, the above S201 includes the following steps S201-B1 and S201-B2:

[0276] S201-B1, determining the filtering cost corresponding to the data non-rearrangement method in M+1 data processing methods;

[0277] S201-B2. Determine, from among the M data rearrangement methods of preset order, a data rearrangement method with a filtering cost corresponding to a first data non-rearrangement method with a minimum filtering cost as a first data processing method.

[0278] The process of determining the filtering generation corresponding to the data non-rearrangement mode in the M+1 data processing modes can refer to the description of the above steps, which will not be repeated here.

[0279] In an embodiment of the present application, the encoding end first determines the filtering cost corresponding to the data non-rearrangement method, and then calculates the filtering cost for the M data rearrangement methods in a preset order. Specifically, first use the first data rearrangement method among the M data rearrangement methods to rearrange the reconstructed image block and other information in the filtering information, and then input it into the neural network filter for filtering to obtain a first filtered image block, and then use the first data rearrangement method to reversely rearrange the data of the first filtered image block to obtain a filtered image block 1 under the first data rearrangement method. Compare the filtered image block 1 with the original image block of the reconstructed image block to determine the filtering cost 1 corresponding to the first data rearrangement method. Then, compare the filtering cost 1 with the filtering cost corresponding to the above-mentioned data non-rearrangement method. If the filtering cost 1 is less than the filtering cost corresponding to the data non-rearrangement method, the first data rearrangement method is determined as the first data rearrangement method, and no subsequent data rearrangement methods are attempted. If the filter cost 1 is greater than or equal to the filter cost corresponding to the data non-rearrangement method, the second data rearrangement method among the M data rearrangement methods is used to rearrange the reconstructed image blocks and other information in the filter information, and the filter cost 2 corresponding to the second data rearrangement method is calculated, and the filter cost 2 is compared with the filter cost corresponding to the data non-rearrangement method. If the filter cost 2 is less than the filter cost corresponding to the data non-rearrangement method, the second data rearrangement method is determined as the first data processing method. If the filter cost 2 is greater than or equal to the filter cost corresponding to the data non-rearrangement method, the third data rearrangement method among the M data rearrangement methods is used to rearrange the reconstructed image blocks and other information in the filter information, and so on, until all the M data rearrangement methods are traversed.

[0280] In some embodiments, if the filtering costs corresponding to the M data rearrangement methods are all greater than or equal to the filtering costs corresponding to the data non-rearrangement method, the data non-rearrangement method is determined as the first data processing method.

[0281] In the above-mentioned method 2 and method 3, based on the i-th data rearrangement method, the reconstructed image blocks and other information in the filtering information are rearranged to obtain the rearranged filtering information, which includes at least the following methods:

[0282] Method 1: If the information to be filtered includes, in addition to the reconstructed image block, at least one of the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type of the reconstructed image block, and the quantization parameter information, then only the reconstructed image block can be rearranged, and other information will not be rearranged.

[0283] Method 2: If the information to be filtered includes, in addition to the reconstructed image block, a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type and at least one of the quantization parameter information of the reconstructed image block, then based on the i-th data processing method, the reconstructed image block is data rearranged, and based on the i-th data processing method, at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information is data rearranged to obtain the rearranged filtered data.

[0284] Method 3: If the information to be filtered includes, in addition to the reconstructed image block, a predicted image block corresponding to the reconstructed image block, a boundary strength image block, and at least one of the frame type and quantization parameter information of the reconstructed image block, then based on the i-th data processing method, data is rearranged on the reconstructed image block, and based on the i-th data processing method, data is rearranged on at least one of the predicted image block and the boundary strength image block, and data rearrangement is skipped for at least one of the frame type and the quantization parameter information, to obtain the rearranged filtered data.

[0285] Based on the above-mentioned methods 1, 2 and 3, the encoding end can determine the first data processing method from M+1 data processing methods.

[0286] In some embodiments, after the encoding end determines the first data processing method, the following steps are further included:

[0287] Step A: indicating a first data processing mode in a bit stream.

[0288] In the embodiment of the present application, after the encoder determines the first data processing method based on the above steps, the encoder directly indicates the first data processing method to the encoder. In this way, the decoder can obtain the first data processing method by decoding the bit stream without performing other additional calculations, thereby reducing the calculation complexity of the decoder and improving the decoding performance.

[0289] In the embodiment of the present application, the implementation methods of the above step A include but are not limited to the following:

[0290] Mode 1, the above step A includes step A-a1 and step A-a2:

[0291] Step A-a1, determining a first flag based on a first data processing method;

[0292] Step A-a2: writing a first flag in the bit stream, where the first flag is used to indicate whether data rearrangement is performed when filtering the information to be filtered.

[0293] In this mode 1, the encoding end first determines the value of the first flag based on the first data processing mode. For example, if the encoding end determines that the information to be filtered needs to be rearranged during filtering, that is, the first data processing mode is a data rearrangement mode, then the value of the first flag is set to yes (for example, 1) and the code stream is written. If the encoding end determines that the information to be filtered does not need to be rearranged during filtering, that is, the first data processing mode is a data non-rearrangement mode, then the value of the first flag is set to no (for example, 0) and the code stream is written. In this way, the decoding end decodes the code stream, obtains a flag, and then determines the first data processing mode based on the first flag.

[0294] In some embodiments, if the first flag indicates that data rearrangement is performed on the information to be filtered during filtering, the encoder also needs to write a second flag in the bitstream, where the second flag is used to indicate the data rearrangement method of the information to be filtered.

[0295] The embodiment of the present application does not limit the specific forms of expression of the first mark and the second mark.

[0296] In an example, the correspondence between the values ​​of the first flag and the second flag and the data processing method is shown in Table 2.

[0297] As shown in Table 2, the encoding end first determines the value of the first flag based on the first data processing mode. For example, if the first data processing mode is a data non-rearrangement mode, the value of the first flag is set to 0. If the first data processing mode is a data rearrangement mode, the value of the first flag is set to 1, and the second flag is further written into the code stream. For example, if the first data processing mode is a 90° left rotation mode, the value of the second flag is set to 00 and then written into the code stream.

[0298] In the method 1, the encoding end indicates the first data processing method through the first flag and the second flag.

[0299] Mode 2: In this mode, the encoding end can replace the first flag and the second flag with a flag, namely, a third flag. That is, the third flag can indicate whether the information to be filtered is to be rearranged during filtering, and can also indicate a specific data rearrangement method. In this case, the above step A includes step A-b1 and step A-b2:

[0300] Step A-b1, determining a third flag based on the first data processing method;

[0301] Step A-b2: writing a third flag into the bit stream, where the third flag is used to indicate the first data processing mode.

[0302] In the method 2, the encoder directly indicates the index of the first data processing method to the encoder through the third flag. Specifically, after the encoder determines the first data processing method of the information to be filtered from the M+1 data processing methods, the encoder indicates the first data processing method to the encoder through the third flag. For example, if the first data processing method is a data non-rearrangement method, the encoder sets the value of the third flag to the first value (for example, 0); if the first data processing method is a data rearrangement method, the third flag is set to other values. For example, when the first data processing method is to rotate 90° to the left, the value of the third flag is set to 100. In this way, the decoder obtains the third flag by decoding the code stream, and then determines the first data processing method from the M+1 data processing methods based on the value of the third flag.

[0303] The embodiment of the present application does not limit the specific form of the third mark.

[0304] In an example, the correspondence between the third flag value and the data processing method is shown in Table 3.

[0305] It should be noted that if the reconstructed image block is a reconstructed image, the first flag, the second flag and the third flag may be picture-level flags. If the reconstructed image block is a partial area of ​​a reconstructed image, the first flag, the second flag and the third flag may be block-level flags.

[0306] The above describes the specific process of the encoder determining the first data processing method for the information to be filtered. After the encoder determines the first data processing method, the following step S202 is performed.

[0307] S202. If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data is rearranged on the reconstructed image block to obtain rearranged information to be filtered, and the rearranged information to be filtered is filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered.

[0308] Based on the above steps, the encoding end determines a first data processing method. If the first data processing method is a data non-rearrangement method, the encoding end does not rearrange the data on the information to be filtered, but instead filters the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

[0309] If the first data processing method is one of the M data rearrangement methods, it means that data rearrangement is required first before the information to be filtered is input into the neural network filter.

[0310] In some embodiments, the information to be filtered only includes the reconstructed image blocks.

[0311] In some embodiments, in addition to the reconstructed image block, the information to be filtered also includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type (slice type) of the reconstructed image block, and quantization parameter information. Optionally, the quantization parameter information includes a slice QP (slice level quantization parameter) and a base QP (basic quantization parameter).

[0312] In the embodiment of the present application, if the first data processing method is a data rearrangement method, the encoding end may rearrange the data of the information to be filtered in at least the following ways:

[0313] In the first method, the encoding end only rearranges data on the reconstructed image blocks in the information to be filtered, and does not rearrange data on other information in the information to be filtered. For example, the information to be filtered includes information such as the reconstructed image block, the predicted image block corresponding to the reconstructed image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block, etc. The encoding end adopts the first data processing method to rearrange data on the reconstructed image block, and does not rearrange data on the predicted image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block corresponding to the reconstructed image block. Since the predicted image block, the boundary strength image block, the frame type and quantization parameter information of the reconstructed image block corresponding to the reconstructed image block are all auxiliary information of the reconstructed image block, data rearrangement of this information is not required, which can reduce the data rearrangement pressure on the encoding end, thereby improving the encoding efficiency of the encoding end.

[0314] The second method is to rearrange the data of the reconstructed image block based on the first data processing method, and to rearrange the data of at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information based on the first data processing method to obtain the rearranged filtered data.

[0315] The third method is to rearrange the data of the reconstructed image block based on the first data processing method, and to rearrange the data of at least one of the predicted image block and the boundary strength image block based on the first data processing method, and skip the data rearrangement of at least one of the frame type and quantization parameter information to obtain the rearranged filtered data.

[0316] The encoding end adopts a first data processing method to rearrange the reconstructed image blocks in the information to be filtered, or rearrange the reconstructed image blocks and other information in the filtering information to obtain rearranged filtering information. Then, the rearranged filtering information is input into the neural network filter, and the rearranged information to be filtered is filtered by the neural network filter to obtain the rearranged filtered image blocks of the information to be filtered.

[0317] In an embodiment of the present application, before filtering the information to be filtered, the encoding end first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks and other information in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image encoding performance.

[0318] S203 . Based on the first data processing method, reversely rearrange the data of the rearranged filtered image of the information to be filtered to obtain a filtered image block of the reconstructed image block.

[0319] Due to the above steps, the reconstructed image blocks and other information in the information to be filtered are data rearranged, so that the neural network filter filters the rearranged information to be filtered, and the result is a filtered image of the rearranged information to be filtered. The data distribution characteristics of the filtered image of the rearranged information to be filtered are inconsistent with the data distribution characteristics of the information to be filtered. Therefore, in an embodiment of the present application, after the encoding end uses the first data processing method to rearrange the data of the reconstructed image blocks and other information in the information to be filtered, and obtains the rearranged filtered image of the information to be filtered, it is also necessary to use the first data processing method to reversely rearrange the data of the rearranged filtered image of the information to be filtered, and obtain the filtered image block of the reconstructed image block.

[0320] For example, the first data processing method is to rotate 90° to the left. At this time, the encoding end needs to rotate the rearranged filtered image of the to-be-filtered information to the right by 90° to obtain a filtered image block of the reconstructed image block.

[0321] For another example, the first data processing method is to rotate right by 90°. At this time, the encoding end needs to rotate the filtered image of the rearranged information to be filtered by 90° to the left to obtain a filtered image block of the reconstructed image block.

[0322] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block under a Y component, a reconstructed image block under a U component, and a reconstructed image block under a V component. That is, the adaptive data rearrangement method of the embodiment of the present application can be applied to at least one of the luminance component Y and the chrominance components U and V.

[0323] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video encoder, for example, in loop filtering, and the filtered reconstructed image is stored in a decoding cache for encoding the next frame of the image. At this time, the filter output affects the video encoding.

[0324] In some embodiments, the image filtering method of the embodiment of the present application can be used in a video post-processing product. For example, it can be applied to a display end, that is, the filtered reconstructed image is directly displayed without being placed in a decoding cache, and the filter output does not affect the video encoding.

[0325] In some embodiments, the image encoding method of the embodiments of the present application can also be used in any module that uses a neural network in neural network-based video encoding, such as a neural network super-resolution module, a neural network intra-frame prediction module, etc., by simply replacing the reconstructed image block with the corresponding image block.

[0326] In the image filtering method provided by the embodiment of the present application, the encoding end determines a first data processing method for information to be filtered from M+1 data processing methods, the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​a reconstructed image, and M is a positive integer; if the first data processing method is one of the M data rearrangement methods, based on the first data processing method, the reconstructed image block in the information to be filtered is rearranged to obtain the rearranged information to be filtered, and the rearranged information to be filtered is filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; based on the first data processing method, the filtered image of the rearranged information to be filtered is reversely rearranged to obtain a filtered image block of the reconstructed image block. That is to say, before filtering the information to be filtered, the present application first determines whether it is necessary to rearrange the data of the information to be filtered. If it is determined that data rearrangement is required, the first data processing method is used to rearrange the data of the reconstructed image blocks in the information to be filtered, so that the data distribution characteristics of the rearranged information to be filtered are consistent with or close to the data distribution characteristics of the training data, so as to improve the filtering effect of the neural network filter on the rearranged information to be filtered, thereby improving the image filtering effect and improving the image encoding performance.

[0327] The preferred embodiments of the present application are described in detail above in conjunction with the accompanying drawings. However, the present application is not limited to the specific details in the above embodiments. Within the technical concept of the present application, the technical solution of the present application can be subjected to a variety of simple modifications, and these simple modifications all belong to the protection scope of the present application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. In order to avoid unnecessary repetition, the present application will not further explain various possible combinations. For another example, the various different embodiments of the present application can also be arbitrarily combined, as long as they do not violate the ideas of the present application, they should also be regarded as the contents disclosed in the present application.

[0328] It should also be understood that in the various method embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0329] Combination of the above Figures 7 to 11 , describes in detail the method embodiment of the present application, and the following is combined with Figure 12 to Figure 13 , describe in detail the device embodiments of the present application.

[0330] Fig.12 It is a schematic block diagram of a neural network-based image filtering device provided in one embodiment of the present application.

[0331] like Fig.12 As shown, the image filtering device 10 may include:

[0332] The rearrangement determination unit 11 is used to determine a first data processing method of the information to be filtered from M+1 data processing methods, wherein the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0333] a rearrangement filtering unit 12, configured to, if the first data processing method is one of the M data rearrangement methods, rearrange the data of the reconstructed image block based on the first data processing method to obtain rearranged information to be filtered, and filter the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0334] an inverse rearrangement unit 13, configured to perform inverse rearrangement of data on the rearranged filtered image of the information to be filtered based on the first data processing method, to obtain a filtered image block of the reconstructed image block; and

[0335] The rearrangement filtering unit 12 is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing mode is the data non-rearrangement mode.

[0336] In some embodiments, the first data processing method is a data processing method with the smallest filtering cost among the M+1 data processing methods.

[0337] In some embodiments, the rearrangement determination unit 11 is specifically configured to determine the first data processing method from the M+1 data processing methods by decoding the code stream.

[0338] In some embodiments, the rearrangement determination unit 11 is specifically used to decode the code stream to obtain a first flag, where the first flag is used to indicate whether data rearrangement is performed on the information to be filtered during filtering; based on the first flag, the first data processing method is obtained from the M+1 data processing methods.

[0339] In some embodiments, the rearrangement determination unit 11 is specifically used to determine the data non-rearrangement method as the first data processing method if the first flag indicates that the information to be filtered does not undergo data rearrangement during filtering; if the first flag indicates that the information to be filtered undergoes data rearrangement during filtering, decode the code stream to obtain a second flag, and based on the second flag, determine the first data processing method from the M data rearrangement methods, where the second flag is used to indicate an index of the data rearrangement method of the information to be filtered.

[0340] In some embodiments, the rearrangement determination unit 11 is specifically used to decode the code stream to obtain a third flag, where the third flag is used to indicate the first data processing method; based on the third flag, the first data processing method is determined from the M+1 data processing methods.

[0341] In some embodiments, the M data rearrangement methods include at least one of the following: rotation method, horizontal flipping and then rotating method, vertical flipping and then rotating method, diagonal flipping and then rotating method, down sampling and then rotating method, up sampling and then rotating method.

[0342] In some embodiments, the rotation method includes any one of rotating to the left by N*90° and rotating to the right by N*90°, where N is an integer positive number.

[0343] In some embodiments, the rearrangement filtering unit 12 is also used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-rearrangement method.

[0344] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block under a Y component, a reconstructed image block under a U component, and a reconstructed image block under a V component.

[0345] In some embodiments, the information to be filtered further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information of the reconstructed image block.

[0346] In some embodiments, the reordering filter unit 12 is specifically used to reorder the data of the reconstructed image block based on the first data processing method, and to reorder the data of at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information based on the first data processing method to obtain the reordered filtered data.

[0347] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.12 The device shown can perform the above Figure 7 The method embodiment shown, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiment corresponding to the decoder, and for the sake of brevity, they are not repeated here.

[0348] Fig.13 It is a schematic block diagram of a neural network-based image filtering device provided in one embodiment of the present application.

[0349] like Fig.13 As shown, the image filtering device 20 may include:

[0350] a rearrangement determination unit 21, configured to determine a first data processing mode of the information to be filtered from M+1 data processing modes, wherein the M+1 data processing modes include a data non-rearrangement mode and M data rearrangement modes, the information to be filtered includes a reconstructed image block, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer;

[0351] a rearrangement filtering unit 22, configured to, if the first data processing method is one of the M data rearrangement methods, rearrange the information to be filtered based on the first data processing method to obtain rearranged information to be filtered, and filter the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered;

[0352] an inverse rearrangement unit 23, configured to perform inverse rearrangement of data on the rearranged filtered image block of the to-be-filtered information based on the first data processing method, to obtain a filtered image block of the reconstructed image block; and

[0353] The rearrangement filtering unit 22 is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing mode is the data non-rearrangement mode.

[0354] In some embodiments, the rearrangement determination unit 21 is specifically used to determine the filtering costs corresponding to the M+1 data processing methods respectively; and determine a data processing method with the smallest filtering cost among the M+1 data processing methods as the first data processing method.

[0355] In some embodiments, the rearrangement determination unit 21 is specifically used to perform data rearrangement on the reconstructed image based on the i-th data processing method for the i-th data rearrangement method among the M data rearrangement methods, and filter the information to be filtered after the rearrangement by the i-th data rearrangement method through the neural network filter to obtain the i-th filtered image block, where i is a positive integer less than or equal to M; based on the i-th data rearrangement method, perform data reverse rearrangement on the i-th filtered image block to obtain the filtered image block under the i-th data rearrangement method; based on the filtered image block under the i-th data rearrangement method and the original image block of the reconstructed image block, determine the filtering cost corresponding to the i-th data rearrangement method.

[0356] In some embodiments, the rearrangement determination unit 21 is specifically used to filter the information to be filtered through the neural network filter to obtain a filtered image block under the data non-rearrangement method; based on the filtered image block under the data non-rearrangement method and the original image block of the reconstructed image block, determine the filtering cost corresponding to the data non-rearrangement method.

[0357] In some embodiments, the reordering filter unit 22 is further configured to indicate the first data processing mode in the bitstream.

[0358] In some embodiments, the reordering filter unit 22 is specifically used to determine a first flag based on the first data processing method; write the first flag into the bit stream, and the first flag is used to indicate whether to reorder the data when filtering the information to be filtered.

[0359] In some embodiments, if the first flag indicates that the information to be filtered is to be rearranged during filtering, the rearrangement filtering unit 22 is further configured to write a second flag in the bit stream, wherein the second flag is configured to indicate a data rearrangement method for the information to be filtered.

[0360] In some embodiments, the reordering filter unit 22 is specifically configured to determine the third flag based on the first data processing method; write the third flag into the bit stream, and the third flag is configured to indicate the first data processing method.

[0361] In some embodiments, the M data rearrangement methods include at least one of the following: rotation method, horizontal flipping and then rotating method, vertical flipping and then rotating method, diagonal flipping and then rotating method, down sampling and then rotating method, up sampling and then rotating method.

[0362] In some embodiments, the rotation method includes any one of rotating to the left by N*90° and rotating to the right by N*90°, where N is a positive integer.

[0363] In some embodiments, the reordering filter unit 22 is also used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-reordering method.

[0364] In some embodiments, the reconstructed image block includes at least one of a reconstructed image block under a Y component, a reconstructed image block under a U component, and a reconstructed image block under a V component.

[0365] In some embodiments, the information to be filtered further includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, a frame type and quantization parameter information of the reconstructed image block.

[0366] In some embodiments, the reordering filter unit 22 is specifically used to reorder the data of the reconstructed image block based on the first data processing method, and to reorder the data of at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information based on the first data processing method to obtain the reordered filtered data.

[0367] It should be understood that the device embodiment and the method embodiment may correspond to each other, and similar descriptions may refer to the method embodiment. To avoid repetition, they will not be described here. Specifically, Fig.13 The device shown can perform the above Fig.11 The embodiments of the method shown, and the aforementioned and other operations and / or functions of each module in the device are respectively for implementing the method embodiments corresponding to the encoder, and for the sake of brevity, they are not repeated here.

[0368] The above describes the device of the embodiment of the present application from the perspective of the functional module in conjunction with the accompanying drawings. It should be understood that the functional module can be implemented in hardware form, can be implemented by instructions in software form, and can also be implemented by a combination of hardware and software modules. Specifically, the steps of the method embodiment in the embodiment of the present application can be completed by the hardware integrated logic circuit and / or software form instructions in the processor, and the steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to perform, or a combination of hardware and software modules in the decoding processor to perform. Optionally, the software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor reads the information in the memory, and completes the steps in the above method embodiment in conjunction with its hardware.

[0369] Fig.14 is a schematic block diagram of an electronic device provided in an embodiment of the present application, Fig.14 The electronic device may be the above-mentioned encoder, decoder or other image filtering device.

[0370] like Fig.14 As shown, the electronic device 30 may include:

[0371] The memory 31 and the processor 32, the memory 31 is used to store the computer program 33 and transmit the program code 33 to the processor 32. In other words, the processor 32 can call and run the computer program 33 from the memory 31 to implement the method in the embodiment of the present application.

[0372] For example, the processor 32 may be configured to execute the steps in the method 200 according to the instructions in the computer program 33 .

[0373] In some embodiments of the present application, the processor 32 may include but is not limited to:

[0374] General-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components, etc.

[0375] In some embodiments of the present application, the memory 31 includes but is not limited to:

[0376] Volatile memory and / or non-volatile memory. Among them, the non-volatile memory can be read-only memory (ROM), programmable ROM (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM) or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link DRAM (SLDRAM), and direct RAM bus random access memory (DR RAM).

[0377] In some embodiments of the present application, the computer program 33 may be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method for recording pages provided by the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 33 in the electronic device.

[0378] like Fig.14 As shown, the electronic device 30 may also include:

[0379] The transceiver 34 may be connected to the processor 32 or the memory 31 .

[0380] The processor 32 may control the transceiver 34 to communicate with other devices, specifically, to send information or data to other devices, or to receive information or data sent by other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include an antenna, and the number of antennas may be one or more.

[0381] It should be understood that the various components in the electronic device 30 are connected via a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus and a status signal bus.

[0382] According to one aspect of the present application, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a computer, the computer can perform the method of the above method embodiment. In other words, the present application embodiment also provides a computer program product containing instructions, and when the instructions are executed by a computer, the computer can perform the method of the above method embodiment.

[0383] According to another aspect of the present application, a computer program product or computer program is provided, the computer program product or computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method of the above method embodiment.

[0384] In other words, when implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a solid state drive (solid state disk, SSD)), etc.

[0385] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0386] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the module is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0387] The modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to implement the scheme of this embodiment. For example, each functional module in each embodiment of the present application may be integrated into a processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0388] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A neural network-based image filtering method, characterized in that: include: Determine a first data processing method for information to be filtered from M+1 data processing methods, wherein the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block to be filtered, and M is a positive integer; If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data is rearranged on the reconstructed image block to obtain rearranged information to be filtered, and filtering is performed on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; Based on the first data processing method, reverse rearrange the data of the rearranged filtered image of the information to be filtered to obtain a filtered image block of the reconstructed image block; as well as If the first data processing method is the data non-rearrangement method, the information to be filtered is filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

2. The method according to claim 1, characterized in that The first data processing method is a data processing method with the smallest filtering cost among the M+1 data processing methods.

3. The method according to claim 1, characterized in that The step of determining a first data processing method for the information to be filtered from the M+1 data processing methods includes: Decoding the code stream to obtain a first flag, where the first flag is used to indicate whether data rearrangement is performed on the information to be filtered during filtering; If the first flag indicates that the information to be filtered is not to be rearranged during filtering, the data non-rearrangement mode is determined as the first data processing mode; If the first flag indicates that the information to be filtered is to be rearranged during filtering, the code stream is decoded to obtain the second flag, and based on the second flag, the first data processing method is determined from the M data rearrangement methods, and the second flag is used to indicate the index of the data rearrangement method of the information to be filtered.

4. The method according to claim 1, characterized in that: The step of determining a first data processing method for the information to be filtered from the M+1 data processing methods includes: Decoding the code stream to obtain a third flag, where the third flag is used to indicate the first data processing mode; Based on the third flag, the first data processing method is determined from the M+1 data processing methods.

5. The method according to any one of claims 1 to 4, characterized in that: The M data rearrangement methods include at least one of the following: a rotation method, a method of first horizontally flipping and then rotating, a method of first vertically flipping and then rotating, a method of first diagonally flipping and then rotating, a method of first downsampling and then rotating, and a method of first upsampling and then rotating.

6. The method according to claim 5, characterized in that The rotation method includes any one of rotating to the left by N*90° and rotating to the right by N*90°, wherein N is an integer positive number.

7. The method according to any one of claims 1 to 4, characterized in that: The information to be filtered also includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, and a frame type and quantization parameter information of the reconstructed image block.

8. The method according to claim 7, characterized in that The step of rearranging data of the reconstructed image block based on the first data processing method to obtain the rearranged filtering information includes: Based on the first data processing method, the reconstructed image block is rearranged, and based on the first data processing method, at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information is rearranged to obtain the rearranged filtered data.

9. A neural network-based image filtering method, characterized in that: include: Determine a first data processing method for the information to be filtered from M+1 data processing methods, wherein the M+1 data processing methods include a data non-rearrangement method and M data rearrangement methods, the information to be filtered includes a reconstructed image block, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer; If the first data processing method is one of the M data rearrangement methods, based on the first data processing method, data rearrangement is performed on the information to be filtered to obtain rearranged information to be filtered, and filtering is performed on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; Based on the first data processing method, reversely rearrange the data of the rearranged filtered image block of the information to be filtered to obtain the filtered image block of the reconstructed image block; as well as If the first data processing method is the data non-rearrangement method, the information to be filtered is filtered through the neural network filter to obtain a filtered image of the reconstructed image block.

10. The method according to claim 9, characterized in that The step of determining a first data processing method for the information to be filtered from the M+1 data processing methods includes: Determine the filtering costs corresponding to the M+1 data processing modes respectively; A data processing method with the smallest filtering cost among the M+1 data processing methods is determined as the first data processing method.

11. The method according to claim 9, characterized in that Determining the filtering costs corresponding to the M data rearrangement modes respectively includes: For the i-th data rearrangement method among the M data rearrangement methods, based on the i-th data processing method, data is rearranged on the reconstructed image, and filtering is performed on the information to be filtered after rearrangement by the i-th data rearrangement method through the neural network filter to obtain the i-th filtered image block, where i is a positive integer less than or equal to M; Based on the i-th data rearrangement mode, reversely rearrange the data of the i-th filtered image block to obtain the filtered image block under the i-th data rearrangement mode; Based on the filtered image block in the i-th data rearrangement mode and the original image block of the reconstructed image block, a filtering cost corresponding to the i-th data rearrangement mode is determined.

12. The method according to claim 9, characterized in that Determining the filtering cost corresponding to the data non-rearrangement method includes: The information to be filtered is filtered by the neural network filter to obtain a filtered image block in a manner where the data is not rearranged; Based on the filtered image block in the data non-rearrangement mode and the original image block of the reconstructed image block, a filtering cost corresponding to the data non-rearrangement mode is determined.

13. The method according to claim 9, characterized in that Further including: Writing a first flag in the bit stream, wherein the first flag is used to indicate whether data rearrangement is performed on the information to be filtered during filtering; If the first flag indicates that the information to be filtered is to be rearranged during filtering, a second flag is written into the bit stream, and the second flag is used to indicate the data rearrangement method of the information to be filtered.

14. The method according to any one of claims 9 to 13, characterized in that: The M data rearrangement methods include at least one of the following: a rotation method, a method of first horizontally flipping and then rotating, a method of first vertically flipping and then rotating, a method of first diagonally flipping and then rotating, a method of first downsampling and then rotating, and a method of first upsampling and then rotating.

15. The method according to any one of claims 9 to 13, characterized in that: The information to be filtered also includes at least one of a predicted image block corresponding to the reconstructed image block, a boundary strength image block, and a frame type and quantization parameter information of the reconstructed image block.

16. The method according to claim 15, characterized in that The step of rearranging data of the reconstructed image block based on the first data processing method to obtain the rearranged filtering information includes: Based on the first data processing method, the reconstructed image block is rearranged, and based on the first data processing method, at least one of the predicted image block, the boundary strength image block, the frame type and the quantization parameter information is rearranged to obtain the rearranged filtered data.

17. An image filtering device based on a neural network, characterized in that: include: a rearrangement determination unit, configured to determine a first data processing mode of the information to be filtered from M+1 data processing modes, wherein the M+1 data processing modes include a mode of not rearranging data and M data rearrangement modes, the information to be filtered includes a reconstructed image block to be filtered, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer; a rearrangement filtering unit, configured to, if the first data processing mode is one of the M data rearrangement modes, perform data rearrangement on the reconstructed image block based on the first data processing mode to obtain rearranged information to be filtered, and perform filtering processing on the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; an anti-rearrangement unit, configured to perform data anti-rearrangement on the rearranged filtered image of the information to be filtered based on the first data processing method, so as to obtain a filtered image block of the reconstructed image block; and The rearrangement filtering unit is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-rearrangement method.

18. An image filtering device based on a neural network, characterized in that: include: a rearrangement determination unit, configured to determine a first data processing mode of the information to be filtered from M+1 data processing modes, wherein the M+1 data processing modes include a mode of not rearranging data and M data rearrangement modes, the information to be filtered includes a reconstructed image block, the reconstructed image block is a reconstructed image or a partial area of ​​the reconstructed image, and M is a positive integer; a rearrangement filtering unit, configured to, if the first data processing mode is one of the M data rearrangement modes, rearrange the information to be filtered based on the first data processing mode to obtain rearranged information to be filtered, and filter the rearranged information to be filtered through a neural network filter to obtain a filtered image block of the rearranged information to be filtered; an anti-rearrangement unit, configured to perform data anti-rearrangement on the rearranged filtered image block of the to-be-filtered information based on the first data processing method, to obtain a filtered image block of the reconstructed image block; and The rearrangement filtering unit is further used to filter the information to be filtered through the neural network filter to obtain a filtered image of the reconstructed image block if the first data processing method is the data non-rearrangement method.

19. An electronic device comprising a processor and a memory; The memory is used to store computer programs; The processor is used to execute the computer program to implement the method according to any one of claims 1 to 8 or 9 to 16.

20. A computer-readable storage medium, characterized in that: Used to store computer programs and code streams; The computer program enables a computer to execute the method according to any one of claims 1 to 8 or 9 to 16 to generate the code stream or decode the code stream.

Images