Image filtering method and apparatus, and device

By aligning the block division method used during training with actual usage, neural network-based filters achieve optimal performance in image filtering, addressing the suboptimal performance issue in existing technologies.

JP2024543310AActive Publication Date: 2024-11-21TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
JP2024522042
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Priority Date
2022-05-18
Filing Date
2023-03-01
Publication Date
2024-11-21
Estimated Expiration
2043-03-01

AI Technical Summary

Technical Problem

Neural network-based filters in image processing exhibit suboptimal filtering performance due to mismatch between block division methods used in training and actual usage processes, leading to ineffective image reconstruction in video compression.

Method used

Implement a block division method during actual filtering that matches the block division scheme used during the neural network-based filter's training process, dividing the image into N blocks and filtering each using the neural network-based filter to obtain a filtered image.

Benefits of technology

This approach enhances the filtering performance of neural network-based filters by aligning training and usage processes, resulting in improved image filtering effects.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024543310000001_ABST
    Figure 2024543310000001_ABST
Patent Text Reader

Abstract

The present application provides an image filtering method and its apparatus and equipment, the method includes the steps of: obtaining an image to be filtered (S801); determining a neural network-based filter, and dividing the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered (N is a positive integer) (S802); and using the neural network-based filter to filter the N image blocks to be filtered, respectively, to obtain a filtered image (S803). Here, the above block division scheme is the block division scheme of the training image used in the neural network-based filter, that is, in the embodiment of the present application, the block division scheme in the actual use process of the neural network-based filter is consistent with the block division scheme used during training, so that the neural network filter can achieve optimal filtering performance, and thus improve the filtering effect of the image.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority to a Chinese patent application filed on May 18, 2022, bearing application number 202210551039.7 and entitled "Image filtering method, apparatus and device thereof," the entire contents of which are incorporated herein by reference.

[0002] The present embodiment relates to the field of image processing technology, and more particularly to an image filtering method and device, and an apparatus therefor. [Background technology]

[0003] Digital video technology can be incorporated into various video devices, such as digital televisions, smartphones, computers, e-book readers or video players. With the development of video technology, the amount of data contained in video data becomes larger, and in order to facilitate the transmission of video data, video devices implement video compression technology to transmit or store video data more effectively.

[0004] In video compression technology, image loss occurs, and in order to reduce the loss, the reconstructed image is filtered. With the rapid development of neural network technology, in some scenarios, neural network-based filters are used to filter the reconstructed image. However, the filtering effect of some neural network-based filters is not good. Summary of the Invention

[0005] This application provides an image filtering method and its device and equipment, and its technical solutions are as follows:

[0006] In one aspect, the present application provides an image filtering method, the method comprising: obtaining an image to be filtered; determining a neural network based filter; Dividing the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, where N is a positive integer, and the block division scheme is a block division scheme of a training image used by the neural network-based filter in a training process; and filtering each of the N image blocks to be filtered using the neural network-based filter to obtain a filtered image.

[0007] In another aspect, the present application provides an image filtering apparatus, the apparatus comprising: an acquisition unit configured to acquire a to-be-filtered image; a segmentation unit configured to determine a neural network-based filter, and segment the image to be filtered according to a block segmentation scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, where the block segmentation scheme is a block segmentation scheme of a training image used by the neural network-based filter in a training process; and a filtering unit configured to filter each of the N image blocks to be filtered using the neural network based filter to obtain a filtered image.

[0008] In another aspect, an electronic device is provided, comprising: a memory configured to store a computer program; and a processor configured to execute a method according to an embodiment of each aspect of the present application by calling and executing the computer program stored in the memory.

[0009] In another aspect, a chip is provided that is configured to implement the method of each aspect of the present application, and illustratively includes a processor configured to call and execute a computer program from a memory to cause a device incorporating the chip to execute the method of each embodiment of the present application.

[0010] In another aspect, a computer readable storage medium is provided that is configured to store a computer program for causing a computer to perform a method according to an embodiment of each aspect of the present application.

[0011] In another aspect, a computer program product is provided that includes computer program instructions for causing a computer to perform a method according to an embodiment of each aspect of the present application.

[0012] In another aspect, there is provided a computer program which, when executed on a computer, causes the computer to carry out a method according to an embodiment of each aspect of the present application. Effect of the Invention

[0013] In summary, the present application obtains an image to be filtered, determines a neural network-based filter, divides the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N (N is a positive integer) image blocks to be filtered, and uses the neural network-based filter to filter the N image blocks to be filtered respectively to obtain a filtered image. The above block division scheme is the block division scheme of the training image used by the neural network-based filter in the training process, that is, in the embodiment of the present application, the block division scheme in the actual use process of the neural network-based filter is matched with the block division scheme used during training, so that the neural network filter can achieve optimal filtering performance, thereby improving the filtering effect of the image. [Brief description of the drawings]

[0014] [Figure 1] FIG. 2 is a schematic diagram of an application scenario according to an embodiment of the present invention. [Diagram 2] 1 is an exemplary block diagram of a video codec system according to an embodiment of the present invention; [Diagram 3] FIG. 1 is a schematic diagram of an encoding framework according to an embodiment of the present invention; [Figure 4] FIG. 1 is a schematic diagram of a decoding framework according to an embodiment of the present invention; [Diagram 5] FIG. 2 is a schematic diagram of image types. [Figure 6] FIG. 2 is a schematic diagram of image types. [Figure 7] FIG. 2 is a schematic diagram of image types. [Figure 8] FIG. 2 is a schematic diagram of filtering according to an embodiment of the present invention. [Figure 9] 1 shows a method for dividing an image into blocks. [Figure 10] 1 shows a method for dividing an image into blocks. [Figure 11] 1 shows a method for dividing an image into blocks. [Figure 12] 1 shows a method for dividing an image into blocks. [Figure 13] 1 is a flowchart of an image filtering method according to one embodiment of the present application. [Figure 14] FIG. 2 is a schematic diagram of block division of an image according to an embodiment of the present invention. [Figure 15] FIG. 2 is a schematic diagram of block division of an image according to an embodiment of the present invention. [Figure 16] FIG. 2 is a schematic diagram of block division of an image according to an embodiment of the present invention. [Figure 17] FIG. 2 is a schematic diagram of block division of an image according to an embodiment of the present invention. [Figure 18] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 19] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 20] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 21] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 22] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Diagram 23] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 24] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Diagram 25] FIG. 4 is a schematic diagram of another image block division according to an embodiment of the present invention. [Figure 26] FIG. 2 is a schematic diagram of image block expansion according to an embodiment of the present invention; [Figure 27] FIG. 2 is a schematic diagram of image block expansion according to an embodiment of the present invention; [Figure 28] FIG. 2 is a schematic diagram of image block expansion according to an embodiment of the present invention; [Figure 29] FIG. 2 is a schematic diagram of image block expansion according to an embodiment of the present invention; [Diagram 30] FIG. 13 is a schematic diagram of another image block expansion according to an embodiment of the present disclosure. [Diagram 31] FIG. 13 is a schematic diagram of another image block expansion according to an embodiment of the present disclosure. [Diagram 32] FIG. 13 is a schematic diagram of another image block expansion according to an embodiment of the present disclosure. [Diagram 33] FIG. 13 is a schematic diagram of another image block expansion according to an embodiment of the present disclosure. [Diagram 34] FIG. 2 is a schematic diagram of a spatial domain reference image block according to an embodiment of the present invention; [Diagram 35] FIG. 2 is a schematic diagram of a spatial domain reference image block according to an embodiment of the present invention; [Diagram 36] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Figure 37] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Figure 38] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Figure 39] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Diagram 40] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Diagram 41] FIG. 4 is a schematic diagram of another spatial domain reference image block according to an embodiment of the present invention; [Diagram 42] FIG. 2 is a schematic diagram of a time-domain reference image block according to an embodiment of the present invention; [Diagram 43] FIG. 2 is a schematic diagram of a time-domain reference image block according to an embodiment of the present invention; [Diagram 44] FIG. 4 is a schematic diagram of another time-domain reference image block according to an embodiment of the present invention; [Diagram 45] FIG. 4 is a schematic diagram of another time-domain reference image block according to an embodiment of the present invention; [Figure 46] 1 is an exemplary flowchart of an image filtering method according to one embodiment of the present application. [Figure 47] 1 is an exemplary block diagram of an image filtering device according to one embodiment of the present application; [Figure 48] FIG. 1 is an exemplary block diagram of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0015] 1 is a schematic diagram of an application scenario according to an embodiment of the present application, which includes an electronic device 100, which is equipped with a neural network-based filter 200. After obtaining an image to be filtered, the electronic device 100 inputs the image to be filtered into the neural network-based filter 200 for filtering.

[0016] In some embodiments, the electronic device 100 includes a display device, such that the electronic device 100 can display the filtered image via the display device.

[0017] The present embodiment is not limited to a specific type of electronic device 100, which may be any device having a data processing function.

[0018] In some embodiments, electronic device 100 may be a terminal device, including, for example, a smartphone, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a vehicle computer, and the like.

[0019] In some embodiments, the electronic device 100 may be a server. The server may be one or more. When the server is a plurality of servers, there may be at least two servers for providing different services and / or at least two servers for providing the same service, for example, the same service may be provided in a load balancing manner, and the present embodiment is not limited thereto.

[0020] Optionally, the above server may be an independent physical server, a server cluster or a distributed system of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, Content Delivery Networks (CDNs), big data and artificial intelligence platforms, etc. The server may be a node of the blockchain.

[0021] In some embodiments, the above-mentioned servers are cloud servers equipped with powerful computing resources, and have highly virtualized and highly distributed characteristics.

[0022] In some embodiments, the image to be filtered is collected by an image collecting device. For example, the image collecting device transmits the collected image to be filtered to the electronic device 100, and the electronic device 100 uses a neural network-based filter to filter the captured image to be filtered. In another example, the electronic device 100 has an image collecting function, so that the electronic device 100 can collect images and input the collected image to be filtered to a neural network-based filter for filtering.

[0023] In some embodiments, the electronic device 100 may be an encoding device, and the image to be filtered can be understood as a reconstructed image, which is obtained by encoding and then reconstructing a current image, and inputting the reconstructed image into a neural network-based filter for filtering.

[0024] In some embodiments, the above electronic device 100 may be a decoding device, which performs image reconstruction after decoding a codestream to obtain a reconstructed image, and then inputs the reconstructed image into a neural network-based filter for filtering.

[0025] The present embodiments are applicable to any scenario where an image needs to be filtered.

[0026] In some embodiments, the present disclosure may be applicable to a variety of scenarios, including, but not limited to, cloud technology (e.g., cloud gaming), artificial intelligence, intelligent transportation, assisted driving, and the like.

[0027] In some embodiments, the present application can be applied to image codec field, video codec field, hardware video codec field, native circuit video codec field, real-time video codec field, etc. For example, the solution of the present application can be combined with audio video coding standards (AVS: Audio Video coding Standard), such as H.264 / Audio Video Coding (AVC: Audio Video Coding) standard, H.265 / High Efficiency Video Coding (HEVC: High Efficiency Video Coding) standard, and H.266 / Versatile Video Coding (VVC: Versatile Video Coding) standard. Alternatively, the solutions of the present application can be combined with other proprietary or industry standards, including ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC, which includes the extensions Scalable Video Coding (SVC) and Multiview Video Coding (MVC)). It should be understood that the technology of the present application is not limited to any particular codec standard or technology.

[0028] For ease of understanding, a video codec system according to an embodiment of the present invention will first be introduced with reference to FIG.

[0029] FIG. 2 is an exemplary block diagram of a video codec system according to an embodiment of the present application. Note that FIG. 2 is merely an example, and the video codec system according to the embodiment of the present application includes, but is not limited to, that shown in FIG. 2. As shown in FIG. 2, the video codec system includes an encoding device 110 and a decoding device 120. Here, the encoding device is configured to encode (or compress) video data to generate a code stream, and transmit the code stream to a decoding device. The decoding device decodes the code stream generated by the encoding device to obtain decoded video data.

[0030] The encoding device 110 of the present embodiment may be understood as a device having a video encoding function, and the decoding device 120 may be understood as a device having a video decoding function, i.e., the encoding device 110 and the decoding device 120 of the present embodiment may include a broader range of devices, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, vehicle computers, etc.

[0031] In some embodiments, encoding device 110 may transmit encoded video data (e.g., a codestream, etc.) to decoding device 120 via channel 130. Channel 130 may include one or more media and / or devices that may transmit the encoded video data from encoding device 110 to decoding device 120.

[0032] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. In this example, encoding device 110 may modulate the encoded video data in accordance with a communication standard and transmit the modulated video data to decoding device 120. Here, the communication media includes a communication media such as a radio frequency spectrum, and optionally, the communication media may include a wired communication media such as one or more physical transmission lines.

[0033] In another example, channel 130 includes a storage medium that can store the video data encoded by encoding device 110. The storage medium can include a variety of locally accessible data storage media, such as optical disks, DVDs, flash memory, etc. In this example, decoding device 120 can obtain the encoded video data from the storage medium.

[0034] In another example, channel 130 may include a storage server, which may store the video data encoded by encoding device 110. In this example, decoding device 120 may download the stored encoded video data from the storage server. Optionally, the storage server may store the encoded video data and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a File Transfer Protocol (FTP) server, or the like.

[0035] In some embodiments, encoding device 110 comprises a video encoder 112 and an output interface 113, where output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0036] In some embodiments, encoding device 110 may further comprise a video source 111 in addition to video encoder 112 and input interface 113 .

[0037] The video source 111 may comprise at least one of a video collection device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, where the video input interface is configured to receive video data from a video content provider and the computer graphics system is configured to generate the video data.

[0038] The video encoder 112 encodes video data from the video source 111 to generate a codestream. The video data may include one or more pictures or a sequence of pictures. The codestream includes coding information for a picture or a sequence of pictures in the form of a bitstream. The coding information may include coded image data and associated data. The associated data may include a sequence parameter set (SPS), a picture parameter set (PPS), and other syntax structures. An SPS may include parameters that apply to one or more sequences. A PPS may include parameters that apply to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the codestream.

[0039] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may also be stored on a storage medium or a storage server so that it can be subsequently read by the decoding device 120.

[0040] In some embodiments, the decoding device 120 comprises an input interface 121 and a video decoder 122 .

[0041] In some embodiments, the decoding device 120 further comprises a display device 123 in addition to the input interface 121 and the video decoder 122 .

[0042] Here, the input interface 121 includes a receiver and / or a modem, and can receive encoded video data via a channel 130.

[0043] The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to a display device 123 .

[0044] Display device 123 displays the decoded video data. Display device 123 may be integrated with decoding device 120 or may be located external to decoding device 120. Display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0045] Furthermore, FIG. 2 is just an example, and the technical solution of the embodiments of the present application is not limited to FIG. 2, for example, the technology of the present application may be applied to one-sided video encoding or one-sided video decoding.

[0046] In the following, a video coding framework according to an embodiment of the present invention is introduced.

[0047] FIG. 3 is a schematic diagram of an encoding framework according to an embodiment of the present invention.

[0048] The coding framework may be used to perform lossy compression on images, or it may be used to perform lossless compression on images, which may be visually lossless compression or mathematically lossless compression.

[0049] The encoding framework is applicable to image data in Luminance Chroma (YCbCr, YUV) format.

[0050] For example, the encoding framework reads video data and for each frame of the video data, divides the image of a frame into several coding tree units (CTUs), where in some examples, a CTU is also called a "tree block", "largest coding unit" (LCU) or "coding tree block" (CTB). Each CTU may be associated with a block of pixels of the same size in the image. Each pixel may correspond to one luminance (or luma) sample and two chrominance (or chroma) samples. Thus, each CTU may be associated with one luminance sample block and two chroma sample blocks. For example, the size of one CTU may be 128×128, 64×64, 32×32, etc. One CTU may be further divided into several coding units (CUs) for encoding, where a CU may be a rectangular block or a square block. The CU can be further divided into a Prediction Unit (PU) and a Transform Unit (TU), which allows for more flexible processing by separating the coding, prediction, and transformation. In one example, the CTU is divided into CUs in a quadtree manner, and the CU is divided into TUs and PUs in a quadtree manner.

[0051] The video encoder and video decoder can support various PU sizes. Assuming that a particular CU is 2N×2N in size, the video encoder and video decoder can support PU sizes of 2N×2N or N×N for intra prediction, and symmetric PUs of 2N×2N, 2N×N, N×2N, N×N or similar sizes for inter prediction. The video encoder and video decoder can also support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter prediction.

[0052] As shown in Fig. 3, the coding framework includes a prediction unit 11, a residual generation unit 12, a transform unit 13, a quantization unit 14, an inverse quantization unit 15, an inverse transform unit 16, a reconstruction unit 17, a filtering unit 18 and an entropy coding unit 19. The prediction unit 11 includes an inter prediction unit 101 and an intra prediction unit 102. The inter prediction unit 101 includes a motion estimation unit 1011 and a motion compensation unit 1012. It should be noted that the coding framework may include more, less or different functional components.

[0053] Optionally, in this application, the current block is also called a current coding unit (CU) or a current prediction unit (PU), etc. The prediction block is also called a predicted image block or an image prediction block, and the reconstructed image block is also called a reconstruction block or an image reconstruction image block.

[0054] Here, after receiving a video, for each frame of the image constituting the video, the encoding side divides the image into a number of image blocks to be encoded. For the current image block to be encoded, the prediction unit 11 first predicts the current image block to be encoded based on the reference reconstructed image block to obtain prediction information of the current image block to be encoded. Here, the encoding side can use inter prediction or intra prediction technology to obtain prediction information.

[0055] Exemplarily, the motion estimation unit 1011 in the inter prediction unit 101 can search a reference image in a list of reference images to find a reference block of the image block to be coded. The motion estimation unit 1011 can generate an index indicating the reference block and a motion vector indicating a spatial displacement between the image block to be coded and the reference block. The motion estimation unit 1011 can output the index of the reference block and the motion vector as the motion information of the image block to be coded. The motion compensation unit 1012 can obtain prediction information of the image block to be coded based on the motion information of the image block to be coded.

[0056] The intra prediction unit 102 can adopt an intra prediction mode to generate prediction information of the current image block to be encoded. Currently, there are 15 intra prediction modes, including a planar mode, a DC mode, and 13 angle prediction modes. The intra prediction unit 102 can also adopt an intra block copy (IBC), an intra string copy (ISC) technique, etc.

[0057] HEVC uses intra prediction modes including Planar, DC, and 33 angle modes, for a total of 35 prediction modes. VVC uses intra modes including Planar, DC, and 65 angle modes, for a total of 67 prediction modes. AVS3 uses intra modes including DC, Planar, Bilinear, and 63 angle modes, for a total of 66 prediction modes.

[0058] The residual generation unit 12 is configured to subtract the prediction information from the original signal of the current image block to be coded to obtain a residual signal. After prediction, the amplitude of the residual signal is much smaller than that of the original signal.

[0059] The transform unit 13 and the quantization unit 14 are configured to perform transform and quantization operations on the residual signal. After transform and quantization, transform and quantization coefficients are obtained.

[0060] The entropy encoding unit 19 is configured to encode the quantized coefficients and other indication information on the encoding using an entropy encoding technique to obtain a codestream.

[0061] In addition, the encoding side needs to reconstruct the current image block to be coded so as to provide reference pixels for subsequent coding of the image block to be coded. Illustratively, after obtaining the transform quantization coefficient of the current image block to be coded, the inverse quantization unit 15 and the inverse transform unit 16 perform inverse quantization and inverse transform on the transform quantization coefficient of the current image block to be coded to obtain a reconstructed residual signal, and the reconstruction unit 17 adds prediction information corresponding to the current image block to be coded to the reconstructed residual signal to obtain a reconstructed signal of the current image block to be coded, and obtains a reconstructed image block according to the reconstructed signal.

[0062] Furthermore, the filtering unit 18 may filter the reconstructed image block, where DeBlocking Filter (DBF), Sample Adaptive Offset (SAO) or Adaptive Loop Filter (ALF), etc., may be adopted, where the reconstructed image block can be used to predict a subsequent image block to be coded.

[0063] In some embodiments, the reconstructed image block may be stored in a decoded image cache, and the inter prediction unit 101 may use a reference image including the reconstructed pixel block to perform inter prediction on PUs of other images. Furthermore, the intra prediction unit 102 may use the reconstructed image block in the decoded image cache to perform intra prediction on other PUs in the same image as the CU.

[0064] FIG. 4 is a schematic diagram of a decoding framework according to an embodiment of the present invention.

[0065] 4, the decoding framework includes an entropy decoding unit 21, a prediction unit 22, an inverse quantization unit 23, an inverse transform unit 24, a reconstruction unit 25, and a filtering unit 26. The prediction unit 22 includes a motion compensation unit 221 and an intra prediction unit 222.

[0066] Exemplarily, after the decoding side obtains a code stream, first, the entropy decoding unit 21 performs entropy decoding on the code stream to obtain a transform quantization coefficient of the current image block to be reconstructed, and then the inverse quantization unit 23 and the inverse transform unit 24 perform inverse quantization and inverse transform on the transform quantization coefficient to obtain a reconstructed residual signal of the current image block to be reconstructed. The prediction unit 22 predicts the current image block to be reconstructed to obtain prediction information of the current image block to be reconstructed. If the prediction unit 22 adopts inter prediction, the motion compensation unit 221 can configure a first reference image list (list 0) and a second reference image list (list 1) based on the syntax elements parsed from the code stream. In addition, the entropy decoding unit 21 can analyze the motion information of the image block to be reconstructed. The motion compensation unit 221 can determine one or more reference blocks of the image block to be reconstructed based on the motion information. The motion compensation unit 221 can generate prediction information of the image block to be reconstructed based on one or more reference blocks. When the prediction unit 22 adopts intra prediction, the entropy decoding unit 21 can analyze the index of the intra prediction mode used, and the intra prediction unit 222 can adopt the intra prediction mode according to the index to perform intra prediction and obtain prediction information of the image block to be reconstructed. The intra prediction unit 222 can also adopt IBC or ISC technology, etc.

[0067] In addition, the reconstruction unit 25 is configured to add the prediction information and the above reconstruction residual signal to obtain a reconstruction signal of the current image block to be reconstructed, and then obtain a current reconstructed image block corresponding to the current image block to be reconstructed according to the reconstruction signal, where the current reconstructed image block can be used to predict other subsequent image blocks to be reconstructed. Optionally, similar to the above encoding side, the filtering unit 26 on the decoding side can filter the current reconstructed image block.

[0068] In addition, if necessary, the block division information determined by the encoding side, and mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., can be included in the code stream. The decoding side can analyze the code stream and determine the same block division information, mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc., as the encoding side by analyzing the code stream based on existing information, thereby ensuring that the decoded image obtained by the encoding side is the same as the decoded image obtained by the decoding side.

[0069] The above description is the basic process of a video codec in a block-based hybrid coding framework, and as technology develops, some modules or steps of the framework or process may be optimized, and the present application is applicable to, but is not limited to, the basic process of a video codec in the block-based hybrid coding framework.

[0070] As can be seen from the prediction method of the video codec above, coded images are divided into fully intra-coded images and images that can be inter-coded, and images include frames, slices, and tiles, as shown in Figures 5 to 7. The dashed blocks in Figure 5 represent the boundaries of the largest coding unit (CTU), the solid black lines in Figure 6 represent slices, and the solid black lines in Figure 7 represent tile boundaries.

[0071] Here, all the reference information for prediction of a fully intra-coded picture is obtained from the spatial domain information of the current picture, while the prediction process of an inter-codeable picture can refer to the temporal domain reference information of other reference frames.

[0072] As can be seen from the above knowledge about video codecs, traditional loop filters include DBF, SAO and ALF, which mainly filter the reconstructed image to reduce block artifacts, linking artifacts, etc., thereby improving the quality of the reconstructed image, and the ideal situation is for the filter to restore the reconstructed image to the original image. However, many filtering coefficients of traditional filters are artificially designed, so there is a lot of room for optimization. In view of the excellent performance of deep learning tools in image processing, a filter based on deep learning is applied to the loop filter module.

[0073] The main technology in the present embodiment is a neural network-based filter, such as a neural network loop filter (NNLF). As shown in Figure 8, the image to be filtered is input to the trained neural network loop filter (NNLF) for filtering to obtain a filtered image.

[0074] In some embodiments, the training process of the neural network-based filter requires inputting an image and specifying a target image as an optimization target to train the filter parameters. During training, the input image and the target image are spatially aligned. Typically, the input image is selected from the reconstructed distorted image, and the target image is selected from the original image as the optimal optimization target.

[0075] A loss function is used in the model training process, which measures the difference between the predicted value and the actual value, and the larger the loss value, the larger the difference, and the goal of training is to reduce the loss. For deep learning-based encoding tools, commonly used loss functions include the L1 norm loss function, the L2 norm loss function, and the smooth L1 loss function.

[0076] When using a neural network-based filter in practice, typically the entire frame image is not directly input for filtering, but rather the image is divided into sub-images and the sub-images are input to the neural network-based filter in stages for filtering.

[0077] Therefore, in the training process of the neural network-based filter, the input image and the target image are divided to obtain matching pairs of input image blocks and target image blocks, and then the neural network-based filter is trained based on the matching pairs.

[0078] In some embodiments, when dividing the input image and the target image, a random cropping method is adopted to select a matching pair consisting of an input image block and a target image block from one frame of image, and a random cropping method is adopted to randomly crop the input image and the target image to obtain image blocks numbered 1 to 5, as shown in Figures 9 and 10. Then, in the actual filtering process, filtering is usually performed using a CTU as a basic unit, for example, five CTUs A to E are used to perform actual filtering, as shown in Figures 11 and 12.

[0079] As can be seen from the above description, the division manner of image blocks in the training process of the neural network-based filter does not match the division manner of image blocks in the actual use process, so the filtering effect of the neural network-based filter is reduced.

[0080] In order to solve the above technical problem, in the embodiment of the present application, in the actual filtering process, the image to be filtered is divided according to the same block division method of the training image used by the neural network-based filter in the training process to obtain N image blocks to be filtered, and for each of the N image blocks to be filtered, the neural network-based filter is used to filter the image block to be filtered to obtain a final filtered image. That is, in the embodiment of the present application, the block division method in the actual use process of the neural network-based filter is consistent with the block division method used during training, so that the neural network filter can achieve optimal filtering performance, thereby improving the filtering effect of the image.

[0081] The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. Some of the following embodiments can be combined with each other, and the same or similar concepts or processes may not be repeatedly described in some embodiments.

[0082] FIG. 13 is a flowchart of an image filtering method according to one embodiment of the present application. As shown in FIG. 13, the embodiment of the present application includes the following steps:

[0083] In step S801, an image to be filtered is obtained.

[0084] In the present embodiment, the method of obtaining the image to be filtered includes, but is not limited to, some of the following situations:

[0085] In situation 1, for a non-video codec scenario, in one example, the image to be filtered may be collected by an image acquisition device, e.g., captured by a camera, and in another example, the image to be filtered may be generated by an image generation device, e.g., rendered by an image rendering device.

[0086] In situation 2, for a video codec scenario, the manner of obtaining the image to be filtered at least includes the following manners:

[0087] In the method 1, in the case of the encoding side, the image to be filtered may be an image before being encoded, that is, before encoding the image to be filtered, the encoding side first filters the image to be filtered, and then encodes the filtered image. For example, before encoding the image to be filtered 1 on the encoding side, the image to be filtered 1 is first input to a neural network-based filter and filtered to obtain a filtered image 2. Next, the image 2 is divided into blocks to obtain a plurality of encoding blocks, and each encoding block is predicted using a prediction method such as inter-prediction or intra-prediction to obtain a predicted block of the encoding block, and the difference between the encoding block and the predicted block is obtained to obtain a residual block, and the residual block is transformed and quantized to obtain a quantization coefficient, and finally, the quantization coefficient is encoded.

[0088] In the second method, in the case of the encoding side, the image to be filtered may be a reconstructed image, that is, the current image is reconstructed to obtain a reconstructed image of the current image, and the reconstructed image is determined as the image to be filtered. For example, the encoding side performs block division on the current image, obtains a plurality of coding blocks, predicts each coding block using a prediction method such as inter prediction or intra prediction to obtain a predicted block of the coding block, obtains the difference between the coding block and the predicted block, obtains a residual block, performs transformation and quantization on the residual block, obtains a quantization coefficient, and finally encodes the quantization coefficient. Furthermore, the encoding side performs inverse quantization on the quantization parameter, obtains the transformation coefficient of the coding block, performs inverse transformation on the transformation coefficient, obtains a residual block, and adds the residual block to the predicted block to obtain a reconstructed block of the coding block. The reconstructed blocks of all the coding blocks in the current image are combined to obtain a reconstructed image of the current image. The reconstructed image is input as an image to be filtered into a neural network-based filter for filtering to obtain a filtered image.

[0089] In the third method, for the decoding side, the image to be filtered may be a reconstructed image, that is, the current image is reconstructed to obtain a reconstructed image of the current image, and the reconstructed image is determined as the image to be filtered. For example, the decoding side decodes the received code stream to obtain a quantized coefficient of a current block in the current image, then performs inverse quantization on the quantized coefficient to obtain a transform coefficient of the current block, and performs inverse transform on the transform coefficient to obtain a residual block. Furthermore, the current block is predicted using a prediction method such as inter prediction or intra prediction to obtain a predicted block of the current block, and the residual block is added to the predicted block to obtain a reconstructed block of the current block. The reconstructed blocks of all blocks in the current image are combined to obtain a reconstructed image of the current image. The reconstructed image is input as an image to be filtered into a neural network-based filter for filtering to obtain a filtered image.

[0090] In addition, in the embodiment of the present application, the method of determining the image to be filtered includes, but is not limited to, the above-mentioned methods, and the present application can also obtain the image to be filtered by other methods, and the embodiment of the present application is not limited thereto.

[0091] In step S802, a neural network-based filter is determined, and the image to be filtered is divided according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered.

[0092] Here, the above block division method is the block division method of the training image used by the neural network-based filter in the training process, that is, the image to be filtered is divided according to the same method as the block division method of the training image used by the neural network-based filter in the training process to obtain N image blocks to be filtered (N is a positive integer).

[0093] In the present embodiment, before the neural network based filter is used to filter the image to be filtered, the filter must first be determined.

[0094] In some embodiments, the neural network-based filter is a pre-configured or default filter, and thus the pre-configured or default neural network-based filter can be directly used for filtering.

[0095] In some embodiments, the present embodiment includes multiple neural network-based candidate filters, hereinafter referred to as candidate filters for short, and thus, one candidate filter can be determined as the neural network-based filter of the present embodiment from the multiple candidate filters.

[0096] In one example, the network structures of at least two of the plurality of candidate filters are not exactly the same.

[0097] In another example, at least two of the multiple candidate filters use different block division schemes of the training image in the training process. For example, during training of candidate filter 1, one CTU in the input image is determined as one input block, one CTU at the same position in the target image is determined as one target block, and the candidate filter 1 is trained with the input block as an input and the target block as a target. In another example, during training of candidate filter 2, two CTUs in the input image are determined as one input block, two CTUs at the same position in the target image are determined as one target block, and the candidate filter 2 is trained with the input block as an input and the target block as a target.

[0098] In other words, the network structures, training parameters, training methods, and other information of the multiple candidate filters in the present embodiment are not exactly the same.

[0099] In the above, the manner of determining the neural network-based filter from multiple candidate filters includes, but is not limited to, the following manners.

[0100] In method 1, any one of the candidate filters is determined as the neural network-based filter in this step.

[0101] In the method 2, among the above-mentioned multiple candidate filters, the candidate filter with the highest filtering effect is the neural network-based filter of the present application. Illustratively, each candidate filter of the above-mentioned multiple candidate filters is used to filter the image to be filtered to obtain a filtered image corresponding to each candidate filter, and these multiple filtered images are compared to determine the filtered image with the best effect, and the candidate filter corresponding to the filtered image with the best effect is determined as the neural network-based filter in this step.

[0102] In the present embodiment, there is no limitation on the method for determining the image effect of the filtered image, and the image effect of the filtered image can be determined by determining image indices such as image resolution, sharpness, artifacts, etc.

[0103] In the third method, among the above-mentioned multiple candidate filters, the candidate filter with the smallest distortion is determined as the neural network-based filter of the present application. Illustratively, each candidate filter of the above-mentioned multiple candidate filters is used to filter the image to be filtered to obtain a filtered image corresponding to each candidate filter, and the filtered image corresponding to each candidate filter is compared with the image to be filtered to determine the distortion corresponding to each candidate filter, and the candidate filter with the smallest distortion is determined as the neural network-based filter in this step.

[0104] The embodiment of the present application is not limited to a method for determining the distortion corresponding to the above candidate filter, and for example, the difference between a filtered image corresponding to the candidate filter and an image to be filtered is determined as the distortion corresponding to the candidate filter.

[0105] In the embodiment of the present application, the method of determining the neural network-based filter includes, but is not limited to, the above methods.

[0106] According to the above method, after determining the neural network-based filter, obtain the block division scheme of the training image that the neural network-based filter uses in the training process. For example, the file of the neural network-based filter includes the block division scheme of the training image that the neural network-based filter uses in the training process, and thus the block division scheme of the training image that the neural network-based filter uses in the training process can be directly read from the file of the neural network-based filter.

[0107] In the present embodiment, in order to improve the filtering performance of the neural network-based filter, the block division scheme in the actual use process of the neural network-based filter is consistent with the block division scheme used during training.

[0108] Based on this, in the actual filtering process, in the embodiment of the present application, the block division method of the training image used by the neural network-based filter in the training process is used to divide the image to be filtered to obtain N image blocks to be filtered. That is, in the embodiment of the present application, in the actual filtering, the block division method of the training image used by the neural network-based filter in the training process is used to divide the image to be filtered to obtain N image blocks to be filtered. For example, in the training process of the neural network-based filter, each CTU in the training image is determined as one training image block to perform model training, and in this way, in the actual filtering, each CTU in the image to be filtered is determined as one image block to be filtered, and filtered, so that the block division method in the actual use process of the neural network-based filter and the block division method in the training process are consistent, so that the neural network-based filter can achieve optimal filtering performance and improve the filtering effect.

[0109] The present embodiment is not limited to a specific type of block partitioning scheme of the training images that the neural network-based filter uses in the training process, and any block partitioning scheme may be used.

[0110] In some embodiments, when a block division scheme of a training image used by the above neural network-based filter for training includes determining M (M is a positive integer) CTUs in the training image as one training image block, in the above step S802, dividing the image to be filtered according to the block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered includes the following step S802-A.

[0111] In step S802-A, M CTUs in the image to be filtered are determined as one image block to be filtered, thereby obtaining N image blocks to be filtered.

[0112] That is, N image blocks to be filtered are determined from the image to be filtered, with M CTUs being treated as one image block to be filtered.

[0113] The present embodiment is not limited to the specific value of M above.

[0114] In Example 1, M=1, and as shown in FIG. 14, in the training process of the neural network-based filter, one CTU of the training image is determined as one training image block to train the neural network-based filter, where the area of ​​one smallest dashed block in FIG. 14 is one CTU.

[0115] In one possible implementation, when the neural network-based filter is obtained by training in a supervised training method, the training image includes an input image and a target image, where the target image can be understood as a teacher image. In the training process, as shown in FIG. 15, one CTU of the input image is determined as one input image block, and as shown in FIG. 16, one CTU of the target image is determined as one target image block, and the input image block and the target image block form one matching pair, that is, the position of the input image block in one matching pair in the input image is consistent with the position of the target image block in the target image. Then, the input image block is input to the neural network-based filter for filtering to obtain a filtered image block corresponding to the input image block. The filtered image block of the input image block is compared with the target image block to calculate a loss, and the parameters of the neural network-based filter are adjusted according to the loss. Then, referring to the above method, the input image block in the next matching pair is used as an input, and the target image block in the next matching pair is used as a target to train the neural network-based filter to obtain a trained neural network-based filter.

[0116] As can be seen from the above description, in Example 1, in the training process of the above neural network-based filter, one CTU in the training image is determined as one training image block, and correspondingly, in the actual filtering process, one CTU in the image to be filtered is determined as one image block to be filtered, as shown in FIG. 17, in this way, block division is performed on the image to be filtered, and N image blocks to be filtered are obtained.

[0117] In Example 2, M=4, and in the training process of the neural network-based filter, as shown in FIG. 18, four CTUs of the training image are determined as one training image block to train the neural network-based filter.

[0118] In one possible implementation, when the neural network-based filter is obtained by training in a supervised training method, the training image includes an input image and a target image, where the target image can be understood as a teacher image. In a specific training process, as shown in FIG. 19, four CTUs of an input image are determined as one input image block, and as shown in FIG. 20, four CTUs of a target image are determined as one target image block, and the input image block and the target image block form one matching pair, that is, the position of the input image block in one matching pair in the input image is consistent with the position of the target image block in the target image. Then, the input image block is input to the neural network-based filter for filtering to obtain a filtered image block corresponding to the input image block. The filtered image block of the input image block is compared with the target image block to calculate a loss, and the parameters of the neural network-based filter are adjusted according to the loss. Next, referring to the above method, the neural network-based filter is trained by continuing to use the input image block in the next matching pair as an input and the target image block in the next matching pair as a target to obtain a trained neural network-based filter.

[0119] As can be seen from the above description, in Example 2, in the training process of the above neural network-based filter, four CTUs in the training image are determined as one training image block, and correspondingly, in the actual filtering process, four CTUs in the image to be filtered are determined as one image block to be filtered, as shown in FIG. 21, and thus N image blocks to be filtered are obtained.

[0120] In the above, M is described as 1 or 4 as an example, but in some embodiments, M may be any positive integer such as 2, 3, 5, etc., and the embodiments of the present application are not limited thereto.

[0121] In some embodiments, when the block division method of the training image used by the above neural network-based filter in training includes determining P (P is a positive integer) incomplete CTUs in the training image as one training image block, in the above step S802, dividing the image to be filtered according to the same method as the block division method of the training image used by the neural network-based filter in the training process to obtain N image blocks to be filtered includes the following step S802-B.

[0122] In S802-B, N image blocks to be filtered are obtained by determining the P incomplete CTUs in the image to be filtered as one image block to be filtered.

[0123] That is, P incomplete CTUs are regarded as one image block to be filtered, and N image blocks to be filtered are determined from the image to be filtered.

[0124] The present embodiment is not limited to the specific value of P above.

[0125] For example, assuming P=4, in the training process of the neural network-based filter, as shown in FIG. 22, four incomplete CTUs in the training image are determined as one training image block to train the neural network-based filter.

[0126] In one possible implementation, when the neural network-based filter is obtained by training in a supervised training method, the training image includes an input image and a target image, where the target image can be understood as a teacher image. In a specific training process, as shown in FIG. 23, four incomplete CTUs of an input image are determined as one input image block, and as shown in FIG. 24, four incomplete CTUs of a target image are determined as one target image block, and the input image block and the target image block form one matching pair. Then, the input image block is input to the neural network-based filter for filtering to obtain a filtered image block corresponding to the input image block. The filtered image block of the input image block is compared with the target image block to calculate a loss, and the parameters of the neural network-based filter are adjusted according to the loss. Then, referring to the above method, the input image block in the next matching pair is used as an input, and the target image block in the next matching pair is used as a target to train the neural network-based filter to obtain a trained neural network-based filter.

[0127] As can be seen from the above description, in this example, in the training process of the above neural network-based filter, four incomplete CTUs in the training image are determined as one training image block, and correspondingly, in the actual filtering process, four incomplete CTUs in the image to be filtered are determined as one image block to be filtered, as shown in FIG. 25, and thus N image blocks to be filtered are obtained.

[0128] In the above, P is described as 4 as an example, but in some embodiments, P may be any positive integer such as 1, 2, 3, 5, etc., and the embodiments of the present application are not limited thereto.

[0129] In the above example, the four incomplete CTUs are adjacent, and in some embodiments, the P residual CTUs may not be completely adjacent when the above P is greater than 1. That is, all of the P residual CTUs are not adjacent to each other, or some of the P residual CTUs are adjacent to each other and some of the P residual CTUs are not adjacent to each other.

[0130] The above embodiment shows that the block division method of the training image includes determining M CTUs in the training image as one training image block, or determining P incomplete CTUs in the training image as one training image block. Note that the block division method of the training image according to the embodiment of the present application includes, but is not limited to, the above examples, and the embodiment of the present application is not limited thereto.

[0131] Through the above steps, the neural network-based filter divides the image to be filtered into N blocks of image to be filtered using the block division method of the training image used in the training process, and then performs the following step S803.

[0132] In step S803, a neural network-based filter is used to filter the N image blocks to be filtered respectively to obtain a filtered image.

[0133] In the present embodiment, the image to be filtered is divided using the same block division method as that of the training image to obtain N image blocks to be filtered. For each of the N image blocks to be filtered, the image block to be filtered is input to a neural network-based filter to be filtered to obtain a filtered image block of the image block to be filtered. According to the above method, a filtered image block of each of the N image blocks to be filtered can be determined, and the filtered image blocks of these N image blocks to be filtered constitute a filtered image.

[0134] In the filtering process, the filtering effect of the boundary area of ​​the image block can be improved by filtering based on the adjacent content outside the boundary of the image block. In view of this, in order to further improve the filtering effect of the neural network-based filter, in some embodiments, the above neural network-based filter is obtained by training using the extended image block of the training image block. That is, in the filter training process in the present embodiment, in addition to dividing the training image according to the block division manner of the training image to obtain the training image block, the training image block can also be extended outward to obtain the extended image block, and the extended image block can be used to train the neural network-based filter. In this case, the above step S803 includes the following steps S803-A1 to S803-A3.

[0135] In step S803-A1, for each image block to be filtered of the N image blocks to be filtered, extend the image block to be filtered according to the extension manner of the training image block to obtain an extended image block to be filtered.

[0136] In step S803-A2, the enhanced to-be-filtered image block is filtered using a neural network-based filter to obtain a filtered enhanced image block.

[0137] In step S803-A3, an image area corresponding to the image block to be filtered in the filtered extended image block is determined as the filtered image block of the image block to be filtered.

[0138] In the embodiment of the present application, in order to further improve the filtering effect of the image to be filtered, the same extension method as that of the training image block is adopted to extend the image block to be filtered. Specifically, in the actual filtering process, the same block division method as that of the training image is used to divide the image to be filtered to obtain N image blocks to be filtered. For each of the N image blocks to be filtered, the same extension method as that of the training image block is adopted to extend the image block to be filtered outward to obtain an extended image block to be filtered. Then, the extended image block to be filtered is input to a neural network-based filter for filtering to obtain a filtered extended image block of the extended image block to be filtered. Since the filtered extended image block does not match the size of the image block to be filtered, the filtered extended image block is cropped, specifically, the image area in the filtered extended image block corresponding to the image block to be filtered is determined as the filtered image block of the image block to be filtered. According to the above steps, the filtered image block of each image block to be filtered of the N image blocks to be filtered can be determined, and the filtered image blocks of these N image blocks to be filtered are stitched together to obtain a final filtered image.

[0139] The present embodiment is not limited to a specific extension method for training image blocks.

[0140] In some embodiments, the extension scheme of the training image block includes extending at least one boundary region of the training image block outward, in which case in step S803-A1, extending the image block to be filtered according to the extension scheme of the training image block to obtain an extended image block to be filtered includes extending at least one boundary region of the image block to be filtered outward to obtain an extended image block to be filtered.

[0141] In some examples, as shown in FIG. 26, the training process of the neural network-based filter includes extending the perimeter of the training image block outward and using the extended training image block to train the neural network-based filter.

[0142] In one possible implementation, when the neural network-based filter is obtained by training in a supervised training method, the training image includes an input image and a target image, where the target image can be understood as a teacher image. In a specific training process, as shown in FIG. 27, one CTU of the input image is determined as one input image block, and the periphery of the input image block is expanded outward to obtain an expanded input image block. As shown in FIG. 28, one CTU of the target image is determined as one target image block, and the periphery of the target image block is expanded outward to obtain an expanded target image block. Then, the expanded input image block is input to the neural network-based filter for filtering to obtain a filtered image block corresponding to the expanded input image block. The filtered image block of the expanded input image block is compared with the expanded target image block to calculate a loss, and the parameters of the neural network-based filter are adjusted according to the loss. Next, referring to the above method, the neural network-based filter is trained by continuing to take the dilated input image block in the next matching pair as input and the dilated target image block in the next matching pair as target to obtain a trained neural network-based filter.

[0143] As can be seen from the above description, in this example, in the training process of the above neural network-based filter, one CTU in the training image is determined as one training image block, and the periphery of the training image block is extended outward to obtain an extended training image block. Correspondingly, in the actual filtering process, as shown in Fig. 29, one CTU in the image to be filtered is determined as one image block to be filtered, and then the periphery of the image block to be filtered is extended outward using the above training image block extension method to obtain an extended image block to be filtered, and then the extended image block to be filtered is input to the neural network-based filter for filtering to obtain a filtered extended image block, and further, the image area of ​​the filtered extended image block corresponding to the image block to be filtered is determined as the filtered image block of the image block to be filtered.

[0144] In some examples, as shown in FIG. 30, the training process of the neural network-based filter includes extending the left and top boundaries of the training image block outward, and using the extended training image block to train the neural network-based filter.

[0145] In one possible implementation, when the neural network-based filter is obtained by training in a supervised training method, the training image includes an input image and a target image, where the target image can be understood as a teacher image. In a specific training process, as shown in FIG. 31, one CTU of the input image is determined as one input image block, and the left and upper boundaries of the input image block are extended outward to obtain an extended input image block. As shown in FIG. 32, one CTU of the target image is determined as one target image block, and the left and upper boundaries of the target image block are extended outward to obtain an extended target image block. Then, the extended input image block is input to the neural network-based filter for filtering to obtain a filtered image block corresponding to the extended input image block. The filtered image block of the extended input image block is compared with the extended target image block to calculate a loss, and the parameters of the neural network-based filter are adjusted according to the loss. Next, referring to the above method, the neural network-based filter is trained using the dilated input image block in the next matching pair as input and the dilated target image block in the next matching pair as target to obtain a trained neural network-based filter.

[0146] As can be seen from the above description, in this example, in the training process of the above neural network-based filter, a CTU in the training image is determined as a training image block, and the left and upper boundaries of the training image block are extended outward to obtain an extended training image block. Correspondingly, in the actual filtering process, as shown in Fig. 33, a CTU in the image to be filtered is determined as a filtering image block, and then the left and upper boundaries of the filtering image block are extended outward using the above training image block extension method to obtain an extended filtering image block, and then the extended filtering image block is input to the neural network-based filter for filtering to obtain a filtered extended image block, and further, the image area of ​​the filtered extended image block corresponding to the filtering image block is determined as the filtered image block of the filtering image block.

[0147] In some examples, the expanding scheme of the training image block further includes expanding other boundaries of the training image block outward, although embodiments of the present application are not limited thereto.

[0148] In some embodiments, in order to further improve the filtering effect of the neural network-based filter, in addition to inputting an input image block, a reference image block of the input image block is also input in the training process. In order to maintain the consistency between the actual filtering process and the training process, the above step S803 includes the following steps S803-B1 to S803-B2.

[0149] In step S803-B1, for each of the N image blocks to be filtered, a reference image block of the image block to be filtered is determined.

[0150] In step S803-B2, the image block to be filtered and a reference image block of the image block to be filtered are input to a neural network-based filter for filtering to obtain a filtered image block of the image block to be filtered.

[0151] In the present embodiment, in the training process of the neural network-based filter, when the input information includes an input image block and a reference image block of the input image block, in the actual filtering process, in addition to the image block to be filtered, the input information further includes a reference image block of the image block to be filtered.

[0152] The embodiment of the present invention is not limited to the above method for determining the reference image block of the image block to be filtered.

[0153] In some embodiments, the manner of determining the reference image blocks of the image blocks to be filtered is different from the manner of determining the reference image blocks of the training image blocks.

[0154] In some embodiments, the method of determining the reference image block of the image block to be filtered is the same as the method of determining the reference image block of the input image block. In this case, in the above step S803-B1, determining the reference image block of the image block to be filtered includes the following steps S803-B11 to S803-B12.

[0155] In step S803-B11, a determination method of a reference image block of an input image block is obtained, and the determination method is used to determine a corresponding reference image block based on at least one of spatial domain information and temporal domain information of the input image block.

[0156] In step S803-B12, a reference image block of the image block to be filtered is determined according to the method of determining a reference image block of the input image block.

[0157] In the present embodiment, the method of determining the reference image block of the input image block can be read from the file of the neural network-based filter. In some embodiments, when the training device of the neural network-based filter is the same device as the actual filtering device, the device stores the method of determining the reference image block of the input image block.

[0158] After obtaining the determination method of the reference image block of the input image block, the determination method of the reference image block of the input image block is used to determine the reference image block of the image block to be filtered. That is, in the embodiment of the present application, the determination method of the reference image block of the image block to be filtered is consistent with the determination method of the reference image block of the input image block.

[0159] The present embodiment is not limited to a specific type of reference image block.

[0160] In some embodiments, when the reference image block of the input image block includes at least one of a time domain reference image block and a spatial domain reference image block of the input image block, the reference image block of the image block to be filtered includes at least one of a time domain reference image block and a spatial domain reference image block of the image block to be filtered.

[0161] Here, the spatial domain reference image block may select an image area at a fixed relative position of the current input image block, i.e., the spatial domain reference image block of the input image block is in the same frame as the input image block, i.e., both are in the input image.

[0162] The difference between the time domain reference image block and the spatial domain reference image block is that the time domain reference image block and the current input image block are in different frames, and here, the reference position of the time domain reference image block may select a reference block at the same spatial position as the current input image.

[0163] In this embodiment, the type of the reference image block of the image block to be filtered is the same as the type of the reference image block of the input image block.

[0164] In example 1, if the reference image block of the input image block includes a spatial domain reference image block, the reference image block of the image block to be filtered also includes a spatial domain reference image block. In this case, the above step S803-B12 includes the following steps:

[0165] In step S803-B12-A, the spatial domain reference image block of the image block to be filtered is determined according to the manner of determining the spatial domain reference image block of the input image block.

[0166] In this example, when the reference image block of the input image block includes a spatial domain reference image block, the spatial domain reference image block of the image block to be filtered is determined according to a method for determining the spatial domain reference image block of the input image block, thereby enabling the spatial domain reference image block of the image block to be filtered to be accurately determined.

[0167] In one possible implementation form, a method for determining a spatial domain reference image block of an input image block is used to determine a spatial domain reference image block of an image block to be filtered, thereby ensuring that the input information in the training process of the neural network-based filter is consistent with the input information in the actual filtering process, and improving the filtering performance of the neural network-based filter.

[0168] The present embodiment is not limited to a specific type of spatial domain reference image block.

[0169] In some embodiments, if the spatial domain reference image block of the input image block includes at least one of the image block located at the upper left, the image block located at the left, and the image block located at the top of the input image block in the input image, the spatial domain reference image block of the image block to be filtered includes at least one of the image block located at the upper left, the image block located at the left, and the image block located at the top of the image block to be filtered in the image to be filtered. In this case, step S803-B12-A includes determining at least one of the image block located at the upper left, the image block located at the left, and the image block located at the top of the image block to be filtered in the image to be filtered as the spatial domain reference image block of the image block to be filtered.

[0170] For example, as shown in Figure 34, if the spatial domain reference image block of an input image block includes an image block located to the upper left of the input image block in the input image, as shown in Figure 35, the image block located to the upper left of the image block to be filtered in the image to be filtered is determined as the spatial domain reference image block of the image to be filtered.

[0171] In another example, when the spatial domain reference image block of an input image block includes an image block located to the left of the input image block in the input image, as shown in FIG. 36, an image block located to the left of the image block to be filtered in the image to be filtered is determined as the spatial domain reference image block of the image to be filtered, as shown in FIG. 37.

[0172] In another example, when the spatial domain reference image block of an input image block includes an image block located above the input image block in the input image, as shown in FIG. 38, an image block located above the image block to be filtered in the image to be filtered is determined as the spatial domain reference image block of the image to be filtered, as shown in FIG. 39.

[0173] In another example, as shown in Figure 40, when the spatial domain reference image blocks of the input image block include an image block located to the upper left, an image block located to the left, and an image block located at the top of the input image block in the input image, as shown in Figure 41, the image block located to the upper left, an image block located to the left, and an image block located at the top of the image block to be filtered in the image to be filtered are determined as the spatial domain reference image blocks of the image to be filtered.

[0174] In the embodiment of the present application, in order to further improve the filtering effect, in addition to inputting an input image block, in the training process, the spatial domain reference image block of the input image block is also input, thereby improving the filtering effect of the neural network-based filter. In this way, in the actual filtering process, in order to maintain the consistency between the input information in the actual filtering process and the input information in the training process, the same determination method as that of the spatial domain reference image block of the input image block is adopted to determine the spatial domain reference image block of the image block to be filtered, and then the image block to be filtered and the spatial domain reference image block of the image block to be filtered are input into the neural network-based filter to realize filtering for the image to be filtered.

[0175] In example 2, if the reference image block of the input image block includes a time-domain reference image block, the reference image block of the image block to be filtered also includes a time-domain reference image block. In this case, the above step S803-B12 includes the following steps:

[0176] In step S803-B12-B, a time domain reference image block of the image block to be filtered is determined according to a manner of determining a time domain reference image block of the input image block.

[0177] In this example, when the reference image block of the input image block includes a time domain reference image block, the time domain reference image block of the image block to be filtered is determined according to the determination method of the time domain reference image block of the input image block, so that the time domain reference image block of the image block to be filtered can be accurately determined.

[0178] In one possible implementation form, a method for determining a time-domain reference image block of an input image block is used to determine a time-domain reference image block of an image block to be filtered, thereby ensuring that the input information in the training process of the neural network-based filter is consistent with the input information in the actual filtering process, and improving the filtering performance of the neural network-based filter.

[0179] In some embodiments, step S803-B12-B includes determining a reference image of the image to be filtered, and determining an image block located in the reference image of the image to be filtered corresponding to the image block to be filtered as a time-domain reference image block of the image block to be filtered.

[0180] For example, as shown in Figure 42 and Figure 43, the time domain reference image block of an input image block is an image block at a position corresponding to the input image block in the reference image of the input image. That is, the position of the time domain reference image block of the input image block in the reference image of the input image is the same as the position of the input image block in the input image. In this case, as shown in Figure 44 and Figure 45, the process of determining the time domain reference image block of the image block to be filtered is to first determine the reference image of the filtered image, and determine the image block at a position corresponding to the image block to be filtered in the reference image of the filtered image as the time domain reference image block of the image block to be filtered.

[0181] The present embodiment is not limited to the type of reference image; for example, when the method of the present embodiment is applied to the encoding side, the reference image of the above-mentioned image to be filtered may be any encoded image, and when the method of the present embodiment is applied to the decoding side, the reference image of the above-mentioned image to be filtered may be any decoded image.

[0182] In the embodiment of the present application, in order to further improve the filtering effect, in addition to inputting an input image block, in the training process, the time domain reference image block of the input image block is also input, thereby improving the filtering effect of the neural network-based filter. In this way, in the actual filtering process, in order to maintain the consistency between the input information in the actual filtering process and the input information in the training process, the same determination method as that of the time domain reference image block of the input image block is adopted to determine the time domain reference image block of the image block to be filtered, and then the image block to be filtered and the time domain reference image block of the image block to be filtered are input into the neural network-based filter, thereby realizing filtering of the image to be filtered.

[0183] In some embodiments, when the reference image block of the input image block includes a spatial domain reference image block and a time domain reference image block, the reference image block of the image block to be filtered also includes a spatial domain reference image block and a time domain reference image block, where the process of determining the spatial domain reference image block and the time domain reference image block of the image block to be filtered can refer to the above process of determining the spatial domain reference image block and the time domain reference image block, and will not be repeated here.

[0184] According to the above method, the neural network-based filter is used to filter each of the N image blocks to be filtered to obtain a filtered image.

[0185] In some embodiments, the filtering method of the present embodiment can be applied to a loop filtering module, in which case the method of the present embodiment further includes: generating a reference image for prediction based on the filtered image, and storing the generated reference image in a decoding cache for use as a reference image for a subsequent decoded image. Here, the manner of generating a reference image based on the filtered image may be to directly use the filtered image as a reference image, or to reprocess the filtered image, for example, by other filtering, and use the reprocessed image as a reference image. Optionally, the filtered image can also be displayed by a display device.

[0186] In some embodiments, the method of the present application can also be applied to video post-processing, i.e., generate a display image based on the filtered image, input the generated display image to a display device for display, and skip storing the filtered image or a reprocessed image of the filtered image in the decoding cache. That is, in the present application, the display image generated based on the filtered image is input to a display device for display, but is not stored in the decoding cache as a reference image. Exemplarily, after determining a reconstructed image of a current image by decoding a video, the reconstructed image is stored in the decoding cache as a reference image, or the reconstructed image is filtered using at least one filter, such as DBF, SAO, and ALF, using a conventional loop filtering method, and the filtered image is stored in the decoding cache as a reference image. Then, the reconstructed image is taken as an image to be filtered, and the reconstructed image is filtered using a neural network-based filter according to the method of the present application to obtain a filtered image, a display image is generated based on the filtered image, and the display image is input to a display device for display.

[0187] In some embodiments, the filtered image may be further filtered by at least one filter such as DBF, SAO, and ALF.

[0188] In some embodiments, the image to be filtered in the present embodiment may be an image that has been filtered by at least one filter such as DBF, SAO, and ALF, and then the method of the present embodiment further filters the filtered image using a neural network-based filter.

[0189] In the image filtering method according to the embodiment of the present application, an image to be filtered is obtained, a neural network-based filter is determined, and the image to be filtered is divided according to the same block division method of the training image used by the neural network-based filter in the training process to obtain N (N is a positive integer) image blocks to be filtered, and the neural network-based filter is used to filter the N image blocks to be filtered respectively to obtain a filtered image. That is, in the embodiment of the present application, the block division method in the actual use process of the neural network-based filter is matched with the block division method used during training, so that the neural network filter can achieve optimal filtering performance, thereby improving the filtering effect of the image.

[0190] 46 is an exemplary flowchart of an image filtering method according to one embodiment of the present application. FIG. 46 may be understood as one specific embodiment of the filtering method shown in FIG. 13 above.

[0191] As shown in FIG. 46, the image filtering method according to the present embodiment includes the following steps:

[0192] In step S901, an image to be filtered is obtained.

[0193] In the case of a non-video codec scenario, the image to be filtered may be one that has been collected by an image acquisition device or one that has been rendered by an image rendering device.

[0194] In the case of a video codec scenario, the image to be filtered may be a reconstructed image.

[0195] For a specific embodiment of the above step S901, please refer to the description of the above step S801, and the description will not be repeated here.

[0196] In step S902, a neural network-based filter is determined, and the image to be filtered is divided according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered.

[0197] Here, the block division scheme is the block division scheme of the training image used by the neural network-based filter in the training process, that is, the image to be filtered is divided according to the same scheme as the block division scheme of the training image used by the neural network-based filter in the training process to obtain N image blocks to be filtered (N is a positive integer).

[0198] For example, the block division method of the training image used by the neural network-based filter in the training process is to determine one CTU in the training image as one training image block, and then, during actual filtering, N image blocks to be filtered are obtained by determining one CTU of the image to be filtered as one image block to be filtered.

[0199] For a specific embodiment of the above step S902, please refer to the description of the above step S802, and the description will not be repeated here.

[0200] In step S903, for each of the N image blocks to be filtered, a reference image block of the image block to be filtered is determined according to the method of determining a reference image block of an input image block.

[0201] In this embodiment, the method of determining the reference image block of the image block to be filtered is the same as the method of determining the reference image block of the input image block.

[0202] In some embodiments, when the reference image block of the input image block includes at least one of a time domain reference image block and a spatial domain reference image block of the input image block, the reference image block of the image block to be filtered includes at least one of a time domain reference image block and a spatial domain reference image block of the image block to be filtered.

[0203] For a specific embodiment of the above step S903, please refer to the description of the above step S803-B12, and the description will not be repeated here.

[0204] In step S904, the image block to be filtered and the reference image block of the image block to be filtered are input to a neural network-based filter for filtering to obtain a filtered image.

[0205] In the embodiment of the present application, in order to further improve the filtering effect, in addition to inputting an input image block, in the training process, the reference image block of the input image block is also input, thereby improving the filtering effect of the neural network-based filter. In this way, in the actual filtering process, in order to maintain the consistency between the input information in the actual filtering process and the input information in the training process, the same determination method as that of the reference image block of the input image block is adopted to determine the reference image block of the image block to be filtered, and then the image block to be filtered and the reference image block of the image block to be filtered are input into the neural network-based filter to realize filtering of the image to be filtered.

[0206] According to the image filtering method of the embodiment of the present application, the image to be filtered is divided in the same manner as the block division manner of the training image used by the neural network-based filter in the training process to obtain N image blocks to be filtered, and for each of the N image blocks to be filtered, the reference image block of the image block to be filtered is determined according to the determination manner of the reference image block of the input image block, and the image block to be filtered and the reference image block of the image block to be filtered are input to the neural network-based filter for filtering to obtain a filtered image. That is, in the embodiment of the present application, the block division manner in the actual use process of the neural network-based filter is consistent with the block division manner used during training, and the determination manner of the reference image block of the input image block is consistent with the determination manner of the reference image block of the image block to be filtered, so that the filtering effect of the neural network-based filter can be further improved.

[0207] It should be understood that Figures 13-46 are merely examples of the present application and should not be construed as limiting the present application.

[0208] An embodiment of the method of the present application has been described in detail above with reference to FIGS. 13 to 46, and an embodiment of the apparatus of the present application will be described below with reference to FIGS.

[0209] FIG. 47 is an exemplary block diagram of an image filtering device according to one embodiment of the present application. The device 10 may be an electronic device or a part of an electronic device. As shown in FIG. 47, the image filtering device 10 includes: an acquisition unit 11 configured to acquire a ready-to-filter image; a segmentation unit 12 configured to determine a neural network-based filter, and segment the image to be filtered according to a block segmentation scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, where N is a positive integer, and the block segmentation scheme is a block segmentation scheme of a training image used by the neural network-based filter in a training process; and a filtering unit 13 configured to filter each of the N image blocks to be filtered using a neural network based filter to obtain a filtered image.

[0210] In some embodiments, the block division scheme of the training image includes determining M CTUs in the training image as one training image block, where M is a positive integer; The division unit 12 is configured to obtain N image blocks to be filtered by determining the M CTUs in the image to be filtered as one image block to be filtered.

[0211] In some embodiments, the block division scheme of the training image includes determining P incomplete CTUs in the training image, where P is a positive integer, as one training image block; The division unit 12 is configured to obtain N image blocks to be filtered by determining the P incomplete CTUs in the image to be filtered as one image block to be filtered.

[0212] In some embodiments, the neural network based filter is obtained by training it using an extended image block of the training image block; The filtering unit 13 is configured to, for each image block to be filtered of the N image blocks to be filtered, extend the image block to be filtered according to the extension scheme of the training image block to obtain an extended image block to be filtered, filter the extended image block to be filtered using a neural network-based filter to obtain a filtered extended image block, and determine an image region in the filtered extended image block corresponding to the image block to be filtered as the filtered image block corresponding to the image block to be filtered.

[0213] In some embodiments, the expanding scheme of the training image block comprises expanding outwardly at least one boundary region of the training image block; The filtering unit 13 is configured to extend outwardly at least one boundary region of the image block to be filtered to obtain an extended image block to be filtered.

[0214] In some embodiments, the training image comprises an input image, and the input data for training the neural network-based filter comprises an input image block and a reference image block for the input image block, the input image block being obtained by performing image segmentation on the input image using a block segmentation scheme; The filtering unit 13 is configured to determine, for each image block to be filtered of the N image blocks to be filtered, a reference image block of the image block to be filtered, and input the image block to be filtered and the reference image block of the image block to be filtered into a neural network-based filter for filtering to obtain a filtered image block of the image block to be filtered.

[0215] In some embodiments, the filtering unit 13 is configured to obtain a determination scheme of a reference image block of an input image block, the determination scheme being used to determine a corresponding reference image block based on at least one of spatial domain information and time domain information of the input image block, and to determine a reference image block of an image block to be filtered according to the determination scheme of the reference image block of the input image block.

[0216] In some embodiments, the reference image block of the input image block includes at least one of a time domain reference image block and a spatial domain reference image block of the input image block, and the reference image block of the image block to be filtered includes at least one of a time domain reference image block and a spatial domain reference image block of the image block to be filtered.

[0217] In some embodiments, the reference image blocks of the input image block also include spatial domain reference image blocks of the input image block; The filtering unit 13 is configured to determine the spatial domain reference image block of the image block to be filtered according to a determining manner of the spatial domain reference image block of the input image block.

[0218] In some embodiments, the spatial domain reference image block of the input image block includes at least one of an image block located to the upper left of the input image block, an image block located to the left of the input image block, and an image block located at the top of the input image block in the input image; The filtering unit 13 is configured to determine at least one of an image block located to the upper left of the image block to be filtered, an image block located to the left, and an image block located at the top in the image to be filtered as a spatial domain reference image block of the image block to be filtered.

[0219] In some embodiments, the reference image blocks of the input image block also include time-domain reference image blocks of the input image block; The filtering unit 13 is configured to determine the time-domain reference image block of the image block to be filtered according to a determining manner of the time-domain reference image block of the input image block.

[0220] In some embodiments, the time-domain reference image block of the input image block comprises an image block in a reference image of the input image at a position corresponding to the input image block; The filtering unit 13 is specifically configured to determine a reference image of the image to be filtered, and determine an image block at a position corresponding to the image block to be filtered in the reference image of the image to be filtered as a time-domain reference image block of the image block to be filtered.

[0221] In some embodiments, the training images include an input image and a target image corresponding to the input image, and the neural network-based filter is obtained by training using the input image blocks as input data and the target image blocks as targets, the input image blocks are obtained by performing image segmentation on the input image using the block partitioning scheme of the training images, and the target image blocks are obtained by performing image segmentation on the target image using the block partitioning scheme of the training images.

[0222] In some embodiments, the acquisition unit 11 is configured to reconstruct a current image to obtain a reconstructed image of the current image, and determine the reconstructed image as the image to be filtered.

[0223] In some embodiments, the filtering unit 13 is further configured to generate, based on the filtered image, a reference image for prediction, and store it in the decoding cache.

[0224] In some embodiments, the filtering unit 13 is further configured to generate a display image based on the filtered image, input the display image to a display device for display, and skip storing the filtered image or a reprocessed image of the filtered image in the decoding cache.

[0225] FIG. 48 is an exemplary block diagram of an electronic device according to an embodiment of the present disclosure, configured to perform the above-described method embodiment. As shown in FIG. 48, the electronic device 30 includes: The computer system may include a memory 31 and a processor 32, and the memory 31 is configured to store a computer program 33 and transmit the computer program 33 to the processor 32. In other words, the processor 32 can realize the method in the embodiment of the present application by calling and executing the computer program 33 from the memory 31.

[0226] For example, the processor 32 may be configured to carry out the steps of the above-mentioned methods according to instructions in the computer program 33 .

[0227] In some embodiments of the present application, the processor 32 may include: This may include, but is not limited to, a general purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic devices, discrete hardware components, and the like.

[0228] In some embodiments of the present application, the memory 31 includes: It may include, but is not limited to, volatile memory and / or non-volatile memory. Here, non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), used as an external cache. By way of illustrative but non-limiting example, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synch Link DRAM (SLDRAM), and Direct Memory Bus Random Access Memory (DR RAM).

[0229] In some embodiments of the present application, the computer program 33 can be divided into one or more modules, which are stored in the memory 31 and executed by the processor 32 to complete the method of the present application. The one or more modules can be a series of computer program instruction segments that can complete a specific function, and the instruction segments are used to explain the execution process of the computer program 33 in the electronic device.

[0230] As shown in FIG. 48, the electronic device 30 further includes: A transceiver 34 may be provided which may be connected to the processor 32 or the memory 31 .

[0231] Here, the processor 32 can control the transceiver 34 to communicate with other devices, and can exemplarily transmit information or data to other devices or receive information or data transmitted by other devices. The transceiver 34 can include a transmitter and a receiver. The transceiver 34 can further include an antenna, and the number of antennas can be one or more.

[0232] It should be understood that the components of the electronic device 30 are connected by a bus system, which may include a power bus, a control bus, and a status signal bus in addition to a data bus.

[0233] According to one aspect of the present application, there is provided a computer storage medium having a computer program stored thereon, the computer program causing a computer to perform the method in the method embodiments described above.

[0234] An embodiment of the present application further provides a computer program product comprising instructions for causing a computer to perform the method in any of the method embodiments described above.

[0235] According to another aspect of the present application, there is provided a computer program product or computer program comprising computer instructions stored on a computer readable storage medium, the computer instructions being read by a processor of an electronic device from the computer readable storage medium and executed by the processor to cause the electronic device to perform the method in the method embodiment described above.

Claims

1. 1. A method of image filtering performed by an electronic device, comprising: obtaining an image to be filtered; determining a neural network based filter; Dividing the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, where N is a positive integer, and the block division scheme is a block division scheme of a training image used by the neural network-based filter in a training process; filtering each of the N image blocks to be filtered using the neural network based filter to obtain a filtered image; The image filtering method includes:

2. The block division method of the training image includes determining M coding tree units in the training image as one training image block, where M is a positive integer; Dividing the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, determining the N image blocks to be filtered from the image to be filtered in a manner that M coding tree units are one image block to be filtered; The image filtering method according to claim 1 .

3. The block division method of the training image includes determining P incomplete coding tree units in the training image as one training image block, where P is a positive integer; Dividing the image to be filtered according to a block division scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, determining the N image blocks to be filtered from the image to be filtered in a manner that P incomplete coding tree units are one image block to be filtered; The image filtering method according to claim 1 .

4. the neural network based filter is obtained by training it with an extended image block of a training image block; The step of filtering each of the N image blocks to be filtered using the neural network-based filter to obtain a filtered image includes: For each of the N image blocks to be filtered, extend the image block to be filtered according to an extension manner of the training image block to obtain an extended image block to be filtered; filtering the enhanced to-be-filtered image block using the neural network based filter to obtain a filtered enhanced image block; determining an image region in the filtered extended image block corresponding to the image block to be filtered as a filtered image block corresponding to the image block to be filtered; The image filtering method according to claim 1 .

5. The training image block expansion scheme includes expanding at least one boundary region of the training image block outward; The step of extending the image block to be filtered according to the extension manner of the training image block to obtain an extended image block to be filtered includes: dilating outward at least one boundary region of the image block to be filtered to obtain the dilated image block to be filtered.

5. The image filtering method according to claim 4.

6. the training image includes an input image, and input data for training the neural network-based filter includes an input image block and a reference image block for the input image block, the input image block being obtained by performing image segmentation on the input image using the block segmentation scheme; The step of filtering each of the N image blocks to be filtered using the neural network-based filter to obtain a filtered image includes: determining, for each of the N image blocks to be filtered, a reference image block for the image block to be filtered; inputting the image block to be filtered and a reference image block of the image block to be filtered into the neural network-based filter for filtering to obtain a filtered image block of the image block to be filtered; The image filtering method according to claim 1 .

7. The step of determining a reference image block of the image block to be filtered includes: obtaining a determination scheme of a reference image block of the input image block, the determination scheme being used to determine a corresponding reference image block based on at least one of spatial domain information and temporal domain information of the input image block; determining a reference image block of the image block to be filtered according to a method for determining a reference image block of the input image block; The image filtering method according to claim 6.

8. the reference image block of the input image block includes at least one of a time domain reference image block and a spatial domain reference image block of the input image block; The reference image block of the image block to be filtered includes at least one of a time domain reference image block and a spatial domain reference image block of the image block to be filtered. The image filtering method according to claim 7.

9. The step of determining a reference image block of the image block to be filtered according to a method of determining a reference image block of the input image block includes: determining a spatial domain reference image block of the image block to be filtered according to a spatial domain reference image block determination manner of the input image block; The image filtering method according to claim 8.

10. the spatial domain reference image block of the input image block includes at least one of an image block located to the upper left of the input image block, an image block located to the left of the input image block, and an image block located at the top of the input image block in the input image; The step of determining a spatial domain reference image block of the image block to be filtered according to a spatial domain reference image block determination manner of the input image block includes: determining at least one of an image block located to the upper left of the image block to be filtered, an image block located to the left of the image block to be filtered, and an image block located at an upper part of the image block to be filtered in the image to be filtered as a spatial domain reference image block of the image block to be filtered; The image filtering method according to claim 9.

11. The step of determining a reference image block of the image block to be filtered according to a method of determining a reference image block of the input image block includes: determining a time domain reference image block of the image block to be filtered according to a time domain reference image block determining manner of the input image block; The image filtering method according to claim 8.

12. the time domain reference image block of the input image block includes an image block in a reference image of the input image at a position corresponding to the input image block; The step of determining a time domain reference image block of the image block to be filtered according to a time domain reference image block determination manner of the input image block includes: determining a reference image for the image to be filtered; determining an image block in a reference image of the image to be filtered that is located at a position corresponding to the image block to be filtered as a time-domain reference image block of the image block to be filtered; The image filtering method according to claim 11.

13. the training images include an input image and a target image corresponding to the input image, the neural network-based filter is obtained by training using an input image block as input data and a target image block as a target, the input image block is obtained by performing image segmentation on the input image using a block segmentation scheme of the training images, and the target image block is obtained by performing image segmentation on the target image using the block segmentation scheme of the training images. The image filtering method according to claim 1 .

14. The step of acquiring an image to be filtered includes: reconstructing an original image to obtain a reconstructed image of the original image; determining the reconstructed image as the image to be filtered; The image filtering method according to claim 1 .

15. 1. An image filtering device, comprising: an acquisition unit configured to acquire a to-be-filtered image; A segmentation unit configured to determine a neural network-based filter, and segment the image to be filtered according to a block segmentation scheme corresponding to the neural network-based filter to obtain N image blocks to be filtered, where N is a positive integer, and the block segmentation scheme is a block segmentation scheme of a training image used by the neural network-based filter in a training process; a filtering unit configured to filter each of the N image blocks to be filtered using the neural network based filter to obtain a filtered image; The image filtering device.

16. The block division method of the training image includes determining M coding tree units in the training image as one training image block, where M is a positive integer; The splitting unit is configured to determine the N image blocks to be filtered from the image to be filtered in a manner that M coding tree units are one image block to be filtered.

16. An image filtering device according to claim 15.

17. The block division method of the training image includes determining P incomplete coding tree units in the training image as one training image block, where P is a positive integer; The splitting unit is configured to determine the N image blocks to be filtered from the image to be filtered in a manner that P incomplete coding tree units are one image block to be filtered.

16. An image filtering device according to claim 15.

18. Electronic device arranged to carry out the image filtering method according to any one of claims 1 to 14.

19. A computer program product for causing a computer to carry out the image filtering method according to any one of claims 1 to 14.

Citation Information

Patent Citations

  • Image restoration filter and learning apparatus

    JP2019087778A

  • Encoding program, decoding program, encoding device, decoding device, encoding method, and decoding method

    JP2020198463A

  • Signal processing device and signal processing method, system, learning method, and program

    JP2021090135A

  • Convolutional-neutral-network based filter for video coding

    US20210329286A1

  • Image filter device, image decoding device, and image coding device

    WO2019031410A1