Image filtering method, device, equipment and program

Abstract

The present application provides an image filtering method, device, apparatus and program applicable to various scenes such as cloud technology, artificial intelligence, smart transportation, driving assistance, video, etc. The method includes: determining encoding information of a filtering target image, the encoding information including classification information, the classification information indicating at least one of an encoding image type and an encoding mode of the filtering target image; and inputting the filtering target image and the classification information into a filtering model to filter the filtering target image according to the classification information to obtain a filtered image.

Description

[Technical Field]

[0001] (CROSS-REFERENCE TO RELATED APPLICATIONS) This application claims priority from a Chinese patent application bearing application number 202210126411.X, filed with the China Patent Office on February 10, 2022, the entire contents of which are incorporated herein by reference.

[0002] The embodiments of the present application relate to the field of image processing technology, and more particularly to an image filtering method, device, apparatus, and program. [Background technology]

[0003] Loop filters in the related art include a deblocking filter (DBF), a sample adaptive offset (SAO), and an adaptive loop filter (ALF), which mainly filter a reconstructed image to reduce blocking effects, ringing effects, etc., thereby improving the quality of the reconstructed image. Ideally, the filter restores the reconstructed image to the original image. Many of the filtering coefficients of the filters in the related art are manually designed, leaving a large room for optimization. In view of the excellent performance of deep learning tools in image processing, deep learning-based loop filters have been applied to loop filter modules. However, deep learning-based loop filters in the related art still have deficiencies in performance optimization, and the performance of the filter needs to be further improved. Summary of the Invention

[0004] The embodiments of the present application provide an image filtering method, apparatus, device, and program that improves the image filtering effect while reducing the storage cost of model parameters.

[0005] An embodiment of the present application provides an image filtering method, the method comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the classification information to obtain a filtered image.

[0006] An embodiment of the present application provides an image filtering device, the device comprising: a determining unit configured to determine coding information of a target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; a filtering unit configured to input the target image and the classification information into a same deep learning-based filtering model, and cause the filtering model to filter the target image based on the classification information to obtain a filtered image.

[0007] An embodiment of the present application provides an electronic device, the electronic device comprising a processor and a memory, the memory being configured to store a computer program, and the processor executing the above-mentioned method by calling and executing the computer program stored in the memory.

[0008] An embodiment of the present application provides a computer-readable storage medium, the computer-readable storage medium being configured to store a computer program, the computer program causing an electronic device to perform the above-mentioned method.

[0009] An embodiment of the present application provides a chip, the chip including a processor, which calls up and executes a computer program from a memory, thereby causing an electronic device equipped with the chip to perform the above-described method.

[0010] An embodiment of the present application provides a computer program product, which includes computer program instructions that, when executed on an electronic device, cause the electronic device to perform the above-described method.

[0011] An embodiment of the present application provides a computer program, which, when executed on an electronic device, causes the electronic device to perform the above method.

[0012] The filtering model of the embodiment of the present application can realize filtering of the filtering target image in intraframe coding mode and can also realize filtering of the filtering target image in interframe coding mode. This corresponds to a general-purpose filtering model for the filtering target image in different modes, and compared with constructing corresponding filtering models for different modes, the storage space occupation rate of the model parameters is significantly reduced and the storage cost of the model parameters is low. In addition, by combining classification information with the filtering model and performing differential filtering on the filtering target image, the filtering target image in different modes can be applied, and the filtering effect of the filtering target image can be improved. [Brief explanation of the drawings]

[0013] [Figure 1] 1 is an exemplary block diagram of a video codec system according to an embodiment of the present application; [Figure 2] 1 is an exemplary block diagram of a video encoder according to an embodiment of the present application; [Figure 3] FIG. 2 is an exemplary block diagram of a video decoder according to an embodiment of the present application; [Figure 4] 1 is a flowchart of an image filtering method according to an embodiment of the present application; [Figure 5A] 1 is a schematic diagram of an application scene according to an embodiment of the present application; [Figure 5B] 1 is a schematic diagram of an application scene according to an embodiment of the present application; [Figure 5C] 1 is a schematic diagram of an application scene according to an embodiment of the present application; [Figure 6A] FIG. 1 is a schematic diagram of image division. [Figure 6B] FIG. 1 is a schematic diagram of image division. [Figure 6C] FIG. 1 is a schematic diagram of image division. [Figure 7A] FIG. 1 is a schematic diagram of a filtering scheme according to an embodiment of the present application; [Figure 7B] FIG. 1 is a schematic diagram of a filtering scheme according to an embodiment of the present application; [Figure 7C] FIG. 1 is a schematic diagram of a filtering scheme according to an embodiment of the present application; [Figure 8A] FIG. 2 is a schematic diagram of a residual block according to an embodiment of the present application; [Figure 8B] FIG. 2 is a schematic diagram of a residual block according to an embodiment of the present application; [Figure 8C] FIG. 2 is a schematic diagram of a residual block according to an embodiment of the present application; [Figure 8D] FIG. 2 is a schematic diagram of a residual block according to an embodiment of the present application; [Figure 9] FIG. 1 is a schematic diagram of model training according to an embodiment of the present application. [Figure 10] 1 is a schematic diagram illustrating a configuration of an image filtering device according to an embodiment of the present application; [Figure 11] FIG. 1 is an exemplary block diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION OF THE INVENTION

[0014] In order to more clearly explain the technical solutions of the embodiments of the present application, the following briefly introduces the drawings used in the description of the embodiments. Obviously, the above drawings are only some embodiments of the present application, and those skilled in the art can also obtain other related drawings based on these drawings without any creative efforts.

[0015] In the following, the technical solutions of the embodiments of the present application will be described clearly and completely with reference to the drawings in the embodiments of the present application, but obviously, the described embodiments are only a part of the embodiments of the present application, not all of the embodiments of the present application, and all other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative work are included in the protection scope of the present application.

[0016] Note that terms such as "first," "second," and the like in the description of the embodiments of the present application, the claims, and the above drawings do not limit a particular order or sequence, but rather distinguish between similar objects. Note that data used in this manner may be interchanged where appropriate, so that the embodiments of the present application described herein may be performed in an order other than that shown or described herein. Furthermore, the terms "comprise" and "have," and any variations thereof, are intended to be non-exclusive inclusive. For example, a process, method, system, product, or server incorporating a series of steps or units need not be limited to those steps or units explicitly recited, but may include other steps or units not explicitly recited or inherent to the process, method, product, or apparatus.

[0017] In the following, "and / or" means at least one of the two; for example, A and / or B means at least one of A and B.

[0018] To facilitate understanding of the embodiments of the present application, a brief description of relevant concepts related to the embodiments of the present application will be provided below.

[0019] Artificial intelligence (AI) is the theory, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and augment human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology in computer science that seeks to understand the nature of intelligence and create new intelligent machines that can respond in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to perceive, reason, and make decisions.

[0020] Artificial intelligence technology is a comprehensive field encompassing a wide range of fields, including both hardware and software technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. Artificial intelligence software technology mainly includes several major directions, such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0021] Machine learning (ML) is a multidisciplinary field that encompasses many disciplines, including probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. ML focuses on the study of how computers can simulate or realize human learning behavior, acquire new knowledge and skills, and restructure existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and a fundamental means of endowing computers with intelligence, and its applications span various fields of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, trust networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration.

[0022] The embodiments of the present application may be applied to the fields of image codecs, video codecs, hardware video codecs, dedicated circuit video codecs, real-time video codecs, etc. For example, the solutions in the embodiments of the present application may be combined with audio video coding standards (AVS), such as the H.264 / Audio Video Coding (AVC) standard, the H.265 / High Efficiency Video Coding (HEVC) standard, and the H.266 / Versatile Video Coding (VVC) standard. Alternatively, the technical solutions in the embodiments of the present application may be combined with other proprietary or industry standards for operation. The standards include ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, and ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. Note that the technology of the embodiments of the present application is not limited to any particular codec standard or technology.

[0023] For ease of understanding, a video codec system according to an embodiment of the present invention will be described first with reference to FIG.

[0024] FIG. 1 is an exemplary block diagram of a video codec system according to an embodiment of the present application. Note that FIG. 1 is merely an example, and video codec systems according to the embodiment of the present application include, but are not limited to, those shown in FIG. 1. As shown in FIG. 1, the video codec system 100 includes an encoding device 110 and a decoding device 120. The encoding device encodes (which can also be understood as compressing) video data to generate a codestream and transmits the codestream to a decoding device. The decoding device decodes the codestream generated by the encoding device to obtain decoded video data.

[0025] The encoding device 110 in the embodiments of the present application can be understood as a device having a video encoding function, and the decoding device 120 can be understood as a device having a video decoding function, i.e., the encoding device 110 and the decoding device 120 in the embodiments of the present application include a wider range of devices such as smartphones, desktop computers, mobile computing devices, notebook computers (such as laptops), tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, game consoles, and in-vehicle computers.

[0026] In some embodiments, encoding device 110 may transmit encoded video data (e.g., a codestream) to decoding device 120 over channel 130. Channel 130 may include one or more media and / or devices that may transmit encoded video data from encoding device 110 to decoding device 120.

[0027] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly in real time to decoding device 120. In this example, encoding device 110 may modulate the encoded video data in accordance with a communication standard and transmit the modulated video data to decoding device 120. Here, the communication media may include wireless communication media, such as a radio frequency spectrum, and in some embodiments, the communication media may also include wired communication media, such as one or more physical transmission lines.

[0028] In another example, channel 130 includes a computer-readable storage medium that can store the video data encoded by encoding device 110. The computer-readable storage medium can include a number of locally accessible data storage media, such as optical disks, DVDs, flash memory, etc. In this example, decoding device 120 can obtain the encoded video data from the computer-readable storage medium.

[0029] In another example, channel 130 may include a storage server capable of storing video data encoded by encoding device 110. In this example, decoding device 120 may download the stored encoded video data from the storage server. In some embodiments, the storage server may store the encoded video data and transmit the encoded video data to decoding device 120, such as a web server (e.g., for a website), a file transfer protocol (FTP) server, or the like.

[0030] In some embodiments, encoding device 110 includes a video encoder 112 and an output interface 113, where output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0031] In some embodiments, encoding device 110 may further include a video source 111 in addition to video encoder 112 and output interface 113 .

[0032] The video source 111 may include at least one of a video collection device (such as a video camera), a video archive, a video input interface, and a computer graphics system, where the video input interface is configured to receive video data from a video content provider and the computer graphics system is configured to generate the video data.

[0033] The video encoder 112 encodes video data from the video source 111 to generate a codestream. The video data may include one or more pictures or a sequence of pictures. The codestream contains coding information for a picture or sequence of pictures in the form of a bitstream. The coding information may include coded image data and associated data. The associated data may include sequence parameter sets (SPSs), picture parameter sets (PPSs), and other syntax structures. An SPS may contain parameters that apply to one or more sequences. A PPS may contain parameters that apply to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the codestream.

[0034] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may be stored on a storage medium or a storage server for later retrieval by the decoding device 120.

[0035] In some embodiments, decoding device 120 includes an input interface 121 and a video decoder 122 .

[0036] In some embodiments, decoding device 120 may include a display device 123 in addition to input interface 121 and video decoder 122 .

[0037] Here, the input interface 121 includes a receiver and / or a modem, and can receive encoded video data via a channel 130.

[0038] The video decoder 122 is configured to decode the encoded video data to obtain decoded video data, and transmit the decoded video data to a display device 123 .

[0039] Display device 123 displays the decoded video data and may be integrated with decoding device 120 or may be external to decoding device 120. Display device 123 may include various display devices such as a liquid crystal display (LCD), a plasma display, an organic light emitting diode (OLED) display, or other types of display devices.

[0040] Furthermore, Figure 1 is only an example, and the technical solution of the embodiments of the present application is not limited to Figure 1, and the technology of the embodiments of the present application can also be applied to one-sided video encoding and one-sided video decoding.

[0041] In the following, a video coding framework according to an embodiment of the present application is described.

[0042] 2 is an exemplary block diagram of a video encoder according to an embodiment of the present application. The video encoder 200 may be configured to perform lossy compression of images or lossless compression of images. The lossless compression may be visually lossless compression or mathematically lossless compression.

[0043] The video encoder 200 is applicable to image data in luminance and chrominance (YCbCr, YUV) formats.

[0044] For example, the video encoder 200 reads video data and, for each frame image in the video data, divides the image into multiple coding tree units (CTUs). In some examples, a CTU may be referred to as a "tree block," "largest coding unit" (LCU), or "coding tree block" (CTB). Each CTU may be associated with an equally sized block of pixels in the image. Each pixel may correspond to one luminance (luma) sample and two chrominance (chroma) samples. Thus, each CTU may be associated with one luminance sample block and two chrominance sample blocks. The size of a CTU may be, for example, 128x128, 64x64, 32x32, etc. A CTU may be further divided into multiple coding units (CUs) for encoding, and a CU may be a rectangular or square block. The CU may be further divided into a prediction unit (PU) and a transform unit (TU), which allows for separation of coding, prediction, and transformation, and improves processing flexibility. In one example, the CTU is divided into CUs in a quadtree manner, and the CU is divided into TUs and PUs in a quadtree manner.

[0045] Video encoders and video decoders can support various PU sizes. Assuming that the size of a specific CU is 2Nx2N, the video encoder and video decoder can support PU sizes of 2Nx2N or NxN for intra-frame prediction, and can support symmetric PUs of 2Nx2N, 2NxN, Nx2N, NxN, or similar sizes for inter-frame prediction. The video encoder and video decoder can further support asymmetric PUs of 2NxnU, 2NxnD, nLx2N, and nRx2N for inter-frame prediction.

[0046] 2, the video encoder 200 may include a prediction unit 210, a residual unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image cache 270, and an entropy coding unit 280. Note that the video encoder 200 may include more, fewer, or different functional components.

[0047] In some embodiments, in this application, a current block may be referred to as a current coding unit (CU) or a current prediction unit (PU), etc. A prediction block may also be referred to as a predicted image block or an image prediction block, and a reconstructed image block may also be referred to as a reconstruction block or an image reconstruction image block.

[0048] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is a strong correlation between adjacent pixels within a frame of video, the use of intra-frame prediction in video codec technology can eliminate spatial redundancy between adjacent pixels. Because there is a strong correlation between adjacent frames in video, the use of inter-frame prediction methods in video codec technology can eliminate temporal redundancy between adjacent frames, thereby improving coding efficiency.

[0049] The inter-frame prediction unit 211 is configured to perform inter-frame prediction, which can refer to image information of different frames. Inter-frame prediction involves finding a reference block from a reference frame using motion information, generating a predicted block based on the reference block, and eliminating temporal redundancy. The frames used for inter-frame prediction may be P frames and / or B frames. A P frame refers to a forward predicted frame, and a B frame refers to a bidirectionally predicted frame. The motion information includes a reference frame list in which the reference frame is located, a reference frame index, and a motion vector. The motion vector may be full-pixel or sub-pixel. If the motion vector is sub-pixel, interpolation filtering must be used on the reference frame to create the required sub-pixel block. Here, the full-pixel or sub-pixel block in the reference frame found based on the motion vector is referred to as the reference block. Some techniques directly use the reference block as the predicted block, while other techniques generate a predicted block by reprocessing based on the reference block. Generating a predicted block by reprocessing based on the reference block can be understood as using the reference block as the predicted block and then processing based on the predicted block to generate a new predicted block.

[0050] The intra-frame prediction unit 212 predicts pixel information in the current coding image block by only referring to information of the same frame image, thereby eliminating spatial redundancy. The frame used for intra-frame prediction may be an I-frame.

[0051] The intra-frame prediction modes used in HEVC are Planar, DC, and 33 angle modes, for a total of 35 prediction modes. The intra-frame modes used in VVC are Planar, DC, and 65 angle modes, for a total of 67 prediction modes. The intra-frame modes used in AVS3 are DC, Planar, Bilinear, and 63 angle modes, for a total of 66 prediction modes.

[0052] In some embodiments, the intra-frame prediction unit 212 may be implemented employing an intra-frame block duplication technique and an intra-frame string duplication technique.

[0053] Residual unit 220 may generate a residual block for a CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, residual unit 220 may generate a residual block for a CU such that each sample in the residual block has a value equal to the difference between a sample in the pixel block of the CU and a corresponding sample in the prediction block of the PU of the CU.

[0054] The transform / quantization unit 230 can quantize the transform coefficients. The residual video signal is transformed into a transform domain through a transform operation such as DFT or DCT, resulting in a transform signal called a transform coefficient. Signals in the transform domain lose certain information through further lossy quantization, making the quantized signal advantageous for compressed representation. Some video coding standards allow multiple transform methods to be selected, so the encoding side must also select one transform for the current coding CU and notify the decoding side. The quantization resolution is typically determined by the quantization parameter (QP). A larger QP value means that a wider range of coefficients are quantized to the same output, typically resulting in greater distortion and a lower bitrate. Conversely, a smaller QP value means that a narrower range of coefficients are quantized to the same output, typically resulting in less distortion and a higher bitrate.

[0055] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct residual blocks from the quantized transform coefficients.

[0056] Reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sample blocks of each TU of the CU in this manner, video encoder 200 may reconstruct pixel blocks of the CU.

[0057] The loop filtering unit 260 can perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with a CU. Compared to an original image, a reconstructed image is affected by quantization, which causes some information to differ from the original image, resulting in distortion. Filtering operations can be performed on the reconstructed image using filters such as DBF, SAO, or ALF to effectively reduce the degree of quantization-induced distortion. These filtered reconstructed images are used as references for subsequent coded images to predict future signals, so the above filtering operation is also referred to as loop filtering, i.e., a filtering operation within the coding loop.

[0058] The decoded image cache 270 can store the reconstructed pixel blocks. The inter-frame prediction unit 211 can perform inter-frame prediction on PUs of other images using reference images including the reconstructed pixel blocks. Furthermore, the intra-frame prediction unit 212 can perform intra-frame prediction on other PUs in the same image as the CU using the reconstructed pixel blocks in the decoded image cache 270.

[0059] The entropy coding unit 280 can receive the quantized transform coefficients from the transform / quantization unit 230. The quantized transform domain signal is statistically compression-coded according to the frequency of occurrence of each value, and finally, a binary (0 or 1) compressed code stream is output. Other information, such as the selected mode and motion vector data, can also be generated through the coding, and entropy coding is also required to reduce the bit rate. Statistical coding is a lossless coding method, which can effectively reduce the bit rate required to represent the same signal. Common statistical coding methods are variable length coding (VLC) or content adaptive binary arithmetic coding (CABAC).

[0060] FIG. 3 is an exemplary block diagram of a video decoder according to an embodiment of the present application.

[0061] 3, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transform unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image cache 360. Note that the video decoder 300 may include more, fewer, or different functional components.

[0062] The video decoder 300 may receive a codestream. The entropy decoding unit 310 may parse the codestream to extract syntax elements from the codestream. As part of parsing the codestream, the entropy decoding unit 310 may parse entropy-encoded syntax elements in the codestream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filtering unit 350 may decode the video data based on the syntax elements extracted from the codestream to generate decoded video data.

[0063] In some embodiments, the prediction unit 320 includes an inter-frame prediction unit 321 and an intra-frame prediction unit 322 .

[0064] The inter-frame prediction unit 321 may perform intra-frame prediction to generate a predictive block of the PU. The inter-frame prediction unit 321 may use an intra-frame prediction mode to generate a predictive block of the PU based on pixel blocks of spatially neighboring PUs. The inter-frame prediction unit 321 may also determine the intra-frame prediction mode of the PU based on one or more syntax elements parsed from the codestream.

[0065] The intra-frame prediction unit 322 may construct a first reference image list (List 0) and a second reference image list (List 1) based on syntax elements parsed from the codestream. Furthermore, if the PU uses inter-frame predictive coding, the entropy decoding unit 310 may analyze motion information of the PU. The intra-frame prediction unit 322 may determine one or more reference blocks for the PU based on the motion information of the PU. The intra-frame prediction unit 322 may generate a predictive block for the PU based on the one or more reference blocks of the PU.

[0066] The inverse quantization / transform unit 330 may inverse quantize (i.e., dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may determine the degree of quantization using a QP value associated with the CU of the TU.

[0067] After dequantizing the transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients to generate a residual block associated with the TU.

[0068] The reconstruction unit 340 may use the residual block associated with the TU of the CU and the prediction block of the PU of the CU to reconstruct the pixel block of the CU. For example, the reconstruction unit 340 may add the samples of the residual block to the corresponding samples of the prediction block to reconstruct the pixel block of the CU to obtain a reconstructed image block.

[0069] The loop filtering unit 350 may perform a deblocking filtering operation to reduce the blocking effect of pixel blocks associated with a CU.

[0070] The video decoder 300 can store the reconstructed image of the CU in the decoded image cache 360. The video decoder 300 can use the reconstructed image in the decoded image cache 360 ​​as a reference image for subsequent prediction, or transmit the reconstructed image to a display device for presentation.

[0071] The basic process of a video codec is as follows: At the encoding side, a frame of image is divided into blocks, and for a current block, a prediction unit 210 generates a predicted block of the current block using intraframe prediction or interframe prediction. A residual unit 220 calculates a residual block, i.e., a difference between the predicted block and the original block of the current block, based on the predicted block and the original block of the current block. The residual block may also be referred to as residual information. The residual block may be subjected to processes such as transformation and quantization by a transform / quantization unit 230 to remove information insensitive to the human eye and thereby remove visual redundancy. In some embodiments, the residual block before transformation and quantization by the transform / quantization unit 230 may be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 may be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 may receive the quantized transform coefficients output from the transform / quantization unit 230, entropy code the quantized transform coefficients, and output a codestream. For example, the entropy coding unit 280 may remove character redundancies based on a target context model and probability information of the binary codestream.

[0072] On the decoding side, the entropy decoding unit 310 analyzes the codestream to obtain prediction information, a quantization coefficient matrix, etc., for the current block. The prediction unit 320 generates a prediction block for the current block using intraframe or interframe prediction based on the prediction information. The inverse quantization / transform unit 330 uses the quantization coefficient matrix obtained from the codestream to perform inverse quantization and inverse transformation on the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed block forms a reconstructed image, and the loop filtering unit 350 loop filters the reconstructed image based on an image or a block to obtain a decoded image. The encoding side also requires similar operations to the decoding side to obtain a decoded image. The decoded image is also called a reconstructed image, and the reconstructed image may be a subsequent frame and is used as a reference frame for interframe prediction.

[0073] Note that the block division information determined on the encoding side, mode information such as prediction, transform, quantization, entropy coding, loop filtering, or parameter information, etc., are carried in the code stream as needed. The decoding side analyzes the code stream and analyzes it based on existing information to determine the same block division information, mode information such as prediction, transform, quantization, entropy coding, loop filtering, or parameter information as the encoding side, thereby ensuring that the decoded image obtained on the encoding side is the same as the decoded image obtained on the decoding side.

[0074] The above is a basic process of a video codec in a block-based hybrid coding framework, and as technology develops, some modules or steps of the framework or process may be optimized. This application applies to, but is not limited to, the basic process of a video codec in a block-based hybrid coding framework.

[0075] In some embodiments, embodiments of the present application may be applied to various image processing scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, driver assistance, and the like.

[0076] The following describes in detail the technical solutions of the embodiments of the present application through several embodiments, which can be combined with each other, and the same or similar concepts or processes may not be described repeatedly in some embodiments.

[0077] FIG. 4 is a flowchart of an image filtering method according to an embodiment of the present application. As shown in FIG. 4, the method includes the following steps:

[0078] In step S401, coding information of the image to be filtered is determined, where the coding information includes classification information of the image to be filtered, and the classification information indicates at least one of a coding image type and a coding mode of the image to be filtered.

[0079] In some embodiments, the embodiments of the present application may be applied to both the encoding side and the decoding side, i.e., the execution entity of the embodiments of the present application may be the encoder in Figures 1 and 2 or the decoder in Figures 1 and 3.

[0080] In some embodiments, the embodiments of the present application may also be applied to other display devices, for example, a decoder sends the decoded reconstructed image to a display device, and the display device filters the reconstructed image using the image filtering method of the embodiments of the present application and displays the filtered image.

[0081] That is, the implementation subject of the embodiments of the present application is a device having an image filtering function, such as an image filtering device. The image filtering device may be an electronic device or a part of an electronic device, such as a processor in an electronic device. The electronic device may be an encoder, decoder, codec, display device, etc.

[0082] For convenience of explanation, the following describes the method of the embodiment of the present application using an example in which the execution entity is an electronic device.

[0083] The filter according to the embodiments of the present application is a neural network filter, i.e., a filtering model, which in some embodiments is also called a neural network in-loop filter (NNLF).

[0084] The filtering model of the embodiment of the present application can not only realize filtering of intraframe coded images, but also realize filtering of interframe coded images, that is, the filtering model of the embodiment of the present application is a general-purpose filtering model, which can reduce the storage cost of model parameters.

[0085] The application scenarios of the embodiments of the present application include, but are not limited to, the following scenarios:

[0086] Scene 1: As shown in Fig. 5A, the above-mentioned filtering target image is an image reconstructed by the above-mentioned reconstruction unit. That is, the loop filtering unit in the embodiment of the present application is the filtering model, and the reconstruction unit 250 in Fig. 2 inputs the reconstructed image to the filtering model in the loop filtering unit 260 to perform filtering, obtains a filtered image, and stores the filtered image in the decoded image cache. Alternatively, the reconstruction unit 340 in Fig. 3 inputs the reconstructed image to the filtering model in the loop filtering unit 350 to perform filtering, obtains a filtered image, and stores the filtered image in the decoded image cache or displays it.

[0087] Scene 2: As shown in FIG. 5B, the above-mentioned image to be filtered is an image filtered by at least one of DBF, SAO, and ALF. That is, the loop filtering unit in the embodiment of the present application includes not only a conventional filter such as at least one of DBF, SAO, and ALF, but also a filtering model. The reconstruction unit 250 in FIG. 2 inputs the reconstructed image to the conventional filter in the loop filtering unit 260 for filtering, and then inputs the filtered image to the filtering model for further filtering to obtain a filtered image, and stores the filtered image in the decoded image cache. Alternatively, the reconstruction unit 340 in FIG. 3 inputs the reconstructed image to the conventional filter in the loop filtering unit 350 for filtering, and then inputs the filtered image to the filtering model for further filtering to obtain a filtered image, and stores the filtered image in the decoded image cache or displays it.

[0088] 5B shows only one example of combining a conventional filter with the filtering model of the embodiment of the present application. In some scenes, the combination of the conventional filter with the filtering model of the embodiment of the present application may include, but is not limited to, first filtering using the filtering model and then filtering using the conventional filter, or first filtering using the conventional filter, then filtering using the filtering model, and then filtering using the conventional filter, or other combinations, and the present application does not limit these.

[0089] Scene 3: As shown in Figure 5C, the image to be filtered is the image to be displayed. For example, the reconstruction unit 340 in Figure 3 inputs the reconstructed image to the loop filtering unit 350 for filtering, and stores the filtered image in the decoded image cache. The image in the decoded image cache is predicted as a reference image for subsequent figures. For example, the prediction unit 320 reads a reference image from the decoded image cache to perform prediction. The image in the decoded image cache is also used for display, but before being used for display, it is first filtered using the filtering model of the embodiment of the present application, and then the filtered image is displayed.

[0090] As shown above in FIGS. 5A and 5B, in some embodiments, the filtered image of the present embodiment is used as a reference image for subsequent images.

[0091] As shown above in FIG. 5C, in some embodiments, the filtered image of the present embodiment is used for display and is not used as a reference image for subsequent images.

[0092] The filtering model in the embodiment of the present application is a general-purpose filtering model and may be used for filtering intra-frame coded images or inter-frame coded images. However, due to the difference between intra-frame coding and inter-frame coding, in order to improve filtering accuracy, in the embodiment of the present application, a filtering target image is input to the filtering model. The input indicates classification information such as the coding image type and coding mode of the filtering target image.

[0093] In the embodiment of the present application, the encoding information of the image to be filtered is not particularly limited and may be encoding type information involved in the encoding process of the image to be filtered.

[0094] In some embodiments, the coding information of the to-be-filtered image comprises classification information, said classification information indicating at least one of a coding image type, a coding mode of the to-be-filtered image.

[0095] Encoded image types according to embodiments of the present application include full intra-frame coded images and inter-frame codeable images.

[0096] Here, a full-frame intra-coded image refers to the fact that the coding type of the image is entirely intra-frame coding, meaning that all reference information for predicting a full-frame intra-coded image comes from the spatial domain information of the image.

[0097] An inter-codable image refers to the fact that the coding type of the image may be inter-coding, which means that the prediction process of the inter-codable image can refer to the time-domain reference information of other reference frames.

[0098] In some embodiments, the coding mode classification includes all coding modes, including, for example, two types: inter-frame coding modes and intra-frame coding modes.

[0099] On the encoding side, the overall coding mode can be determined based on the prediction modes used by all prediction units corresponding to the image to be filtered. For example, in the encoding process of the image to be filtered, if all prediction units corresponding to the image to be filtered use an intra-frame prediction mode for prediction, the overall coding mode of the image to be filtered is determined to be an intra-frame coding mode, and if all prediction units corresponding to the image to be filtered use an inter-frame prediction mode for prediction, the overall coding mode of the image to be filtered is determined to be an inter-frame coding mode.

[0100] On the decoding side, the method for obtaining the overall coding mode of the image to be filtered includes at least the following two methods:

[0101] Method 1: The codestream includes the overall coding mode of the image to be filtered, and thus the decoding side decodes the codestream to obtain the overall coding mode of the image to be filtered.

[0102] Method 2: The decoding side determines the overall coding mode of the target image to be filtered based on the prediction modes used by all prediction units corresponding to the target image to be filtered. For example, in the decoding process of the target image to be filtered, if all prediction units corresponding to the target image to be filtered use an intra-frame prediction mode for prediction, the overall coding mode of the target image to be filtered is determined to be an intra-frame coding mode, and if all prediction units corresponding to the target image to be filtered use an inter-frame prediction mode for prediction, the overall coding mode of the target image to be filtered is determined to be an inter-frame coding mode.

[0103] In some embodiments, the classification of coding modes can also be embodied in coding sub-modes, which can be understood as specific coding methods for the image to be filtered, such as specific sub-modes in an intra-frame coding mode or specific sub-modes in an inter-frame coding model.

[0104] Common intraframe coding techniques include normal intraframe coding mode, MIP mode (MIP: Matrix weighted Intra Prediction), intraframe block copy (IBC: Intra Block Copy), intraframe string prediction (ISP: Intra String Prediction), palette technique (PLT: Palette), etc.

[0105] Common inter-frame coding techniques include a normal inter-frame coding mode, a motion merge mode (Merge mode), a motion merge and residual skip mode (Skip mode), a combined inter-frame and intra-frame prediction mode (CIIP: Combined Inter and Intra Prediction), etc.

[0106] On the encoding side, the coding sub-mode can be determined based on the prediction modes used by all prediction units corresponding to the image to be filtered. For example, in the encoding process of the image to be filtered, if all prediction units corresponding to the image to be filtered use a normal intra-frame coding mode for prediction, the coding sub-mode of the image to be filtered is determined to be a normal intra-frame coding mode, and if all prediction units corresponding to the image to be filtered use a merge mode for prediction, the coding sub-mode of the image to be filtered is determined to be a merge mode.

[0107] On the decoding side, the manner of obtaining the coding sub-mode of the image to be filtered includes at least the following two manners:

[0108] Method 1: The codestream includes the coding submode of the image to be filtered, and thus the decoding side decodes the codestream to obtain the coding submode of the image to be filtered.

[0109] Method 2: The decoding side determines the coding sub-mode of the image to be filtered based on the prediction modes used by all prediction units corresponding to the image to be filtered. For example, in the decoding process of the image to be filtered, if all prediction units corresponding to the image to be filtered use the normal intra-frame coding mode for prediction, the coding sub-mode of the image to be filtered is determined to be the normal intra-frame coding mode, and if all prediction units corresponding to the image to be filtered use the merge mode for prediction, the coding sub-mode of the image to be filtered is determined to be the merge mode.

[0110] In an embodiment of the present application, during filtering, a filtering target image and classification information indicating at least one of the encoding image type and encoding mode of the filtering target image are input to a filtering model to obtain a filtered image of the filtering target image.

[0111] In some embodiments, the classification information of the image to be filtered may be a matrix including multiple elements.

[0112] In some embodiments, the classification information of the image to be filtered is one element of data.

[0113] The process of determining the coding information of the filtering target image in the embodiment of the present application will be described in detail below with reference to a specific example.

[0114] Case 1: If the classification information indicates the coding image type of the image to be filtered, in the above step S401, the process of determining the coding information of the image to be filtered includes the following steps:

[0115] In step S401-A1, obtain coded image types corresponding to a plurality of image regions in the image to be filtered, and determine that the value of the classification information is a first value if the plurality of image regions all belong to full-frame intra-coded images.

[0116] In step S401-A2, if the plurality of image regions all belong to inter-codable images, it is determined that the value of the classification information is a second value.

[0117] In step S401-A3, if the image to be filtered crosses a slice boundary or a tile boundary, the value of the classification information is determined to be a third value, or the classification information is determined based on the respective proportions in the image to be filtered of image areas belonging to full-frame intra-coded images and image areas belonging to inter-frame codeable images among the plurality of image areas.

[0118] Specifically, the coded images in the embodiments of the present application are divided into full intra-frame coded images and inter-frame codable images, where the images include frames, slices, and tiles, as shown in Figures 6A to 6C. If the classification information represents a coded image type, the classification information can be indicated at one of a slice level, a tile level, or a frame header level.

[0119] In practice, when using a filtering model, rather than directly inputting and filtering an entire frame, it is common to divide an image into sub-images and then sequentially input the sub-images into the filtering model for filtering. Since the sub-images vary in size, the image to be filtered input to the filtering model may cross a slice / tile boundary. In the embodiments of the present application, the image to be filtered can be understood as one sub-image.

[0120] Based on this, in the embodiment of the present application, the classification information of the image to be filtered is determined based on the coded image type of the image to be filtered and whether the image to be filtered crosses a slice boundary or a tile boundary.

[0121] The multiple image regions in the image to be filtered may be all image regions in the image to be filtered, or may be a portion of the image regions in the image to be filtered, for example, 80% of the image regions in the image to be filtered.

[0122] Specifically, if all image areas within the multiple image areas of the image to be filtered belong to full intra-frame coded images, the value of the classification information is determined to be a first value, and if all image areas within the multiple image areas of the image to be filtered belong to inter-frame coded images, the value of the classification information is determined to be a second value.

[0123] The embodiments of the present application do not limit the specific values ​​of the first value and the second value.

[0124] In some embodiments, the first value is zero.

[0125] In some embodiments, the second value is one.

[0126] In one example, the classification information in the embodiment of the present application is data of one element, for example, if all image regions in the plurality of image regions of the image to be filtered belong to full intra-frame coded images, the value of the classification information is determined to be 1, and if all image regions in the plurality of image regions of the image to be filtered belong to inter-frame coded images, the value of the classification information of the image to be filtered is determined to be 0.

[0127] In another example, the classification information in the embodiments of the present application is a matrix, and the matrix may be the same as or different from the size of the image to be filtered. For example, if all of the image regions of the current image to be filtered belong to full intra-frame coded images, the classification information is determined to be an all-zero matrix. If all of the image regions of the current image to be filtered belong to inter-frame coded images, the classification information is determined to be an all-one matrix.

[0128] When the image to be filtered crosses a slice boundary or a tile boundary, the classification information of the image to be filtered can be determined based on the following two methods.

[0129] Method 1: If the image to be filtered crosses a slice boundary or a tile boundary, the value of the classification information is determined to be the third value.

[0130] In the embodiment of the present application, the specific value of the third value is not limited as long as it is a value different from the first and second values.

[0131] In one example, when the classification information is data of one element, for example, when the image to be filtered crosses a slice boundary or a tile boundary, the classification information is determined to be 2.

[0132] In another example, when the above classification information is a single matrix, for example, when the image to be filtered crosses a slice boundary or a tile boundary, the classification information is determined to be an all-two matrix.

[0133] Method 2: When the image to be filtered crosses a slice boundary or a tile boundary, classification information is determined based on the respective proportions in the image to be filtered of image areas belonging to full-frame intra-coded images and image areas belonging to inter-frame codeable images among multiple image areas of the image to be filtered.

[0134] In the above method 2, embodiments for determining classification information based on the respective proportions in the filtering target image of image areas belonging to full-frame intra-coded images and image areas belonging to inter-frame codeable images among multiple image areas of the filtering target image include, but are not limited to, some examples below.

[0135] Example 1: If the proportion of image areas in the filtered image that belong to full-frame intra-coded images among multiple image areas of the filtered image is greater than or equal to a first threshold, the value of the classification information is determined to be a first value, and if the proportion of image areas in the filtered image that belong to full-frame intra-coded images among multiple image areas of the filtered image is less than the first threshold, the value of the classification information is determined to be a second value.

[0136] Example 2: If, among the multiple image regions of the image to be filtered, the proportion in the filtered image of image regions that belong to full-frame intra-coded images is greater than or equal to a first threshold, the value of the classification information is determined to be a first value; if, among the multiple image regions of the image to be filtered, the proportion in the filtered image of image regions that belong to full-frame intra-coded images is less than the first threshold and greater than or equal to a second threshold, the value of the classification information is determined to be a second value; if, among the multiple image regions of the image to be filtered, the proportion in the filtered image of image regions that belong to full-frame intra-coded images is less than the second threshold, the value of the classification information is determined to be a third value.

[0137] In the embodiment of the present application, the specific values ​​of the first threshold and the second threshold are not limited, and the first threshold is greater than the second threshold.

[0138] In some embodiments, the first threshold is greater than 50% and less than 100%.

[0139] In the above case 1, the classification information indicates the coded image type of the image to be filtered, and after the classification information is input into the filtering model, the filtering model can accurately filter the image to be filtered based on the coded image type of the image to be filtered.

[0140] Case 2: the coding mode includes an intraframe coding mode and an interframe coding mode, and the classification information indicates the coding mode of the image to be filtered, and the classification information includes at least one of first classification information and second classification information. In this embodiment, the classification information is indicated at a block level.

[0141] The first classification information indicates an encoding mode corresponding to one or more pixels in the image to be filtered.

[0142] The second classification information indicates an encoding mode corresponding to a predetermined size region in the image to be filtered.

[0143] For the first classification information and the second classification information, the process of determining the coding information of the image to be filtered in Case 2 includes subcases of Case 1 and Case 2. Here, Case 1 is the process of determining the first classification information, and Case 2 is the process of determining the second classification information.

[0144] Case 1: When the classification information includes first classification information, and the first classification information is a matrix consisting of multiple elements, each element corresponding to the coding mode of one or more pixels in the image to be filtered, the process of determining the coding information of the image to be filtered in the above step S401 includes the following steps:

[0145] In step S401-B, an encoding mode corresponding to one or more pixels in the to-be-filtered image is obtained, and first classification information is determined based on the encoding mode corresponding to the one or more pixels in the to-be-filtered image.

[0146] In the first case, the first classification information is determined based on the coding mode of one or more pixels in the image to be filtered. For example, if the coding mode of each pixel in the image to be filtered is an intra-frame coding mode, the value of each matrix element in the first classification information is determined to be a first value.

[0147] In some examples, the implementation of step S401-B above includes, but is not limited to, the following examples.

[0148] Example 1: When the coding mode corresponding to all pixels in the image to be filtered is the intra-frame coding mode, the value of each matrix element in the first classification information is determined to be a first value.

[0149] Example 2: When the coding mode corresponding to all pixels in the image to be filtered is the inter-frame coding mode, the value of each matrix element in the first classification information is determined to be the second value.

[0150] Example 3: If the coding mode corresponding to one or more pixels in a first image region in the image to be filtered is an intraframe coding mode and the coding mode corresponding to one or more pixels in a second image region is an interframe coding mode, it is determined that the values ​​of matrix elements corresponding to one or more pixels in the first image region in the first classification information are a first value and the values ​​of matrix elements corresponding to one or more pixels in the second image region are a second value.

[0151] The embodiments of the present application do not limit the specific values ​​of the first value and the second value.

[0152] In some embodiments, the first value is zero.

[0153] In some embodiments, the second value is one.

[0154] In one possible embodiment, the size of the first classification information matches the size of the image to be filtered, i.e., each matrix element in the first classification information corresponds one-to-one with each pixel in the image to be filtered. In this way, the value of the first classification information can be determined based on the encoding mode corresponding to each pixel in the image to be filtered. For example, if the encoding mode corresponding to each pixel in the image to be filtered is an intraframe encoding mode, the value of each matrix element in the first classification information is determined to be a first value (e.g., 0). If the encoding mode corresponding to each pixel in the image to be filtered is an interframe encoding mode, the value of each matrix element in the first classification information is determined to be a second value (e.g., all 1). If the encoding mode corresponding to the first portion of subpixels in the image to be filtered is an intraframe encoding mode and the encoding mode corresponding to the second portion of subpixels is an interframe encoding mode, the value of each matrix element corresponding to the first portion of subpixels in the first classification information is determined to be a first value (e.g., all 0), and the value of each matrix element corresponding to the second portion of subpixels is determined to be a second value.

[0155] In another possible embodiment, the size of the first classification information does not match the size of the image to be filtered. For example, the size of the first classification information is smaller than the size of the image to be filtered. For example, if the size of the image to be filtered is 64×64, the size of the first classification information is 64 / n×64 / m, where n and m are both positive integers. For example, the size of the first classification information is 32×32, and one matrix element in the first classification information corresponds to four pixels in the image to be filtered. In this way, the value of the corresponding matrix element in the first classification information can be determined based on the encoding mode corresponding to every four pixels in the image to be filtered.

[0156] For example, pixel 1, pixel 2, pixel 3, and pixel 4 in the image to be filtered correspond to matrix element 1 in the first classification information. If the coding modes corresponding to pixel 1, pixel 2, pixel 3, and pixel 4 are all intra-frame coding modes, the value of matrix element 1 in the first classification information is determined to be a first value. If the coding modes corresponding to pixel 1, pixel 2, pixel 3, and pixel 4 are all inter-frame coding modes, the value of matrix element 1 in the first classification information is determined to be a second value. If the coding mode corresponding to most of pixel 1, pixel 2, pixel 3, and pixel 4 is inter-frame coding mode, the value of matrix element 1 in the first classification information is determined to be a second value. If the coding mode corresponding to most of pixel 1, pixel 2, pixel 3, and pixel 4 is intra-frame coding mode, the value of matrix element 1 in the first classification information is determined to be a first value.

[0157] Case 2: If the classification information includes second classification information, the process of determining the coding information of the filtering target image in the above step S401 includes the following steps:

[0158] In step S401-C, second classification information is determined based on the proportions in the target image of image regions that correspond to different encoding modes in the target image.

[0159] In the second case, the second classification information can be determined based on the proportions (or ratios) of image regions in the target image that correspond to different coding modes. For example, if the image region in the target image that corresponds to the intraframe coding mode is larger than the image region in the target image that corresponds to the interframe coding mode, the value of the second classification information is determined to be the first value.

[0160] In some examples, the implementation of step S401-C above includes, but is not limited to, the following forms.

[0161] Form 1: If the proportion of image areas in the filtered image that correspond to the intraframe coding mode is greater than or equal to a first predetermined threshold, the value of the second classification information is determined to be a first value, and if the proportion of image areas in the filtered image that correspond to the intraframe coding mode is less than the first predetermined threshold, the value of the second classification information is determined to be a second value.

[0162] In some embodiments, the second classification information is a value of one element, for example, 0 or 1. For example, if the proportion of image regions in the filtered image that correspond to the intraframe coding mode is greater than or equal to a first predetermined threshold, the value of the second classification information is determined to be 0, and if the proportion of image regions in the filtered image that correspond to the intraframe coding mode is greater than or equal to the first predetermined threshold, the value of the second classification information is determined to be 1.

[0163] In some embodiments, the second classification information is a matrix. For example, if the proportion of image regions in the target image that correspond to intra-frame coding modes is greater than or equal to a first predetermined threshold, the second classification information is determined to be an all-zero matrix, and if the proportion of image regions in the target image that correspond to intra-frame coding modes is less than the first predetermined threshold, the second classification information is determined to be an all-one matrix.

[0164] The embodiment of the present application does not limit the specific value of the first threshold value.

[0165] In some embodiments, the first predetermined threshold is 80%.

[0166] For example, if the image area corresponding to the intraframe coding mode in the image to be filtered is greater than or equal to 80% of the size of the image to be filtered, the value of the second classification information is determined to be 0 (if the second classification information is a matrix, the value of each element in the second classification information is 0). If the image area corresponding to the intraframe coding mode in the image to be filtered is smaller than 80% of the size of the image to be filtered, the value of the second classification information is determined to be 1 (if the second classification information is a matrix, the value of each element in the second classification information is 1).

[0167] Form 2: If, in the image to be filtered, the proportion of the image region in the filtered image that corresponds to the intraframe coding mode is greater than or equal to a first predetermined threshold, the value of the second classification information is determined to be a first value; if, in the image to be filtered, the proportion of the image region in the filtered image that corresponds to the intraframe coding mode is less than the first predetermined threshold and greater than or equal to a second predetermined threshold (the second predetermined threshold is less than the first predetermined threshold), the value of the second classification information is determined to be a second value; if, in the image to be filtered, the proportion of the image region in the filtered image that corresponds to the intraframe coding mode is less than the second predetermined threshold, the value of the second classification information is determined to be a third value.

[0168] The embodiments of the present application do not limit the specific values ​​of the first value, the second value, and the third value.

[0169] In some embodiments, the first value is zero.

[0170] In some embodiments, the second value is one.

[0171] In some embodiments, the third value is two.

[0172] The embodiment of the present application does not limit the specific values ​​of the first predetermined threshold and the second predetermined threshold.

[0173] In some embodiments, the first predetermined threshold is 80%.

[0174] In some embodiments, the second predetermined threshold is 60%.

[0175] For example, if an image region corresponding to the intraframe coding mode in the image to be filtered is greater than or equal to 80% of the size of the image to be filtered, the value of the second classification information is determined to be 0 (if the second classification information is a matrix, the value of each element in the second classification information is 0). If an image region corresponding to the intraframe coding mode in the image to be filtered is smaller than 80% of the size of the image to be filtered and greater than or equal to 60% of the size of the image to be filtered, the value of the second classification information is determined to be 1 (if the second classification information is a matrix, the value of each element in the second classification information is 1). If an image region corresponding to the intraframe coding mode in the image to be filtered is smaller than 60% of the size of the image to be filtered, the value of the second classification information is determined to be 2 (if the second classification information is a matrix, the value of each element in the second classification information is 2).

[0176] In the second case, the classification information indicates a coding mode of the target image, which can be understood as an overall coding mode of the target image, such as an intraframe coding mode or an interframe coding mode. After the classification information is input to the filtering model, the filtering model can accurately filter the target image based on the coding mode of the target image to improve the filtering effect.

[0177] Case 3: The coding mode of the filtering target image includes at least one of a normal intraframe coding mode, a MIP mode, an IBC mode, an ISP mode, a PLT mode, a normal interframe coding mode, a Skip mode, a Merge mode, and a CIIP mode. In this case, if the classification information indicates the coding mode of the filtering target image, the classification information is indicated at a block level. If the classification information indicates the coding mode of the filtering target image, the process of determining the coding information of the filtering target image in the above step S401 includes the following steps:

[0178] In step S401-D, encoding modes corresponding to a plurality of image regions in the filtering target image are obtained, and classification information is determined based on the encoding modes corresponding to each image region in the filtering target image.

[0179] In an embodiment of the present application, one image region of the image to be filtered may be one pixel, several consecutive pixels, or one coding block. In an embodiment of the present application, coding modes corresponding to the plurality of image regions in the image to be filtered are obtained, and classification information of the image to be filtered is determined based on the coding modes corresponding to the plurality of image regions in the image to be filtered. For example, if the coding modes corresponding to the plurality of image regions in the image to be filtered are normal intraframe coding modes, the value of the classification information is determined to be a first value.

[0180] In some embodiments, the classification information can be determined based on the indexes of the coding modes. That is, the above step S401-D includes the following steps: in step S401-D1, determine the indexes of the coding modes corresponding to a plurality of image regions in the image to be filtered; and in step S401-D2, determine the classification information based on the indexes of the coding modes corresponding to a plurality of image regions in the image to be filtered.

[0181] In some embodiments, one coding mode corresponds to one index, and different index values ​​are pre-set for the normal intra-frame coding mode, MIP mode, IBC mode, ISP mode, PLT mode, normal inter-frame coding mode, skip mode, merge mode, and CIIP mode. Based on this, the classification information can be determined based on the index of the coding mode corresponding to each image region in the image to be filtered. For example, if the coding modes corresponding to each image region in the image to be filtered are all normal intra-frame coding modes and the index value of the normal intra-frame coding mode is 0, the classification information is determined to be a value of 0 or a matrix of all 0s.

[0182] In some embodiments, multiple coding modes correspond to one index. For example, the index value of the normal intra-frame coding mode is 0, the index values ​​of the MIP mode, IBC mode, ISP mode, and PLT mode are 1, the index value of the normal inter-frame coding mode is 2, the index values ​​of the skip mode and merge mode are 3, and the index value of the CIIP mode is 4. Based on this, classification information can be determined based on the indexes of coding modes corresponding to multiple image regions in the image to be filtered. For example, the coding modes corresponding to a first region in the image to be filtered are the skip mode and merge mode, and the coding mode corresponding to a second region is the normal inter-frame coding mode. In this way, the value of the element corresponding to the first region in the classification information is 3, and the value of the element corresponding to the second region in the coding mode indication information is 2. In this case, the classification information is a matrix.

[0183] In the above cases 1 to 3, the coding image type and coding method of the filtering target image indicated by the classification information have been described.

[0184] In some embodiments, the classification information may indicate the coded image type and coding mode described in Cases 1 to 3 above using three matrices. Exemplarily, matrix A indicates the coded image type of the image to be filtered, and specifically, the value of matrix A is determined according to the description of Case 1 above. Matrix B indicates whether the coding mode of the image to be filtered is an intra-frame coding mode, an inter-frame coding mode, or a combination of the intra-frame coding mode and the inter-frame coding mode, and specifically, the value of matrix B is determined according to the description of Case 2 above. Matrix C indicates the coding mode used for the image to be filtered, for example, whether the intra-frame coding mode or the inter-frame coding mode is used, and specifically, the value of matrix C is determined according to the description of Case 3 above.

[0185] When the classification indication information indicates both the encoding image type and the encoding mode, the classification information includes at least one of matrix A, matrix B, and matrix C.

[0186] For example, if the classification indication information indicates the encoding image type but does not indicate the encoding mode, the classification information may include only matrix A, without including matrix B and matrix C.

[0187] Illustratively, if the classification indication information does not indicate the encoding image type but indicates the encoding mode, the classification information does not include matrix A, but includes at least one of matrix B and matrix C.

[0188] In some embodiments, depending on the actual situation, the above matrix may be data having only one element.

[0189] In the embodiment of the present application, according to the above method, after the classification information of the filtering target image is determined, the following step S402 is executed.

[0190] In step S402, the filtering target image and classification information of the filtering target image are input into the same deep learning-based filtering model, and the filtering model filters the filtering target image based on the classification information to obtain a filtered image.

[0191] The filtering model of the present embodiment is a general deep learning-based model. To improve the model's prediction accuracy, as shown in FIG. 7A, in addition to inputting the filtering target image to the filtering model, classification information of the filtering target image is also input to the filtering model. That is, by introducing auxiliary information of the coding type as input to the filtering model, it is possible to avoid the degradation of filtering performance caused by using the same filter to process images with many features (simultaneous processing of images including intraframe coding mode and interframe coding mode), and to provide more precise image information to the image. This allows for improved filtering effectiveness when filtering each image region of the filtering target image based on the classification information of the filtering target image.

[0192] The embodiments of the present application are not limited to a specific network structure of the filtering model, and FIG. 7A is merely an example.

[0193] In some embodiments, in the above step S402, the process of inputting the image to be filtered and the classification information of the image to be filtered into the same deep learning-based filtering model includes the following steps: In step S402-A, at least one of the image to be filtered and the classification information is pre-processed and then input into the filtering model.

[0194] Preprocessing example 1: After merging the layers of the image to be filtered and the classification information, input them into the filtering model.

[0195] For example, layer merging is achieved by the concatenation operation of DenseNet, and DenseNet is used to merge channels. As an example of layer merging, one new classification information channel is added to record classification information based on the grayscale data of three channels (red, green, and blue) contained in the filtered image. As a result, the merged filtered image contains data of four channels.

[0196] Preprocessing example 2: The image to be filtered is multiplied by classification information and then input to the filtering model.

[0197] For example, multiplication can be realized by dot multiplication: the grayscale data of each channel (red, green, blue) of each pixel in the image to be filtered is multiplied by the embedding vector representation of the classification information, and the resulting grayscale data of the three channels is used as the input of the filtering model.

[0198] Pre-processing example 3: As shown in Figure 7B, the image to be filtered is multiplied by the classification information (for example, the point multiplication mentioned above) and then input into the first feature extraction unit to obtain the first characteristic information after fusing the image to be filtered with the classification information; the image to be filtered is input into the second feature extraction unit to perform feature extraction to obtain the second characteristic information of the image to be filtered; and the first characteristic information and the second characteristic information are input into the filtering model to obtain the filtered image of the image to be filtered.

[0199] The embodiments of the present application do not limit the specific network structure of the filtering model, and FIG. 7B is merely an example. The embodiments of the present application do not limit the specific network structures of the first feature extraction unit and the second feature extraction unit. For example, as shown in FIG. 7B, the first feature extraction unit includes one convolutional layer conv (step length S=1) and one activation function (prelu activation function), and the second feature extraction unit also includes one convolutional layer (step length S=1) and one activation function (prelu function). In some embodiments, the activation function may be other types of activation functions, such as a relu activation function, in addition to the prelu activation function shown in FIG. 7B, and the embodiments of the present application are not limited thereto. The structure of the second feature extraction unit may be the same as that of the first feature extraction unit.

[0200] In some embodiments, in the above step S402, the process of inputting the to-be-filtered image and the classification information of the to-be-filtered image into the same deep learning-based filtering model includes the following steps: in step S402-B, input at least one of a predicted image corresponding to the to-be-filtered image, a quantization parameter, and block division information, and the to-be-filtered image and the classification information into the filtering model, where the predicted image is a predicted image corresponding to the current to-be-filtered image and is formed by stitching each predicted block.

[0201] The block division information is a block division image corresponding to the current filtering target image. There are two ways to generate it. The first way is to assign different values ​​to the block division boundary and the non-block division boundary, and the second way is to fill the area (coding unit) within each block division boundary with the average value of the filtering target image within the current area.

[0202] The quantization parameters include at least one of a sequence-level quantization parameter and a slice-level quantization parameter. The sequence-level quantization parameter is a quantization parameter that operates on the current sequence (such as all frames, slices, and tiles), and other quantization parameters are obtained by adding offsets based on the sequence-level quantization parameter. The slice-level quantization parameter is a quantization parameter that operates on the current slice.

[0203] In one example, at least one of a predicted image, a quantization parameter, and block division information corresponding to the image to be filtered is combined with the image to be filtered and classification information, and then input into a filtering model to obtain a filtered image of the image to be filtered.

[0204] In another example, step S402-B includes the following steps: in step S402-B1, a predicted image is input to a first processing unit to obtain feature information of the predicted image; in step S402-B2, a target image to be filtered is input to a second processing unit to obtain feature information of the target image; in step S402-B3, at least one of the feature information of the predicted image, the quantization parameter, and the block division information, and the feature information and classification information of the target image to be filtered are input to a filtering model to obtain a filtered image of the target image to be filtered.

[0205] In the embodiment of the present application, first, a predicted image corresponding to the image to be filtered is input to a first processing unit to obtain feature information of the predicted image. The image to be filtered is input to a second processing unit to obtain feature information of the image to be filtered. As shown in Figure 7C, at least one of the feature information of the predicted image, the quantization parameter, and the block division information, and the feature information and classification information of the image to be filtered are input to a filtering model to obtain a filtered image of the image to be filtered.

[0206] The embodiments of the present application do not limit the specific network structures of the first processing unit, the second processing unit, and the filtering model. For example, as shown in FIG. 7C, the first processing unit includes one convolution layer and one activation function. For example, as shown in FIG. 7C, the second processing unit includes one convolution layer and one activation function.

[0207] In some embodiments, the activation function may be other types of activation functions in addition to the prelu activation function shown in FIG. 7C, and the embodiments of the present application are not limited thereto.

[0208] The network structure of the filtering model in the embodiment of the present application may be other structures in addition to the structures shown in FIGS. 7A to 7C, and the present application does not limit the specific network structure of the filtering model.

[0209] In some embodiments, at least one of the feature information, quantization parameters, and block division information of the predicted image is combined with the feature information and classification information of the image to be filtered, and then input to the filtering model.

[0210] For example, in FIG. 7C, the feature information of the predicted image, the quantization parameter, the feature information of the image to be filtered, and the classification information are merged and then input into the filtering model.

[0211] In another example, in FIG. 7C, the feature information of the predicted image, the quantization parameter, the feature information of the image to be filtered, and the classification information are multiplied together and then input to the filtering model.

[0212] 7A to 7C, in some embodiments, the filtering model of the present application includes a network structure of the filtering model, which includes the following sequentially connected structures: a convolutional layer (step length s=1), an activation function (prelu activation function), N residual blocks (N is a positive integer), a convolutional layer (step length s=1), and an upsampling layer. The network structures of the N residual blocks may be the same or different.

[0213] In some embodiments, the number of input channels and the number of output channels of each convolutional layer included in at least one of the N residual blocks are the same.

[0214] For example, as shown in FIG. 8A , at least one of the N residual blocks includes one 1×1 convolutional layer and one 3×3 convolutional layer, and the number of input channels of the 1×1 convolutional layer and the 3×3 convolutional layer is equal to the number of output channels.

[0215] In some embodiments, an attention layer may be added based on the residual block shown in Figure 8A. For example, a Convolutional Block Attention Module (CBAM) may be added as shown in Figure 8B.

[0216] In some embodiments, the number of input channels and the number of output channels of each convolutional layer included in at least one of the N residual blocks are different.

[0217] For example, as shown in FIG. 8C , at least one of the N residual blocks includes one 1x1 convolutional layer and one 3x3 convolutional layer, and the number of input channels of the 1x1 convolutional layer and the 3x3 convolutional layer is not equal to the number of output channels. For example, the number of channels may be increased before the activation function and decreased after the activation function. That is, the number of input channels and the number of output channels of the first 1x1 convolutional layer are 64 and 192, respectively, the number of input channels and the number of output channels of the second 1x1 convolutional layer are 192 and 64, respectively, and the number of input channels and the number of output channels of the third 3x3 convolutional layer are 64 and 64, respectively.

[0218] In some embodiments, an attention layer may be added based on the residual block as shown in Figure 8C, for example, a Convolutional Block Attention Module (CBAM) is added as shown in Figure 8D.

[0219] 7A to 8D are merely schematic diagrams of the network structure of the filtering model according to the present application. The network structure of the filtering model according to the embodiment of the present application includes, but is not limited to, those shown in the above-mentioned FIGS. 7A to 8D. The specific network parameters shown in the above-mentioned FIGS. 7A to 8D are also merely examples, and the embodiment of the present application is not limited to these.

[0220] The filtering model of the embodiment of the present application can realize filtering of the filtering target image in intraframe coding mode and can also realize filtering of the filtering target image in interframe coding mode. This corresponds to a general-purpose filtering model for the filtering target image in different modes, and compared with constructing corresponding filtering models for different modes, the storage space occupation rate of the model parameters is significantly reduced and the storage cost of the model parameters is low. In addition, by combining classification information with the filtering model and performing differential filtering on the filtering target image, the filtering target image in different modes can be applied, and the filtering effect of the filtering target image can be improved.

[0221] The image filtering method according to the embodiment of the present application has been described in detail above. Before using a filtering model to perform the image filtering, the filtering model needs to be trained first. The following describes the training process of the filtering model.

[0222] As shown in FIG. 9, the training process of the filtering model in the embodiment of the present application includes the following steps:

[0223] Step 901: Generate a dataset.

[0224] Generate a training set using the (Codec) software.

[0225] The training images included in the training set include images in the intraframe coding mode, and also images in the interframe coding mode.

[0226] In addition, the training set further includes coding information for each training image, where the coding information includes classification information, and the classification information indicates at least one of a coding image type and a coding mode of the training image.

[0227] Step 902: Training the model.

[0228] The training set is used to train the filtering model, specifically, the training images and classification information of the training images are input into the filtering model to train the filtering model, and a training filtering model is obtained.

[0229] In the model training process, a loss function is used. The loss function measures the difference between the predicted value and the actual value. A larger loss value indicates a larger difference, and the goal of training is to reduce the loss. The loss functions in the present embodiment include an L1 norm loss function, an L2 norm loss function, and a smooth L1 loss function.

[0230] In practice, when using a filtering model, rather than directly inputting an entire frame for filtering, it is common to divide the image into sub-images and then input the sub-images sequentially into the filtering model for filtering. Because the sub-images vary in size, the image to be filtered that is input to the filtering model may cross slice / tile boundaries.

[0231] Based on this, in some embodiments, in addition to inputting the training image and the classification information of the training image into the filtering model, at least one of a predicted image corresponding to the training image, a quantization parameter, and block division information is further input into the filtering model, so that the filtered image can learn more information of the training image, thereby improving the filtering effect of the model.

[0232] Step 903: Software integration.

[0233] The trained filtering model is integrated into software for filtering, for example, the trained filtering model is integrated into the encoder and / or decoder to filter the reconstructed image, thereby improving the reconstruction effect.

[0234] 4 to 9 are merely examples of the embodiments of the present application and should not be construed as limiting the embodiments of the present application.

[0235] Although preferred embodiments of the present application have been described above with reference to the drawings, the present application is not limited to the specific details of the above embodiments. Various simple modifications can be made to the technical solutions of the present application within the scope of the technical concepts of the present application, and all such simple modifications fall within the scope of protection of the present application. For example, the specific technical features described in the above specific embodiments can be combined in any appropriate manner without contradiction, and to avoid unnecessary repetition, the present application does not describe various possible combinations. In other words, various different embodiments of the present application can also be combined in any manner, and as long as they do not violate the concept of the present application, such combinations should also be considered as disclosed in the present application.

[0236] Thus, the method of the present invention will be described in detail with reference to FIGS. 4 to 12, and the apparatus of the present invention will be described in detail below.

[0237] 10 is a schematic diagram illustrating a configuration of an image filtering device according to an embodiment of the present application. The image filtering device 10 includes: a determining unit 11 configured to determine coding information of a target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; and a filtering unit 12 configured to input the target image and the classification information into a same deep learning-based filtering model, and cause the filtering model to filter the target image based on the classification information to obtain a filtered image.

[0238] In some embodiments, the coded image type includes a full intra-frame coded image and an inter-frame codeable image, and when the classification information indicates the coded image type of the image to be filtered, in some embodiments, the classification information is indicated at a slice level, an image block level, or an image header level.

[0239] In this case, the determination unit 11 is specifically configured to obtain coded image types corresponding to multiple image regions in the image to be filtered, and determine that the value of the classification information is a first value if all of the multiple image regions belong to the full-frame intra-coded image, determine that the value of the classification information is a second value if all of the multiple image regions belong to the inter-frame codable image, and determine that the value of the classification information is a third value if the image to be filtered crosses a slice boundary or a tile boundary, or determine the value of the classification information based on the respective proportions of image regions in the image to be filtered that belong to the full-frame intra-coded image and image regions that belong to the inter-frame codable image among the multiple image regions.

[0240] In some embodiments, the determination unit 11 is specifically configured to determine that the value of the classification information is the first value if the proportion of image areas in the filtered image that belong to the full-frame intra-coded image among the multiple image areas of the filtered image is greater than or equal to a first threshold, and to determine that the value of the classification information is the second value if the proportion of image areas in the filtered image that belong to the full-frame intra-coded image among the multiple image areas of the filtered image is less than the first threshold.

[0241] In some embodiments, the determination unit 11 is specifically configured to determine that the value of the classification information is the first value if, among the multiple image regions of the filtered image, a proportion of image regions in the filtered image that belong to the full-frame intra-coded image is greater than or equal to a first threshold; to determine that the value of the classification information is the second value if, among the multiple image regions of the filtered image, a proportion of image regions in the filtered image that belong to the full-frame intra-coded image is less than the first threshold and greater than or equal to a second threshold; and to determine that the value of the classification information is the third value if, among the multiple image regions of the filtered image, a proportion of image regions in the filtered image that belong to the full-frame intra-coded image is less than the second threshold.

[0242] In some embodiments, the coding modes include an intraframe coding mode and an interframe coding mode, and when the classification information indicates a coding mode of the target image, the classification information includes at least one of first classification information and second classification information, where the first classification information indicates a coding mode corresponding to one or more pixels in the target image and the second classification information indicates a coding mode corresponding to a predetermined size region in the target image. In some embodiments, the classification information is indicated at a block level.

[0243] In some embodiments, when the classification information includes the first classification information, and the first classification information is a matrix consisting of multiple elements, and each element in the matrix corresponds to an encoding mode of one or more pixels in the filtered image, the determining unit 11 is specifically configured to obtain the encoding modes corresponding to one or more pixels in the filtered image, and determine the first classification information based on the encoding modes corresponding to the one or more pixels in the filtered image.

[0244] In some embodiments, the determining unit 11 is specifically configured to determine that if the coding mode corresponding to all pixels in the image to be filtered is the intra-frame coding mode, the value of each matrix element in the first classification information is a first value; if the coding mode corresponding to all pixels in the image to be filtered is the inter-frame coding mode, the value of each matrix element in the first classification information is a second value; and if the coding mode corresponding to one or more pixels in a first image region in the image to be filtered is the intra-frame coding mode and the coding mode corresponding to one or more pixels in a second image region is the inter-frame coding mode, the value of the matrix element corresponding to one or more pixels in the first image region in the first classification information is the first value and the value of the matrix element corresponding to one or more pixels in the second image region is the second value.

[0245] In some embodiments, when the classification information includes the second classification information, the determining unit 11 is specifically configured to determine the second classification information based on the proportions in the filtered image of image areas that correspond to different encoding modes in the filtered image.

[0246] In some embodiments, the determination unit 11 is specifically configured to determine that the value of the second classification information is a first value if the proportion of image areas in the filtered image that correspond to the intra-frame coding mode in the filtered image is greater than or equal to a first predetermined threshold, and to determine that the value of the second classification information is a second value if the proportion of image areas in the filtered image that correspond to the intra-frame coding mode in the filtered image is less than the first predetermined threshold.

[0247] In some embodiments, the determination unit 11 is specifically configured to determine that the value of the second classification information is a first value if the proportion of image areas in the filtered image that correspond to the intra-frame coding mode is greater than or equal to a first predetermined threshold, to determine that the value of the second classification information is a second value if the proportion of image areas in the filtered image that correspond to the intra-frame coding mode is less than the first predetermined threshold and greater than or equal to a second predetermined threshold, and to determine that the value of the second classification information is a third value if the proportion of image areas in the filtered image that correspond to the intra-frame coding mode is less than the second predetermined threshold.

[0248] In some embodiments, the coding mode includes at least one of a normal intraframe coding mode, a matrix weighted intraframe prediction (MIP) mode, an intraframe block replication (IBC) mode, an intraframe string prediction (ISP) mode, a palette (PLT) mode, a normal interframe coding mode, a skip mode, a motion merge mode, and a combined interframe and intraframe prediction (CIIP) mode, and when the classification information indicates the coding mode of the image to be filtered, in some embodiments, the classification information is indicated at the block level.

[0249] In this case, the determination unit 11 is specifically configured to obtain encoding modes corresponding to multiple image regions in the image to be filtered, and determine the classification information based on the encoding modes corresponding to the multiple image regions in the image to be filtered.

[0250] In some embodiments, the determination unit 11 is specifically configured to determine indices of coding modes corresponding to a plurality of image regions in the image to be filtered, and to determine the classification information based on the indices of coding modes corresponding to a plurality of image regions in the image to be filtered.

[0251] In some embodiments, one coding mode corresponds to one index, or multiple coding modes correspond to one index.

[0252] In some embodiments, the filtering unit 12 is specifically configured to pre-process at least one of the image to be filtered and the classification information before inputting them into the filtering model.

[0253] In some embodiments, the filtering unit 12 is specifically configured to input the image to be filtered and the classification information into the filtering model after layer merging.

[0254] In some embodiments, the filtering unit 12 is specifically configured to multiply the image to be filtered by the classification information and then input it into a first feature extraction unit to obtain first feature information after fusing the image to be filtered with the classification information, input the image to be filtered to a second feature extraction unit to perform feature extraction to obtain second feature information of the image to be filtered, and input the first feature information and the second feature information into the filtering model.

[0255] In some embodiments, the filtering unit 12 is specifically configured to input at least one of a predicted image, a quantization parameter, and block division information corresponding to the image to be filtered, the image to be filtered, and the classification information into the filtering model.

[0256] In some embodiments, the filtering unit 12 is specifically configured to input the predicted image to a first processing unit to obtain feature information of the predicted image, input the image to be filtered to a second processing unit to obtain feature information of the image to be filtered, and input at least one of the feature information of the predicted image, the quantization parameter, and the block division information, and the feature information and the classification information of the image to be filtered to the filtering model.

[0257] In some embodiments, the quantization parameter comprises at least one of a sequence-level quantization parameter and a slice-level quantization parameter.

[0258] In some embodiments, the filtering model includes N residual blocks, where N is a positive integer.

[0259] In some embodiments, the number of input channels and the number of output channels of each convolutional layer included in at least one residual block among the N residual blocks are the same, or the number of input channels and the number of output channels of each convolutional layer included in at least one residual block among the N residual blocks are different.

[0260] In some embodiments, at least one residual block of the N residual blocks comprises a layer of interest.

[0261] In some embodiments, the filtered image is used as a reference image for subsequent images, or the filtered image is used for display and is not used as a reference image for subsequent images.

[0262] It should be noted that the device embodiments and the method embodiments correspond to each other, and similar descriptions can be referred to in the method embodiments. To avoid redundancy, the description will not be repeated here. Specifically, the device 10 shown in FIG. 10 can execute the above-mentioned method embodiments, and the above-mentioned operations and other operations and / or functions of each module in the device 16 are respectively used to realize the above-mentioned method embodiments, and for brevity, the description will not be repeated here.

[0263] The above describes the apparatus of the embodiments of the present application in terms of functional modules, with reference to the drawings. The functional modules may be implemented in the form of hardware, software instructions, or a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application may be completed by a hardware integrated logic circuit in a processor and / or software instructions. The method steps disclosed in the embodiments of the present application may be completed by a hardware decoding processor or a combination of hardware and software modules in the decoding processor. Optionally, the software modules may be located in a conventional storage medium, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in a memory, and the processor reads information in the memory and completes the steps of the above method embodiments in combination with the hardware.

[0264] 11 is an exemplary block diagram of an electronic device according to an embodiment of the present application, which may be the image filtering device described above, or a video encoder, or a video decoder.

[0265] 11, the electronic device 40 may include a memory 41 and a processor 42, where the memory 41 is configured to store a computer program and transmit the program code to the processor 42. In other words, the processor 42 can implement the method in the embodiments of the present application by calling and executing the computer program from the memory 41. For example, the processor 42 may be configured to execute the above-described method embodiments according to instructions in the computer program.

[0266] In some embodiments of the present application, the processor 42 may include, but is not limited to, a general purpose computer, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, or the like.

[0267] In some embodiments of the present application, memory 41 may include, but is not limited to, volatile and / or non-volatile memory. Non-volatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM) used as an external cache. By way of illustrative but non-limiting example, RAM may come in many forms, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (DR RAM).

[0268] In some embodiments of the present application, the computer program may be divided into one or more modules, which are stored in the memory 41 and executed by the processor 42 to complete the method provided by the embodiments of the present application. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, and the instruction segments are used to describe the execution process of the computer program in the video production device.

[0269] 11, the electronic device 40 may further include a transceiver 43, which may be connected to the processor 42 or memory 41. Here, the processor 42 may control the transceiver 43 to communicate with other devices, specifically, to transmit information or data to other devices or receive information or data transmitted by other devices. The transceiver 43 may include a transmitter and a receiver. The transceiver 43 may further include one or more antennas.

[0270] The components of the video production device are connected by a bus system, which includes a power bus, a control bus, and a status signal bus in addition to a data bus.

[0271] The present application further provides a computer-readable storage medium having a computer program stored therein, the computer program causing the electronic device to perform the method in the above method embodiments when executed by the electronic device, or a computer program product including instructions, the instructions causing the electronic device to perform the method in the above method embodiments when executed by a computer.

[0272] Modules described as separate parts may or may not be physically separated, and parts shown as modules may or may not be physical modules, located in one location, or distributed across multiple network units. Depending on actual needs, some or all of the parts may be selected to achieve the objectives of the technical solutions in the embodiments. For example, each functional module in each embodiment of the present application may be integrated into one processing module, or each unit may exist physically separately, or two or more modules may be integrated into one module.

[0273] The above content is merely a specific embodiment of the present application, and the protection scope of the present application is not limited thereto. Any modifications or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; The coded image type includes a full intra-frame coded image and an inter-frame codable image, and when the classification information indicates the coded image type of the filtering target image, the step of determining coding information of the filtering target image includes: obtaining coded image types corresponding to a plurality of image regions in the image to be filtered; determining that the value of the classification information is a first value if all of the plurality of image regions belong to the full-frame intra-coded image; determining that the value of the classification information is a second value if all of the image regions belong to the inter-codable image; determining that the value of the classification information is a third value if the image to be filtered crosses a slice boundary or a tile boundary, or determining the value of the classification information based on the respective proportions in the image to be filtered of image regions that belong to the full-frame intra-coded image and image regions that belong to the inter-frame codable image among the plurality of image regions, Image filtering methods.

2. The step of determining the classification information based on the respective proportions of the image regions belonging to the full-frame intra-coded image and the image regions belonging to the inter-frame codable image in the filtering target image, among the plurality of image regions, includes: determining that the value of the classification information is the first value when a proportion of the image regions in the filtering target image that belong to the full-frame intra-coded image among the plurality of image regions is greater than or equal to a first threshold value; determining that the value of the classification information is the second value when a ratio of image regions belonging to the full-frame intra-coded image among the plurality of image regions in the filtering target image is smaller than the first threshold value; The image filtering method of claim 1 .

3. The step of determining the value of the classification information based on the respective proportions in the filtering target image of image regions that belong to the full-frame intra-coded image and image regions that belong to the inter-frame codable image among the plurality of image regions, includes: determining that the value of the classification information is the first value when a proportion of the image regions in the filtering target image that belong to the full-frame intra-coded image among the plurality of image regions is greater than or equal to a first threshold value; determining that the value of the classification information is the second value when a proportion of image regions belonging to the full-frame intra-coded image among the plurality of image regions in the filtering target image is smaller than the first threshold value and greater than or equal to a second threshold value; determining that the value of the classification information is the third value when a ratio of image regions belonging to the full-frame intra-coded image among the plurality of image regions in the filtering target image is smaller than the second threshold value; The image filtering method of claim 1 .

4. The classification information is indicated at a slice level, an image block level, or an image header level. The image filtering method of claim 1 .

5. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; the coding mode includes an intraframe coding mode and an interframe coding mode, and when the classification information indicates a coding mode of a filtering target image, the classification information includes at least one of first classification information and second classification information; the first classification information indicates an encoding mode corresponding to one or more pixels in the filtered image, and the second classification information indicates an encoding mode corresponding to a predetermined size region in the filtered image; When the classification information includes the second classification information, the step of determining coding information of the filtering target image includes: determining the second classification information based on a proportion in the target image of image regions corresponding to different encoding modes in the target image; Image filtering methods.

6. When the classification information includes the first classification information, the step of determining coding information of the filtering target image includes: obtaining an encoding mode corresponding to one or more pixels in the to-be-filtered image; determining first classification information in the coded information based on a coding mode corresponding to one or more pixels in the filtered image; the first classification information is a matrix of elements, each element of the matrix corresponding to an encoding mode of one or more pixels in the image to be filtered; 6. The image filtering method according to claim 5.

7. determining the first classification information based on an encoding mode corresponding to one or more pixels in the target image, determining that a value of each matrix element in the first classification information is a first value when a coding mode corresponding to all pixels in the filtering target image is the intra-frame coding mode; determining that a value of each matrix element in the first classification information is a second value when a coding mode corresponding to all pixels in the filtering target image is the inter-frame coding mode; determining, when a coding mode corresponding to one or more pixels in a first image region in the filtered image is the intra-frame coding mode and a coding mode corresponding to one or more pixels in a second image region is the inter-frame coding mode, that values ​​of matrix elements in the first classification information corresponding to one or more pixels in the first image region are the first value and that values ​​of matrix elements corresponding to one or more pixels in the second image region are the second value; 6. The image filtering method according to claim 5.

8. determining the second classification information based on a proportion in the target image of image regions corresponding to different encoding modes in the target image of filtering, determining that the value of the second classification information is a first value if a proportion of image regions in the filtering target image that correspond to the intraframe coding mode is greater than or equal to a first predetermined threshold; determining that the value of the second classification information is a second value when a ratio of an image region in the filtering target image corresponding to the intra-frame coding mode is smaller than the first predetermined threshold value; 6. The image filtering method according to claim 5.

9. determining the second classification information based on a proportion in the target image of image regions corresponding to different encoding modes in the target image of filtering, determining that the value of the second classification information is a first value if a proportion of image regions in the filtering target image that correspond to the intraframe coding mode is greater than or equal to a first predetermined threshold; determining that the value of the second classification information is a second value when a ratio of an image region in the filtering target image that corresponds to the intraframe coding mode is smaller than the first predetermined threshold and is greater than or equal to a second predetermined threshold; determining that the value of the second classification information is a third value when a ratio of an image region in the filtering target image that corresponds to the intra-frame coding mode is smaller than the second predetermined threshold value; 6. The image filtering method according to claim 5.

10. The classification information is indicated at the block level.

6. The image filtering method according to claim 5.

11. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; the coding mode includes at least one of a normal intra-frame coding mode, a matrix weighted intra-frame prediction (MIP) mode, an intra-frame block cloning (IBC) mode, an intra-frame string prediction (ISP) mode, a palette (PLT) mode, a normal inter-frame coding mode, a skip mode, a motion merge mode, and a combined inter- and intra-frame prediction (CIIP) mode; If the classification information indicates an encoding mode of the filtering target image, determining encoding information of the filtering target image comprises: obtaining coding modes corresponding to a plurality of image regions in the target image; determining classification information in the encoding information based on encoding modes corresponding to a plurality of image regions in the filtered image; The step of determining classification information in the encoding information based on encoding modes corresponding to a plurality of image regions in the filtering target image includes: determining indices of coding modes corresponding to a plurality of image regions in the to-be-filtered image; determining the classification information based on indexes of coding modes corresponding to a plurality of image regions in the target image; Image filtering methods.

12. One coding mode corresponds to one index, or multiple coding modes correspond to one index; The image filtering method according to claim 11.

13. The classification information is indicated at the block level. The image filtering method according to claim 11.

14. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; The step of inputting the image to be filtered and the classification information into the same deep learning-based filtering model includes: and preprocessing at least one of the image to be filtered and the classification information before inputting the preprocessed information into the filtering model. Image filtering methods.

15. The step of preprocessing at least one of the image to be filtered and the classification information and then inputting the preprocessed image and the classification information into the filtering model includes: a step of inputting the filtering target image and the classification information into the filtering model after layer merging the images; 15. The image filtering method of claim 14.

16. The step of preprocessing at least one of the image to be filtered and the classification information and then inputting the preprocessed image and the classification information into the filtering model includes: multiplying the target image and the classification information, and then inputting the result into a first feature extraction unit to obtain first feature information after fusing the target image and the classification information; inputting the filtering target image to a second feature extraction unit to perform feature extraction, and obtaining second feature information of the filtering target image; inputting the first characteristic information and the second characteristic information into the filtering model; 15. The image filtering method of claim 14.

17. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; The step of inputting the image to be filtered and the classification information into the same deep learning-based filtering model includes: inputting at least one of a predicted image, a quantization parameter, and block division information corresponding to the filtering target image, the filtering target image, and the classification information into the filtering model; The step of inputting at least one of a predicted image corresponding to the filtering target image, a quantization parameter, and block division information, the filtering target image, and the classification information into the filtering model includes: inputting the predicted image into a first processing unit to obtain feature information of the predicted image; inputting the filtering target image to a second processing unit to obtain feature information of the filtering target image; inputting at least one of feature information of the predicted image, the quantization parameter, and the block division information, and feature information and classification information of the filtering target image into the filtering model, Image filtering methods.

18. the quantization parameters include at least one of a sequence-level quantization parameter and a slice-level quantization parameter.

18. The image filtering method of claim 17.

19. 1. A method for filtering an image, comprising: determining coding information of the target image, the coding information including classification information of the target image, the classification information indicating at least one of a coding image type and a coding mode of the target image; inputting the target image and the classification information into a same deep learning-based filtering model, and causing the filtering model to filter the target image based on the encoding information to obtain a filtered image; the filtering model includes N residual blocks, where N is a positive integer; Image filtering methods.

20. the number of input channels and the number of output channels of each convolutional layer included in at least one residual block among the N residual blocks are the same, or the number of input channels and the number of output channels of each convolutional layer included in at least one residual block among the N residual blocks are different; 20. The image filtering method of claim 19.

21. At least one residual block among the N residual blocks includes a layer of interest.

20. The image filtering method of claim 19.

22. The filtered image is used as a reference image for a subsequent image, or the filtered image is used for display and is not used as a reference image for a subsequent image. The image filtering method of claim 1 .

23. Image filtering device configured to perform the method of any one of claims 1 to 22.

24. An electronic device, An electronic device comprising a processor and a memory, the memory being configured to store a computer program, and the processor executing the method of any one of claims 1 to 22 by calling and executing the computer program stored in the memory.

25. A computer program comprising computer program instructions which, when executed on an electronic device, cause said electronic device to carry out the method of any one of claims 1 to 22.

Images