Neural network-based image filtering method, encoding / decoding method, device, equipment, and computer program

Abstract

The present invention provides a neural network-based image filtering method, encoding / decoding method, device, equipment, and storage medium applicable to fields such as image processing, video encoding / decoding, and video live streaming. The method includes: determining input information for a neural network filter based on an image to be filtered, the input information including the image to be filtered; and using the neural network filter to perform a filtering process on the input information to obtain a filtered image of the image to be filtered. The neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method. This reduces the complexity of the filtering operation of the neural network filter and improves the image filtering effect without degrading the filtering performance of the neural network filter.

Description

[Technical Field]

[0001] This application claims priority from a Chinese patent application bearing application number 202310760900.5, filed with the China Patent Office on June 26, 2023, and entitled "Neural network-based image filtering method, encoding / decoding method, device, equipment, and storage medium," the entire contents of which are incorporated herein by reference.

[0002] The present invention relates to the field of image processing technology, and more particularly to an image filtering method, encoding / decoding method, device, equipment, and storage medium based on a neural network. [Background technology]

[0003] With the development of video technology, the amount of data contained in video data has increased. To facilitate the transmission of video data, video devices implement video compression techniques to transmit or store video data more efficiently. In video compression, both the encoding side and the decoding side need to perform operations such as inverse quantization and inverse transform to obtain a reconstructed image. Since loss is introduced in video compression, the reconstructed image can be filtered to reduce the compression loss of the image.

[0004] With the rapid development of neural network technology, neural network filters have been widely applied in video processing. However, current neural network filters cannot balance filtering performance and filtering complexity, resulting in poor image filtering performance. Summary of the Invention

[0005] The present invention provides an image filtering method, encoding / decoding method, device, equipment, and storage medium based on a neural network, which ensures image filtering performance, reduces the complexity of filtering operations, and improves filtering effects.

[0006] A first aspect of an embodiment of the present invention provides a neural network-based image filtering method executed by a filtering device, the method comprising: a step of determining input information of a neural network filter based on an image to be filtered, the input information including the image to be filtered; and a step of performing a filtering process on the input information by the neural network filter to obtain a filtered image of the image to be filtered, wherein the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0007] A second aspect of an embodiment of the present invention provides an image decoding method executed by a decoding device, the method including the steps of: decoding a bitstream; determining a reconstructed image of a current image; and determining input information of a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; and performing a filtering process on the input information by the neural network filter to obtain a filtered image of the reconstructed image, wherein the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0008] A third aspect of an embodiment of the present invention provides an image encoding method executed by an encoding device, the method including the steps of: encoding a current image; determining a reconstructed image of the current image; and determining input information for a neural network filter based on the reconstructed image, where the input information includes the reconstructed image; and filtering the input information using the neural network filter to obtain a filtered image of the reconstructed image, where the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

[0009] A fourth aspect of the present invention provides an image filtering device based on a neural network, which is applied to a filtering device, and includes: a determination unit that determines input information of a neural network filter based on an image to be filtered, where the input information includes the image to be filtered; and a filtering unit that performs a filtering process on the input information using the neural network filter to obtain a filtered image of the image to be filtered, wherein the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two methods selected from the group consisting of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0010] A fifth aspect of the present invention provides an image decoding device applicable to a decoding device, the image decoding device including: a decoding unit that decodes a bitstream, determines a reconstructed image of a current image, and determines input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image; and a filtering unit that performs a filtering process on the input information using the neural network filter to obtain a filtered image of the reconstructed image, the neural network filter including a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0011] A sixth aspect of the present invention provides an image encoding device applicable to an encoding device, the image encoding device including: an encoding unit that encodes a current image, determines a reconstructed image of the current image, and determines input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image; and a filtering unit that filters the input information using a neural network filter to obtain a filtered image of the reconstructed image, the neural network filter including a plurality of convolution layers for extracting image feature information, the plurality of convolution layers being obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

[0012] In a seventh aspect of an embodiment of the present invention, there is provided a decoder including a processor and a memory having a computer program stored therein, wherein the processor calls and executes the computer program stored in the memory to perform a method according to the second aspect or any of the embodiments thereof.

[0013] In an eighth aspect of an embodiment of the present invention, there is provided an encoder including a processor and a memory having a computer program stored therein, wherein the processor calls and executes the computer program stored in the memory to perform a method according to the third aspect or any of the embodiments thereof.

[0014] In a ninth aspect of the present invention, there is provided a chip for performing a method according to any one of the first to third aspects or respective embodiments thereof, the chip including a processor for calling and executing a computer program from a memory so as to cause a device to which the chip is attached to perform the method according to any one of the first to third aspects or respective embodiments thereof.

[0015] In a tenth aspect of the present invention, there is provided a computer-readable storage medium for storing a computer program, the computer program causing a computer to perform a method according to any one of the first to third aspects or respective embodiments thereof.

[0016] In an eleventh aspect of the present invention, there is provided a computer program product including computer program instructions for causing a computer to perform a method according to any one of the first to third aspects or respective embodiments thereof.

[0017] In a twelfth aspect of the present invention, there is provided a computer program which, when executed by a computer, causes the computer to perform a method according to any one of the first to third aspects or respective embodiments thereof.

[0018] Thus, the present invention provides a new neural network filter, which includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method. That is, the present invention decomposes the existing standardized convolution operation into a convolution operation with lower computational complexity using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method, thereby reducing the complexity of the filtering operation of the neural network filter without degrading the filtering performance of the neural network filter and improving the image filtering effect. [Brief explanation of the drawings]

[0019] [Figure 1] 1 is a schematic block diagram of a video encoding / decoding system according to an embodiment of the present invention; [Figure 2] 1 is a schematic diagram of an encoding framework according to an embodiment of the present invention; [Figure 3] FIG. 1 is a schematic diagram of a decoding framework according to an embodiment of the present invention; [Figure 4] FIG. 2 is a schematic diagram of an example of an encoding unit. [Figure 5] FIG. 1 is a schematic diagram of an example of a filtering process of a neural network filter. [Figure 6] FIG. 1 is a schematic diagram of an example of a single-branch neural network filter. [Figure 7A] 7A to 7C are schematic diagrams of existing residual blocks. [Figure 7B] 7A to 7C are schematic diagrams of existing residual blocks. [Figure 7C] 7A to 7C are schematic diagrams of existing residual blocks. [Figure 8] FIG. 1 is a schematic diagram of an example of a double-branch neural network filter. [Figure 9]1 is a flowchart of a neural network-based image filtering method according to an embodiment of the present invention. [Figure 10] FIG. 1 is a schematic diagram of CP decomposition. [Figure 11A] 11A to 12B are schematic diagrams of the network after processing the first or last convolutional layer in the neural network filter. [Figure 11B] 11A to 12B are schematic diagrams of the network after processing the first or last convolutional layer in the neural network filter. [Figure 12A] 11A to 12B are schematic diagrams of the network after processing the first or last convolutional layer in the neural network filter. [Figure 12B] 11A to 12B are schematic diagrams of the network after processing the first or last convolutional layer in the neural network filter. [Figure 13] FIG. 2 is a schematic diagram of a neural network filter according to an embodiment of the present invention. [Figure 14] FIG. 10 is a schematic diagram of another neural network filter according to an embodiment of the present invention. [Figure 15A] 15A to 15E are schematic diagrams of several network structures of the i-th residual block. [Figure 15B] 15A to 15E are schematic diagrams of several network structures of the i-th residual block. [Figure 15C] 15A to 15E are schematic diagrams of several network structures of the i-th residual block. [Figure 15D] 15A to 15E are schematic diagrams of several network structures of the i-th residual block. [Figure 15E] 15A to 15E are schematic diagrams of several network structures of the i-th residual block. [Figure 16A] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 16B]16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 16C] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 16D] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17A] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17B] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17C] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17D] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17E] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17F] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 17G] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 18A] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 18B] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 18C] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 18D] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 19A] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 19B] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 19C] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 19D] 16A to 19D are schematic diagrams of several network structures of the i-th residual block. [Figure 20] FIG. 2 is a schematic diagram of an example of the structure of the i-th residual block. [Figure 21A] 21A to 21D are schematic diagrams of several network structures of the i-th residual block. [Figure 21B] 21A to 21D are schematic diagrams of several network structures of the i-th residual block. [Figure 21C] 21A to 21D are schematic diagrams of several network structures of the i-th residual block. [Figure 21D] 21A to 21D are schematic diagrams of several network structures of the i-th residual block. [Figure 22A] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 22B] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 22C] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 23A] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 23B] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 23C] 22A to 23C are schematic diagrams of several network structures of neural network filters. [Figure 24] 1 is a flowchart of an image decoding method according to an embodiment of the present invention. [Figure 25] 2 is a flowchart of an image filtering method according to an embodiment of the present invention. [Figure 26] 1 is a schematic block diagram of a neural network-based image filtering device according to an embodiment of the present invention; [Figure 27] 1 is a schematic block diagram of an image decoding device according to an embodiment of the present invention. [Figure 28] 1 is a schematic block diagram of an image encoding device according to an embodiment of the present invention; [Figure 29] 1 is a schematic block diagram of an electronic device according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE INVENTION

[0020] The following clearly and completely describes the technical solutions in the embodiments of the present invention with reference to the accompanying drawings in the embodiments of the present invention. It is clear that the described embodiments are only a part of the embodiments of the present invention, and are not all of the embodiments. All other embodiments that can be obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of the present invention.

[0021] In the specification, claims, and drawings of this application, terms such as “first,” “second,” etc. are used to distinguish between similar objects and are not necessarily used to describe a particular order or sequence. Data used in this manner may be interchangeable, as appropriate, so that embodiments of the present invention may be implemented in orders other than those illustrated or described herein. In embodiments of the present invention, “B corresponding to A” means that B is related to A. In one aspect, B may be determined based on A. Determining B based on A does not simply mean determining B based on A, but may also determine B based on A and / or other information. The terms “comprise” and “have,” as well as any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or server comprising a series of steps or units is not necessarily limited to the explicitly listed steps or units, and may include steps or units not explicitly listed or other steps or units inherent in the process, method, product, or device. In the description of the present invention, unless otherwise specified, “plurality” means two or more.

[0022] The present invention can be applied to image encoding / decoding, video encoding / decoding, hardware video encoding / decoding, circuit-specific video encoding / decoding, real-time video encoding / decoding, and the like. For example, aspects of the present invention can be incorporated into a deep learning-based end-to-end image encoding standard, such as JPEG AI. Alternatively, aspects of the present invention can operate in combination with other proprietary or industry standards, including ITU-TH.261, ISO / IEC MPEG-1 Visual, ITU-TH.262 or ISO / IEC MPEG-2 Visual, ITU-TH.263, ISO / IEC MPEG-4 Visual, and ITU-TH.264 (also known as ISO / IEC MPEG-4 AVC), including scalable video encoding / decoding (SVC) and multiview video encoding / decoding (MVC) extensions. Note that the technology of the present invention is not limited to any particular encoding / decoding standard or technology.

[0023] For ease of understanding, a video encoding / decoding system according to an embodiment of the present invention will first be introduced with reference to FIG.

[0024] FIG. 1 is a schematic block diagram of a video encoding / decoding system according to an embodiment of the present invention. Note that FIG. 1 is merely an example, and video encoding / decoding systems according to embodiments of the present invention include, but are not limited to, the one shown in FIG. 1. As shown in FIG. 1, a video encoding / decoding system 100 includes an encoding device 110 and a decoding device 120. Here, the encoding device encodes (which can be understood as compressing) video data to generate a bitstream and transmits the bitstream to a decoding device. The decoding device decodes the bitstream generated by the encoding device through encoding, and obtains decoded video data.

[0025] In an embodiment of the present invention, the encoding device 110 can be understood as a device having a video encoding function, and the decoding device 120 can be understood as a device having a video decoding function. That is, the encoding device 110 and the decoding device 120 according to an embodiment of the present invention include various devices, such as smartphones, desktop computers, mobile computing devices, notebook (e.g., laptop) computers, tablet computers, set-top boxes, televisions, cameras, display devices, digital media players, video game consoles, and in-vehicle computers.

[0026] In some embodiments, encoding device 110 may transmit encoded video data (e.g., a bitstream) to decoding device 120 over channel 130. Channel 130 may include one or more media and / or devices capable of transmitting encoded video data from encoding device 110 to decoding device 120.

[0027] In one example, channel 130 includes one or more communication media that enable encoding device 110 to transmit encoded video data directly to decoding device 120 in real time. In this example, encoding device 110 can modulate the encoded video data in accordance with a communication standard and transmit the modulated video data to decoding device 120. Here, the communication medium includes, for example, a wireless communication medium such as a radio frequency spectrum, and preferably, the communication medium may also include a wired communication medium such as, for example, one or more physical transmission lines.

[0028] In another example, channel 130 includes a storage medium capable of storing video data encoded by encoding device 110. The storage medium may include various locally accessible data storage media, such as optical disks, DVDs, flash memory, etc. In this example, decoding device 120 may obtain the encoded video data from the storage medium.

[0029] In another example, channel 130 may include a storage server that can store video data encoded by encoding device 110. In this example, decoding device 120 may download the stored encoded video data from the storage server. Preferably, the storage server stores the encoded video data and may transmit the encoded video data to decoding device 120, and may be, for example, a web server (e.g., for a website), a file transfer protocol (FTP) server, etc.

[0030] In some embodiments, encoding device 110 includes a video encoder 112 and an output interface 113, where output interface 113 may include a modulator / demodulator (modem) and / or a transmitter.

[0031] In some embodiments, encoding device 110 may further include a video source 111 in addition to video encoder 112 and output interface 113 .

[0032] Video source 111 may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface, and a computer graphics system, where the video input interface is used to receive video data from a video content provider and the computer graphics system is used to generate the video data.

[0033] The video encoder 112 encodes video data from the video source 111 to generate a bitstream. The video data may include one or more pictures or a sequence of pictures. The bitstream includes coding information for a picture or a sequence of pictures in the form of a bitstream. The coding information may include coded image data and associated data. The associated data may include a sequence parameter set (abbreviated as SPS), a picture parameter set (abbreviated as PPS), and other syntax structures. An SPS may include parameters that apply to one or more sequences. A PPS may include parameters that apply to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in the bitstream.

[0034] The video encoder 112 transmits the encoded video data directly to the decoding device 120 via the output interface 113. The encoded video data may be stored on a storage medium or a storage server for later reading by the decoding device 120.

[0035] In some embodiments, decoding device 120 includes an input interface 121 and a video decoder 122 .

[0036] In some embodiments, decoding device 120 may further include a display device 123 in addition to input interface 121 and video decoder 122 .

[0037] Here, the input interface 121 includes a receiver and / or a modem, and can receive encoded video data via a channel 130.

[0038] The video decoder 122 decodes the encoded video data to obtain decoded video data, and transmits the decoded video data to the display device 123 .

[0039] Display device 123 displays the decoded video data and may be integrated into decoding device 120 or may be external to decoding device 120. Display device 123 may include a variety of display devices, such as a liquid crystal display (LCD), a plasma display, an organic light-emitting diode (OLED) display, or other types of display devices.

[0040] Also, FIG. 1 is merely an example, and the techniques of the embodiments of the present invention are not limited to FIG. 1, for example, the techniques of the present invention may be applied to one-side video encoding or other-side video decoding.

[0041] The following introduces a video coding framework according to an embodiment of the present invention.

[0042] 2 is a schematic diagram of an encoding framework according to an embodiment of the present invention. Note that a video encoder 200 may be used to perform lossy or lossless compression of images. The lossless compression may be visually lossless or mathematically lossless compression.

[0043] The video encoder 200 can be applied to image data in a luminance-chrominance (YCbCr, YUV) format. For example, the YUV ratio may be 4:2:0, 4:2:2, or 4:4:4. Y represents luminance (Luma), Cb (U) represents blue, and Cr (V) represents red, and U and V represent chroma, or color and saturation. For example, in color format, 4:2:0 represents four luminance components and two chrominance components (YYYYCbCr) for every four pixels, 4:2:2 represents four luminance components and four chrominance components (YYYYCbCrCbCr) for every four pixels, and 4:4:4 represents full pixel display (YYYYCbCrCbCrCbCrCbCr).

[0044] For example, the video encoder 200 reads video data and divides each frame image in the video data into several coding tree units (CTUs). In some examples, a CTU may be referred to as a "tree block," a "largest coding unit (LCU)," or a "coding tree block (CTB)." Each CTU may be associated with a pixel block having an equal size in the image. Each pixel may correspond to one luminance (luma) sample and two chrominance (chroma) samples. Thus, each CTU may be associated with one luminance sampling block and two chrominance sampling blocks. The size of the CTU may be, for example, 128×128, 64×64, 32×32, etc. The CTU may be further divided into multiple coding units (CUs) for encoding, and the CU may be a rectangular or square block. A CU may be further divided into a prediction unit (abbreviated as PU) and a transform unit (abbreviated as TU) so that separation of coding, prediction, and transformation can be more flexibly processed. In one example, a CTU is divided into CUs in a quadtree manner, and a CU is divided into TUs and PUs in a quadtree manner.

[0045] The video encoder and decoder may support various PU sizes. Assuming a specific CPU size is 2N×2N, the video encoder and decoder may support a PU size of 2N×2N or N×N for intra-frame prediction, and may support symmetric PUs of 2N×2N, 2N×N, N×2N, N×N, or similar sizes for inter-frame prediction. The video encoder and decoder may further support asymmetric PUs of 2N×nU, 2N×nD, nL×2N, and nR×2N for inter-frame prediction.

[0046] 2, video encoder 200 may include a prediction unit 210, a convolution unit 220, a transform / quantization unit 230, an inverse transform / quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image buffer 270, and an entropy coding unit 280. Note that video encoder 200 may include more, fewer, or different functional components.

[0047] Preferably, in the present invention, the current block may be called a current coding unit (CU) or a current prediction unit (PU), etc. The prediction block may be called a predicted image block or an image prediction block, and the reconstructed image block may be called a reconstruction block or an image reconstruction image block.

[0048] In some embodiments, the prediction unit 210 includes an inter-frame prediction unit 211 and an intra-frame prediction unit 212. Because there is a strong correlation between adjacent pixels within a frame of a video, the intra-frame prediction method is used in video encoding / decoding techniques to remove spatial redundancy between adjacent pixels. Because there is a strong similarity between adjacent frames of a video, the inter-frame prediction method is used in video encoding / decoding techniques to remove temporal redundancy between adjacent frames and improve coding efficiency.

[0049] The inter-frame prediction unit 211 may be used for inter-frame prediction, which may include motion estimation and motion compensation. Motion estimation may search a reference image in a list of reference images to find a reference block for the image block to be coded. Motion estimation may generate an index indicating the reference block and a motion vector indicating a spatial shift between the image block to be coded and the reference block. Motion estimation may output the index and motion vector of the reference block as motion information for the image block to be coded. Motion compensation may obtain prediction information for the image block to be coded based on the motion information of the image block to be coded. Inter-frame prediction may refer to image information of different frames, and inter-frame prediction uses the motion information to find a reference block from the reference frame and generate a prediction block from the reference block to eliminate temporal redundancy. The frames used for inter-frame prediction may be P frames and / or B frames, where P frames refer to forward-predicted frames and B frames refer to bidirectionally predicted frames. Inter-frame prediction uses the motion information to find a reference block from the reference frame and generate a prediction block from the reference block. The motion information includes a list of reference frames where the reference frames exist, reference frame indexes, and motion vectors. The motion vectors may be for all pixels or for each pixel. If the motion vectors are for each pixel, interpolation filtering must be used to create a desired block for each pixel in the reference frame, and the block for each pixel or for each pixel in the reference frame found based on the motion vector is called a reference block. Some techniques use the reference block as a prediction block, while other techniques generate a prediction block by reprocessing from the reference block. Generating a prediction block by reprocessing from the reference block can be understood as using the reference block as a prediction block and reprocessing from the prediction block to generate a new prediction block.

[0050] The intra-frame prediction unit 212 is used to predict pixel information in the image block currently being coded by only referring to information in the same frame image, and to remove spatial redundancy. The frame used for intra-frame prediction may be an I-frame.

[0051] There are several prediction modes for intra-frame prediction. Taking the international digital video coding standard H series as an example, the H.264 / AVC standard has eight angular prediction modes and one non-angular prediction mode, while H.265 / HEVC has been expanded to 33 angular prediction modes and two non-angular prediction modes. The intra-frame prediction modes used in HEVC are planar, DC, and 33 angular modes, for a total of 35 prediction modes. The intra-frame prediction modes used in VVC are planar, DC, and 65 angular modes, for a total of 67 prediction modes.

[0052] Furthermore, as the number of angle modes increases, the intra-frame prediction becomes more accurate, meeting the demands of the development of high-definition and ultra-high-definition digital video.

[0053] Convolution unit 220 may generate a residual block for the CPU based on the pixel block of the CPU and the prediction block of the PU of the CPU. For example, convolution unit 220 may generate a residual block for the CU such that each sample in the residual block has a value equal to the difference between a sample in the pixel block of the CU and a corresponding sample in the prediction block of the PU of the CU.

[0054] The transform / quantization unit 230 may quantize transform coefficients. The transform / quantization unit 230 may quantize transform coefficients associated with a TU of a CU based on a quantization parameter (QP) value associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with a CU by adjusting the QP value associated with the CU. For example, a residual video signal is transformed into a transform domain, called a transform coefficient, by a transform operation such as a DFT or DCT. The signal in the transform domain is further subjected to a lossy quantization operation, and some upper portions are lost, so that the quantized signal can be easily represented in a compressed form. In some video coding standards, multiple transforms are selectable, and therefore the encoding side must select one of the transforms for the current encoding CU and inform the decoding side. The quantization fineness is generally determined by the quantization parameter (QP). A larger QP value results in coefficients representing a wider range of values ​​being quantized to the same output, generally resulting in greater distortion and a lower code rate. Conversely, smaller values ​​of QP result in coefficients representing a smaller range of values ​​being quantized to the same output, and therefore generally result in less distortion and correspond to a larger code rate.

[0055] The inverse transform / quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients and reconstruct residual blocks from the quantized transform coefficients.

[0056] Reconstruction unit 250 may add samplings of the reconstructed residual blocks to corresponding samplings of one or more prediction blocks generated by prediction unit 210 to generate a reconstructed image block associated with the TU. By reconstructing the sampling blocks of each TU of a CU in this manner, video encoder 200 may reconstruct the pixel blocks of the CU.

[0057] The loop filtering unit 260 processes the inversely transformed and inversely quantized pixels to correct distortion information and provide a better reference for subsequent coding pixels. For example, it can perform deblocking filtering to reduce the blocking effect of pixel blocks associated with a CU. As described above, a coded image can undergo inverse quantization, inverse transform, and predictive compensation to obtain a reconstructed decoded image. Compared with the original image, the reconstructed image has some information that differs from the original image due to the effects of quantization, resulting in distortion. Filtering the reconstructed image, such as a deblocking filter (DBF), sample adaptive offset (SAO), or adaptive loop filter (ALF), can effectively reduce the degree of distortion caused by quantization. Because such filtered reconstructed image is used as a reference for subsequent coding images to predict future signals, the above-mentioned filtering process is also referred to as loop filtering and filtering within the coding loop.

[0058] The decoded image buffer 270 can store the reconstructed pixel blocks. The inter-frame prediction unit 211 can perform intra-frame prediction on PUs of other images using a reference image including the reconstructed pixel blocks. Also, the intra-frame prediction unit 212 can perform intra-frame prediction on other PUs in the same image as the CU using the reconstructed pixel blocks in the decoded image buffer 270.

[0059] The entropy coding unit 280 may receive the quantized transform coefficients from the transform / quantization unit 230. The entropy coding unit 280 may perform one or more entropy coding operations on the quantized transform coefficients to generate entropy-coded data. For example, the entropy coding unit 280 may perform statistical compression coding on the quantized transform domain signal based on the frequency of occurrence of each value, and finally output a binarized (0 or 1) compressed bitstream. At the same time, other information may be generated by coding, such as a selected mode, motion vector, etc., and entropy coding may need to be performed to reduce the bitrate. For example, statistical coding is a lossless coding method that can effectively reduce the bitrate required to represent the same signal. Common statistical coding methods include variable length coding (VLC) and content adaptive binary arithmetic coding (CABAC).

[0060] FIG. 3 is a schematic diagram of a decoding framework according to an embodiment of the present invention.

[0061] 3, the video decoder 300 includes an entropy decoding unit 310, a prediction unit 320, an inverse quantization / transform unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image buffer 360. Note that the video decoder 300 may include more, fewer, or different functional components.

[0062] The video decoder 300 may receive a bitstream. The entropy decoding unit 310 may parse the bitstream and extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization / transform unit 330, the reconstruction unit 340, and the loop filtering unit 350 may decode video data based on the syntax elements extracted from the bitstream, i.e., generate decoded video data.

[0063] In some embodiments, the prediction unit 320 includes an intra-frame prediction unit 322 and an inter-frame prediction unit 321 .

[0064] The intra prediction unit 322 may perform intra prediction to generate a predictive block of the PU. The intra prediction unit 322 may use an intra prediction mode to generate a predictive block of the PU based on pixel blocks of spatially neighboring PUs. The intra prediction unit 322 may also determine the intra prediction mode of the PU based on one or more syntax elements parsed from the bitstream.

[0065] The inter-frame prediction unit 321 may construct a first reference image list (list 0) and a second reference image list (list 1) based on syntax elements parsed from the bitstream. In addition, if the PU uses inter-frame predictive coding, the entropy decoding unit 310 may analyze motion information of the PU. The inter-frame prediction unit 321 may determine one or more reference blocks for the PU based on the motion information of the PU. The inter-frame prediction unit 321 may generate a predictive block for the PU based on the one or more reference blocks of the PU.

[0066] The inverse quantization / transform unit 330 may inverse quantize (i.e., dequantize) the transform coefficients associated with the TU. The inverse quantization / transform unit 330 may use a QP value associated with the CU of the TU to determine the level of quantization.

[0067] Following the inverse quantization transform coefficients, the inverse quantization / transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients to generate a residual block associated with the TU.

[0068] The reconstruction unit 340 reconstructs the pixel block of the CU using the residual block associated with the TU of the CU and the prediction block of the PU of the CU. For example, the reconstruction unit 340 can add the sampling of the residual block to the corresponding sampling of the prediction block to reconstruct the pixel block of the CU and obtain a reconstructed image block.

[0069] The loop filtering unit 350 may perform a deblock filtering operation to reduce the blocking effect of pixel blocks associated with a CU.

[0070] Video decoder 300 may store the reconstructed image of the CU in decoded image buffer 360. Video decoder 300 may use the reconstructed image in decoded image buffer 360 as a reference image for subsequent prediction, or may input the reconstructed image to a display device for display.

[0071] The basic flow of video encoding / decoding is as follows: On the encoding side, a frame image is divided into blocks, and for a current block, a prediction unit 210 generates a predicted block of the current block using intraframe prediction or interframe prediction. Based on the predicted block and the original block of the current block, a convolution unit 220 can calculate a residual block, i.e., the difference between the predicted block and the original block of the current block. The residual block is also referred to as residual information. The residual block can be subjected to processes such as transformation and quantization by the transform / quantization unit 230 to remove information imperceptible to the human eye to eliminate visual redundancy. Preferably, the residual block before transformation and quantization by the transform / quantization unit 230 may be referred to as a time-domain residual block, and the time-domain residual block after transformation and quantization by the transform / quantization unit 230 may be referred to as a frequency residual block or a frequency-domain residual block. The entropy coding unit 280 may receive the quantized transformation coefficients output from the transform / quantization unit 230, entropy code the quantized transformation coefficients, and output a bitstream. For example, the entropy coding unit 280 may remove character redundancy based on a target context model and probability information of the binary bitstream.

[0072] On the decoding side, the entropy decoding unit 310 analyzes the bitstream to obtain prediction information, a quantization coefficient matrix, etc., for the current block. The prediction unit 320 can generate a prediction block for the current block using intraframe prediction or interframe prediction based on the prediction information. The inverse quantization / transform unit 330 uses the quantization coefficient matrix obtained from the bitstream to inverse quantize and inverse transform the quantization coefficient matrix to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed block forms a reconstructed image. The loop filtering unit 350 loop filters the reconstructed image based on the image or block to obtain a decoded image. Obtaining a decoded image requires similar operations on both the encoding side and the decoding side. The decoded image is also called a reconstructed image, and can be used as a reference frame for interframe prediction for subsequent frames.

[0073] The block division information determined on the encoding side, and mode or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering are stored in the bitstream as necessary. The decoding side analyzes the codestream and determines the same block division information, mode or parameter information such as prediction, transformation, quantization, entropy coding, and loop filtering as the encoding side based on existing information, thereby ensuring that the decoded image obtained on the encoding side is the same as the decoded image obtained on the decoding side.

[0074] The above is the basic flow of video encoding / decoding in a block-based mixed coding framework. As technology develops, some modules or steps of the framework or flow may be optimized. The present invention applies to the basic flow of video encoding / decoding in a block-based mixed coding framework, but is not limited to this framework and flow.

[0075] In existing mixed coding frameworks, each frame image in a video is often divided into units of a certain size before the subsequent encoding / decoding flow is performed. As shown in Figure 4, the maximal coding unit (CTU) is the basic coding unit in the mixed coding framework and usually contains two parts: luma Y and chroma UV. Because the U and V components of chroma have similar characteristics, they are usually processed sequentially in the order of U and V using the same coding parameters, and the coding results of U and V are obtained accordingly.

[0076] Existing hybrid coding frameworks use conventional loop filtering to suppress distortion in the reconstructed image, improve the quality of the reconstructed image, and restore the encoded reconstructed image to the original image. However, conventional loop filtering is manually designed, making it difficult to effectively reduce distortion in the reconstructed image and leaving significant room for optimization. Due to the excellent performance of deep learning tools in image processing, deep learning-based loop filtering has been applied to the loop filtering module.

[0077] The key technology in this invention is a neural network loop filter (NNLF). As shown in Figure 5, the unfiltered image is input to the trained filter to obtain the filtered enhanced image.

[0078] During the training process, neural networks typically use a loss function to constrain the filtered image to reconstruct the original image as closely as possible. The loss function evaluates the difference between the filtered value and the correct value, with a larger loss value indicating a larger difference. The goal of training is to reduce the loss. Deep learning-based coding tools typically use the L1 norm loss function, L2 norm loss function, and smooth L1 loss function as loss functions.

[0079] Currently commonly used neural network filters include convolutional neural network (CNN) filters. Convolution is a feedforward neural network that includes convolutional calculations and has a depth structure, and is one of the representative algorithms of deep learning. A convolutional neural network is mainly composed of a series of basic convolutional layers or convolutional modules built on convolutional layers. Figure 6 shows a specific example of a network structure model. Here, the network includes a head, a body, and a tail, and the body is composed of at least one residual block (ResBlock / RB).

[0080] The residual block ResBlock is a typical convolution module used in various neural network models, and its general structure is shown in FIGS. 7A to 7C.

[0081] The specific usage of ResBlock is as follows:

[0082] Method 1: The input is image-level information. The input of ResBlock may be one or more of image information such as reconstructed image (rec), predicted image (rec), edge information such as boundary strength (BS), frame quantization parameter (sliceQP), base quantization parameter (baseQP), frame type (sliceType / IPB). The output is a feature-level image with K channels (K is a positive integer. If K=1, the output is image-level information).

[0083] Method 2: Input is feature-level information. The input of ResBlock is feature information with K channels extracted by the previous convolutional layer / convolutional block. In this case, the output of ResBlock is feature-level information with M channels (K and M are both positive integers, and when M=1, the output is image-level information).

[0084] Also, different neural network models can use ResBlock to replace any convolutional layer or convolutional block in the model. For example, in the network configuration shown in Figure 8, which is different from Figure 6, the network processes luma and chroma components separately, but can still use ResBlock in the structure.

[0085] As shown in Figure 6 or Figure 8, existing neural network filters typically build their overall network structure based on common basic modules such as 3x3 convolution and residual blocks to train the neural network filter. In the network structure of a neural network filter, repeatedly stacking 3x3 convolution and residual blocks can achieve significant filtering performance, but introduces excessive computational complexity. On the other hand, limiting the number of iterations of 3x3 convolution and residual blocks, i.e., specifying a certain computational complexity, makes it difficult to achieve high filtering performance. Therefore, how to achieve high image filtering performance without introducing excessive computational complexity is currently an important research direction in neural network-based image filter technology.

[0086] Therefore, to solve this technical problem, an embodiment of the present invention provides a new neural network filter, which includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method. For example, in an embodiment of the present invention, a standard convolution layer in an existing neural network filter, such as a standard convolution layer (e.g., a 3x3 convolution layer) in the head, main, or tail section of a neural network filter shown in Figure 6 or Figure 8, is processed using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method to convert it into a convolution operation with low computational complexity, thereby reducing the complexity of the filtering operation of the neural network filter and improving the image filtering effect without degrading the filtering performance of the neural network filter.

[0087] The following describes in detail the technology of the embodiments of the present invention with reference to several embodiments, which can be combined with each other, and the description of the same or similar concepts or processes in several embodiments will be omitted.

[0088] First, an image decoding method according to an embodiment of the present invention will be described, taking the decoding side as an example.

[0089] 9 is a flowchart of a neural network-based image filtering method according to an embodiment of the present invention. The embodiment of the present invention may be applied to the decoder or decoding device shown in FIG. 1, 2, or 3, and the encoder or encoding device, or may be applied to other electronic devices. For convenience of explanation, the following description will be given using a filtering device as an example. The filtering device can be understood as any electronic device having an image filtering function, such as the above-mentioned decoding device or encoding device.

[0090] As shown in FIG. 9, the method according to the embodiment of the present invention includes the following steps.

[0091] Step S101: The filtering device determines input information for the neural network filter based on the image to be filtered.

[0092] The input information includes the image to be filtered.

[0093] The image to be filtered according to an embodiment of the present invention may be any image that needs to be filtered, for example a reconstructed image or any other image to be filtered.

[0094] In one example, the image to be filtered may be an image, for example, a frame of reconstructed image or an entire reconstructed image.

[0095] In one example, the image to be filtered may be a local region within one image, for example, a reconstructed region that is part of one reconstructed image.

[0096] The embodiment of the present invention is not limited to the specific content of the input information of the neural network filter, as long as it is guaranteed that the input information includes the image to be filtered.

[0097] For example, in some embodiments, input information according to embodiments of the present invention includes only the image to be filtered and no other information.

[0098] In some embodiments, the input information according to the embodiments of the present invention further includes, in addition to the image to be filtered, other information related to the image to be filtered. For example, if the image to be filtered is a reconstructed image, the input information further includes, in addition to the reconstructed image, at least one of a predicted image corresponding to the reconstructed image, a boundary strength, a frame type (slice type) of the reconstructed image, and quantization parameter information. Preferably, the quantization parameter information includes a slice level quantization parameter (Slice QP) and a base QP (Base Quantization Parameter).

[0099] In some embodiments, the input information may further include information about other images to be filtered, and embodiments of the present invention are not limited thereto.

[0100] After determining the input information of the neural network filter based on the above steps, the filtering device executes the following step S102.

[0101] Step S102: The filtering device performs filtering processing on the input information using a neural network filter, and obtains a filtered image of the filtering target image.

[0102] Here, the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0103] Although existing neural network filters can provide considerable filtering performance by repeatedly stacking 3x3 convolutions and residual blocks, they introduce excessive computational complexity, making it impossible to achieve a balance between filtering performance and computational complexity. To solve this technical problem, an embodiment of the present invention provides a new neural network filter, which includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method. For example, in an embodiment of the present invention, a standard convolution layer in an existing neural network filter, such as a standard convolution layer (e.g., a 3x3 convolution layer) in the head, main body, or tail of the neural network filter shown in Figure 6 or Figure 8, is processed using at least two of the multi-level receptive field method, the tensor decomposition method, and the group convolution method to convert it into a convolution operation with low computational complexity, thereby reducing the complexity of the filtering operation of the neural network filter and improving the image filtering effect without degrading the filtering performance of the neural network filter.

[0104] Below we introduce related techniques such as multi-level receptive fields, tensor decomposition, and group convolution.

[0105] Multi-Level Receptive Field (MLRF): In a convolutional neural network, a single element in the output feature map of each network layer is mapped to the region size of the input feature map, i.e., the receptive field. Existing NNLFs generally use standard 1x1 convolutional layers or 3x3 convolutional layers as the basic building blocks of the network structure. Different convolutional window sizes result in different numbers of elements in the input feature map being processed, and different convolutional layers typically have different receptive field sizes. To effectively extract and fuse input feature information, 1x1 convolutional layers and 3x3 convolutional layers can be used in parallel to obtain feature information at different receptive field levels, thereby improving the generalization of neural network models.

[0106] Canonical Polyadic Decomposition (CPD): Also known as CP decomposition, this is a classical tensor decomposition method used to reduce the amount of convolution calculations in neural networks. As shown in Figure 10, the principle of CP decomposition allows a 3x3 convolution to be approximately expressed as a 1x3 convolution and a 3x1 convolution, effectively reducing the complexity of the convolution calculation without significantly degrading network performance.

[0107] Group Convolution (GC): A variation of standard convolution, it divides the number of input convolution channels into G groups and performs standard convolution processing on the features of each group. Since the number of input channels in each group is 1 / G of the original, the number of channels of the corresponding required convolution kernel is also 1 / G of the original. In this way, group convolution reduces the number of parameters and the amount of calculation to 1 / G of the conventional number, effectively reducing the complexity of the convolution calculation.

[0108] The neural network filter according to the embodiment of the present invention may be understood as processing at least one standard convolution layer of an existing neural network filter using at least two of the multi-level receptive field method, the tensor decomposition method, and the group convolution method to obtain any neural network filter with multiple convolution layers with low computational complexity.

[0109] In some embodiments, embodiments of the present invention may convert at least one standard convolutional layer at the beginning of the neural network filter shown in FIG. 6 into multiple convolutional layers with lower complexity by processing it using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0110] In one example, the 3x3 convolution layer at the beginning of the neural network filter shown in Figure 6 is decomposed into a 1x3 convolution layer and a 3x1 convolution layer using a tensor decomposition method. Next, as shown in Figure 11A, the input feature information of the 1x3 convolution layer is divided into multiple groups using a group convolution method, and the feature information of each group is convolved, followed by merging the number of channels to obtain the output information of the 1x3 convolution layer. Preferably, as shown in Figure 11A, the input feature information of the 3x1 convolution layer may be divided into multiple groups using the group convolution method, and the feature information of each group is convolved, followed by merging the number of channels to obtain the output information of the 3x1 convolution layer.

[0111] In one example, the 3x3 convolutional layer at the beginning of the neural network filter shown in Figure 6 is converted into two parallel-connected 1x1 convolutional layers and a 3x3 convolutional layer using a multilevel receptive field method. Alternatively, the 3x3 convolutional layer in the parallel branch is decomposed into a 1x3 convolutional layer and a 3x1 convolutional layer using a tensor decomposition method. Next, as shown in Figure 11B, the input feature information of the 1x3 convolutional layer may be divided into multiple groups using a group convolutional method, and the feature information of each group may be convolved, followed by channel number merging, to obtain output information of the 1x3 convolutional layer. Preferably, as shown in Figure 11B, the input feature information of the 3x1 convolutional layer may be divided into multiple groups using a group convolutional method, and the feature information of each group may be convolved, followed by channel number merging, to obtain output information of the 3x1 convolutional layer.

[0112] In some embodiments, embodiments of the present invention may convert at least one standard convolutional layer at the end of the neural network filter shown in FIG. 6 into multiple convolutional layers with lower complexity by processing it using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

[0113] In one example, the 3x3 convolution layer at the end of the neural network filter shown in Figure 6 may be decomposed into a 1x3 convolution layer and a 3x1 convolution layer using a tensor decomposition method. Next, as shown in Figure 12A, the input feature information of the 1x3 convolution layer may be divided into multiple groups using a group convolution method, and the feature information of each group may be convolved, followed by channel number merging, to obtain output information of the 1x3 convolution layer. Preferably, as shown in Figure 12A, the input feature information of the 3x1 convolution layer may be divided into multiple groups using a group convolution method, and the feature information of each group may be convolved, followed by channel number merging, to obtain output information of the 3x1 convolution layer.

[0114] In one example, the 3x3 convolutional layer at the end of the neural network filter shown in Figure 6 may be converted into two parallel-connected 1x1 convolutional layers and a 3x3 convolutional layer using a multilevel receptive field method. Next, the 3x3 convolutional layer in the parallel branch is decomposed into a 1x3 convolutional layer and a 3x1 convolutional layer using a tensor decomposition method. Next, as shown in Figure 12B, the input feature information of the 1x3 convolutional layer may be divided into multiple groups using a group convolution method, and the feature information of each group may be convolved, followed by merging the number of channels to obtain the output information of the 1x3 convolutional layer. Next, as shown in Figure 12B, the input feature information of the 3x1 convolutional layer may be divided into multiple groups using a group convolution method, and the feature information of each group may be convolved, followed by merging the number of channels to obtain the output information of the 3x1 convolutional layer.

[0115] In some embodiments, a neural network filter according to an embodiment of the present invention includes a residual unit including a plurality of sequentially connected residual blocks, and at least one of the plurality of residual blocks, e.g., the i-th residual block, includes all or some of the plurality of convolutional layers obtained by processing standard convolutional layers using at least two of the above-mentioned multi-level receptive field method, tensor decomposition method, and group convolution method, where i is a positive integer. In other words, an embodiment of the present invention proposes a new residual block structure. The new residual block structure may be understood as processing at least one standard convolutional layer in an existing residual block using at least two of the above-mentioned multi-level receptive field method, tensor decomposition method, and group convolution method to obtain a plurality of convolutional layers with lower computational complexity.

[0116] The residual unit of the embodiment of the present invention may be understood as the main body in FIG. 6 above.

[0117] The neural network filter according to this embodiment includes a plurality of residual blocks. At least one of the plurality of residual blocks may be understood to be obtained by processing a standard convolutional layer of an existing residual block shown in FIGS. 7A to 7C. For example, the standard convolutional layer of at least one of the plurality of residual blocks included in the main body of the neural network filter shown in FIG. 6 is processed using at least two of the multilevel receptive field method, the tensor decomposition method, and the group convolution method to obtain at least one residual block according to this embodiment. For convenience of explanation, the i-th residual block will be described here as an example.

[0118] For the i-th residual block included in the residual unit of the neural network filter, the network structure of the i-th residual block mainly includes the following two scenarios:

[0119] Scenario 1: The filtering device processes a standard convolution layer based on the multi-level receptive field method and the group convolution method to obtain the i-th residual block.

[0120] Specifically, as shown in FIG. 13 , an i-th residual block according to an embodiment of the present invention includes a first convolution unit, a second convolution unit, and a third convolution unit, where the first convolution unit and the second convolution unit are connected in parallel and then connected in series with the third convolution unit. The first convolution unit includes a first convolution layer, the second convolution unit includes a second convolution layer, and the third convolution unit includes a third convolution layer, where the first convolution layer and the second convolution layer have different convolution kernel sizes, and at least one of the first convolution layer, the second convolution layer, and the third convolution layer includes a grouped convolution layer. The grouped convolutional layer divides the input feature information of the grouped convolutional layer into multiple groups based on the number of channels using a grouped convolution method, performs a convolution operation on each group feature information, and then merges the number of channels to obtain the output information of the grouped convolutional layer.

[0121] The embodiments of the present invention are not limited to the number of convolution layers included in the first convolution unit, the second convolution unit, and the third convolution unit, for example, at least one of the first convolution unit, the second convolution unit, and the third convolution unit includes one or more convolution layers.

[0122] In some embodiments, at least one of the first convolution unit, the second convolution unit, and the third convolution unit may further include an activation function in addition to a convolution layer.

[0123] The activation function according to the embodiment of the present invention may be an activation function such as Sigmoid, ReLU, LeakyReLU, PReLU, or ELU.

[0124] In one example, as shown in FIG. 14, the first convolution unit includes a first convolution layer and a first activation function, the second convolution unit includes a second convolution layer and a second activation function, and the third convolution unit includes a third convolution layer.

[0125] As can be seen from the above-mentioned Figures 13 and 14, the first convolutional layer and the second convolutional layer are connected in parallel, and the first convolutional layer and the second convolutional layer have different convolutional kernel sizes. Therefore, when the parallel first convolutional layer and the second convolutional layer process feature information, feature information at different receptive field levels can be obtained, thereby improving the generalization of the neural network filter.

[0126] For example, the filtering device records the input feature information input to the i-th residual block as the i-1-th feature information, and the size of the convolution kernel of the first convolution layer is 3×3, and the size of the convolution kernel of the second convolution layer is 1×1. Thus, when the i-1-th feature information is input to the i-th residual block shown in FIG. 13 or 14, the first convolution layer in the first convolution unit extracts the i-1-th feature information using a 3×3 convolution window to obtain feature information at the first receptive field level. The second convolution layer in the second convolution unit extracts the i-1-th feature information using a 1×1 convolution window to obtain feature information at the second receptive field level. This allows feature information at different receptive field levels to be obtained. The extracted feature information at different receptive field levels is then fused and input to the third convolution unit for feature extraction again, after which it is fused with the i-1-th feature information to obtain output feature information for the i-th residual block.

[0127] In other words, by adopting a multi-level receptive field method, the residual block in the embodiment of the present invention can obtain feature information at different receptive field levels, thereby improving the feature processing effect of the residual block.

[0128] 13 or 14, at least one of the first, second, and third convolutional layers includes a grouping convolutional layer, which divides input feature information of the grouping convolutional layer into multiple groups based on the number of channels using a grouping convolutional method, performs convolution on the feature information of each group, and then merges the number of channels to obtain the output information of the grouping convolutional layer.

[0129] In one example, as shown in FIG. 15A , the first convolutional layer includes a grouping convolutional layer, and the second and third convolutional layers are not grouping convolutional layers. The input information of the first convolutional layer is the (i-1)th feature information, and the number of channels of the (i-1)th feature information is K. In this case, the filtering device inputs the (i-1)th feature information to the ith residual block. The first convolutional layer of the ith residual block divides the (i-1)th feature information into multiple groups based on the number of channels of the (i-1)th feature information, and performs a standard convolution operation on each of the multiple groups of feature information obtained by grouping to obtain post-convolution feature information for each group. Next, the number of channels of the post-convolution feature information for each group is merged to obtain output feature information for the first convolutional layer.

[0130] For example, if the number of channels of the (i-1)th feature information is 100, the (i-1)th feature information is divided into 10 groups, i.e., feature information of the first group, feature information of the second group, feature information of the third group, ..., feature information of the tenth group, and each group has 10 channels. Next, standard convolution operations are performed on the feature information of the first group, feature information of the second group, feature information of the third group, ..., feature information of the tenth group, respectively, to obtain Feature Information 1 after the convolution operation of the feature information of the first group, Feature Information 2 after the convolution operation of the feature information of the second group, Feature Information 3 after the convolution operation of the feature information of the third group, ..., Feature Information 10 after the convolution operation of the feature information of the tenth group. Next, the number of channels of the above-mentioned Feature Information 1, Feature Information 2, Feature Information 3, ..., Feature Information 10 are merged, i.e., concatenated, to obtain output feature information of the first convolution layer.

[0131] As described above, in the embodiment of the present invention, a convolution operation with K (e.g., 100) channels is divided into G (e.g., 10) convolution operations with K / G (e.g., 10) channels by group convolution, and the computational complexity of the convolution operation with K / G channels is significantly smaller than that of the convolution operation with K channels. In this way, the group convolution can reduce the number of parameters and the computational complexity to 1 / G of the conventional level, effectively reducing the complexity of the convolution operation.

[0132] In one example, as shown in FIG. 15B , the first convolutional layer and the second convolutional layer include grouped convolutional layers, and the third convolutional layer is not a grouped convolutional layer. The input information of the first convolutional layer and the second convolutional layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this case, the filtering device inputs the i-1th feature information to the i-th residual block, and the first convolutional layer of the i-1th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, performs a standard convolution operation on the feature information of each group obtained by dividing the multiple feature information, and obtains post-convolution feature information of the feature information of each group. Next, the number of channels of the post-convolution feature information of each group is merged to obtain output feature information of the first convolutional layer. Similarly, the second convolutional layer of the ith residual block divides the (i-1)th feature information into multiple groups based on the number of channels of the (i-1)th feature information, performs a standard convolution operation on each of the multiple groups obtained by dividing the feature information, obtains convolutional feature information for each group, and then combines the number of channels of the convolutional feature information for each group to obtain output feature information of the second convolutional layer.

[0133] In one example, as shown in FIG. 15C , the first convolutional layer and the third convolutional layer include grouped convolutional layers, and the second convolutional layer is not a grouped convolutional layer. The input information of the first convolutional layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this case, the filtering device inputs the i-1th feature information to the i-th residual block. The first convolutional layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, and performs a standard convolution operation on each group of the resulting multiple groups of feature information to obtain post-convolution feature information for each group. Next, the number of channels of the post-convolution feature information for each group of feature information is combined to obtain output feature information of the first convolutional layer, which is recorded as first feature information. At the same time, the second convolutional layer performs a convolution operation on the i-1th feature information to obtain second feature information. Next, the first feature information and the second feature information are concatenated to obtain third feature information, and the number of channels of the third feature information is K1. The third feature information is input to a third convolutional layer, which divides the third feature information into multiple groups based on the number of channels K1 of the third feature information and performs a standard convolution operation on each group feature information obtained by dividing the multiple group feature information to obtain feature information after the convolution operation for each group feature information. Next, the number of channels of the feature information after the convolution operation for each group feature information is merged to obtain output feature information of the third convolutional layer.

[0134] In one example, as shown in FIG. 15D , the second convolutional layer and the third convolutional layer include grouped convolutional layers, and the first convolutional layer is not a grouped convolutional layer. The input information of the second convolutional layer is the i-1th feature information, and the number of channels of the i-1th feature information is K. In this case, the filtering device inputs the i-1th feature information to the i-th residual block, and the first convolutional layer of the i-th residual block performs a standard convolution operation on the i-1th feature information to obtain first feature information. The second convolutional layer of the i-th residual block divides the i-1th feature information into multiple groups based on the number of channels of the i-1th feature information, and performs a standard convolution operation on each group of feature information obtained by dividing the multiple groups of feature information, thereby obtaining post-convolution feature information for each group. Next, the number of channels of the post-convolution feature information for each group of feature information is merged to obtain output feature information of the second convolutional layer, which is recorded as second feature information. Next, the first feature information and the second feature information are concatenated to obtain third feature information, and the number of channels of the third feature information is K1. The third feature information is input to a third convolutional layer, which divides the third feature information into multiple groups based on the number of channels K1 of the third feature information and performs a standard convolution operation on each of the multiple group feature information obtained by dividing them to obtain feature information after the convolution operation for each group feature information. Next, the number of channels of the feature information after the convolution operation for each group feature information is merged to obtain output feature information of the third convolutional layer.

[0135] In one example, as shown in FIG. 15E , the first convolutional layer, the second convolutional layer, and the third convolutional layer include grouped convolutional layers. The input information of the first convolutional layer and the second convolutional layer is the (i-1)th feature information, and the number of channels of the (i-1)th feature information is K. In this case, the filtering device inputs the (i-1)th feature information to the ith residual block, and the first convolutional layer of the ith residual block divides the (i-1)th feature information into multiple groups based on the number of channels of the (i-1)th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by dividing the feature information into multiple groups, obtains feature information after the convolution operation of the feature information of each group, and then merges the number of channels to obtain first feature information output from the first convolutional layer. The second convolutional layer of the i-th residual block divides the i-1-th feature information into multiple groups based on the number of channels of the i-1-th feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by dividing, obtains post-convolution feature information for each group, and then merges the number of channels to obtain second feature information output from the second convolutional layer. The first feature information and the second feature information are concatenated to obtain third feature information, and the number of channels of the third feature information is set to K1. The third feature information is input to the third convolutional layer, which divides the third feature information into multiple groups based on the number of channels K1 of the third feature information, performs a standard convolution operation on each of the multiple groups of feature information obtained by dividing, obtains post-convolution feature information for each group, and then merges the number of channels to obtain output feature information from the third convolutional layer.

[0136] The above has described, with reference to Figures 15A to 15E, the network structure of the i-th residual block when at least one of the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a grouping convolutional layer.

[0137] In some embodiments, some of the first, second, and third convolutional layers include grouped convolutional layers, and some of the convolutional layers include sub-convolutional layers obtained by CP decomposition. For example, some of the first, second, and third convolutional layers include grouped convolutional layers, and at least one of the remaining convolutional layers includes a sub-convolutional layer obtained by performing tensor decomposition on a convolutional layer whose convolution kernel size is m×n, where m and n are both positive integers greater than 2.

[0138] In one example, the first convolutional layer and / or the second convolutional layer includes a grouped convolutional layer, and the third convolutional layer includes two subconvolutional layers obtained by a tensor decomposition method. As shown in FIG. 16A , the first convolutional layer is a block convolutional layer with a convolutional kernel size of 3×3, the second convolutional layer is a convolutional layer with a convolutional kernel size of 1×1, and the third convolutional layer includes a 1×3 subconvolutional layer and a 3×1 subconvolutional layer. In this way, by decomposing one 3×3 convolutional operation into one simple 1×3 convolutional operation and one simple 3×1 convolutional operation, the computational complexity of the convolutional operation can be significantly reduced, and image filtering efficiency can be improved.

[0139] In one example, the third convolutional layer and / or the second convolutional layer includes a grouped convolutional layer, and the first convolutional layer includes two subconvolutional layers obtained by tensor decomposition. As shown in FIG. 16B , the third convolutional layer is a block convolutional layer with a convolutional kernel size of 3×3, the second convolutional layer is a convolutional layer with a convolutional kernel size of 1×1, and the first convolutional layer includes a 1×3 subconvolutional layer and a 3×1 subconvolutional layer.

[0140] In one example, the third convolutional layer and / or the first convolutional layer includes a grouped convolutional layer, and the second convolutional layer includes two subconvolutional layers obtained by tensor decomposition. As shown in FIG. 16C , the first convolutional layer is a block convolutional layer with a convolutional kernel size of 3×3, the third convolutional layer is a convolutional layer with a convolutional kernel size of 1×1, and the second convolutional layer includes a 1×3 subconvolutional layer and a 3×1 subconvolutional layer.

[0141] With reference to the above examples, a network structure has been described in which one or two of the first, second, and third convolutional layers include a grouped convolutional layer, and the other convolutional layer includes two subconvolutional layers obtained by tensor decomposition. In some embodiments, one of the first, second, and third convolutional layers includes a grouped convolutional layer, and the other two convolutional layers include a network structure of two subconvolutional layers obtained by tensor decomposition.

[0142] For example, the second convolutional layer includes a grouping convolutional layer, and the first and second convolutional layers each include two subconvolutional layers obtained by tensor decomposition. As shown in FIG. 16D, the second convolutional layer is a block convolutional layer with a convolution kernel size of 1×1, the first convolutional layer includes a 1×3 subconvolutional layer and a 3×1 subconvolutional layer, and the third convolutional layer also includes a 1×3 subconvolutional layer and a 3×1 subconvolutional layer.

[0143] In this example, some of the first, second, and third convolutional layers in the i-th convolutional kernel include grouped convolutional layers, and some of the convolutional layers include at least two sub-convolutional layers obtained by CP decomposition. Specific embodiments include, but are not limited to, the cases shown in the above examples, and include any other possible cases.

[0144] In some embodiments, for a j-th convolutional layer among the first convolutional layer, the second convolutional layer, and the third convolutional layer, as shown in FIG. 17A , the j-th convolutional layer includes a first grouping sub-convolutional layer and a second grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer whose convolutional kernel size is m×n using a tensor decomposition method and a group convolution method, where m and n are both positive integers greater than 2, and j is a positive integer.

[0145] The embodiments of the present invention are not limited to the sizes of the convolution kernels of the first grouping convolution layer and the second grouping convolution layer, as long as the sizes of the convolution kernels of the first grouping convolution layer and the second grouping convolution layer are both less than m×n.

[0146] In one example, the size of the convolution kernel of the first grouping sub-convolution layer is 1×n, and the size of the convolution kernel of the second grouping sub-convolution layer is m×1.

[0147] In this embodiment, at least one of the first convolutional layer, the second convolutional layer, and the third convolutional layer employs both the tensor decomposition scheme and the group convolution scheme. For convenience of explanation, the jth convolutional layer will be taken as an example.

[0148] In this embodiment, a convolutional layer having a convolutional kernel size of m×n is decomposed and channel grouped using a tensor decomposition method and a group convolution method to obtain a first grouped subconvolutional layer and a second grouped subconvolutional layer. The convolutional kernel sizes of the first grouped subconvolutional layer and the second grouped subconvolutional layer are both smaller than m×n. For example, a convolutional layer having a convolutional kernel size of m×n is first decomposed into two subconvolutional layers, i.e., a first subconvolutional layer and a second subconvolutional layer, using a tensor decomposition method. Then, a convolution is performed on the first subconvolutional layer using a group convolution method to obtain a first grouped subconvolutional layer. Then, a convolution is performed on the second subconvolutional layer using a group convolution method to obtain a second grouped subconvolutional layer. In the embodiment of the present invention, the convolution operation with a large convolution kernel can be decomposed into convolution operations with smaller convolution kernels by using tensor decomposition, and the convolution operation with a large number of channels can be decomposed into convolution operations with a smaller number of channels by using a grouping method, which can further reduce the computational complexity of the convolution operation and greatly improve the efficiency of image filtering.

[0149] For example, let the number of channels of input feature information of the jth convolutional layer be K, the size of the convolutional kernel of the first grouping convolutional layer be 1×n, and the size of the convolutional kernel of the second grouping convolutional layer be m×1. In this case, the filtering device inputs input feature information with number of channels K to the jth convolutional layer, and the first grouping convolutional layer of the jth convolutional layer divides the input feature information with number of channels K into G groups, performs a 1×n convolution operation on the feature information of each group, and then merges the number of channels to obtain feature information 1 output by the first grouping convolutional layer. Let the number of channels of feature information 1 be H. Next, the feature information 1 with number of channels H is input to the second grouping convolutional layer, and the second grouping convolutional layer divides the feature information 1 with number of channels H into P groups, performs an m×1 convolution operation on the feature information of each group, and then merges the number of channels to obtain feature information 2 output by the second grouping convolutional layer.

[0150] The network configuration of the i-th residual block according to this embodiment will be described below with reference to a specific example.

[0151] 17B , the first convolutional layer includes a first grouping sub-convolutional layer and a second grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer having a convolutional kernel size of m×n using a tensor decomposition method and a group convolution method, where the convolutional kernel size of the first grouping convolutional layer is 1×n, and the convolutional kernel size of the second grouping convolutional layer is m×1. The input information of the first convolutional layer and the second convolutional layer is the (i-1)th feature information, and the number of channels of the (i-1)th feature information is K. In this case, the filtering device inputs feature information with the number of channels K to the first convolutional layer, and the first grouping convolutional layer in the first convolutional layer divides the input feature information with the number of channels K into G groups, performs an m×1 convolution operation on the feature information of each group, and then merges the channels to obtain feature information 1 output from the first grouping convolutional layer. Assume that feature information 1 has H channels. Next, feature information 1 with H channels is input to a second grouping convolutional layer, which divides feature information 1 with H channels into P groups. A 1×n convolutional operation is performed on the feature information of each group, followed by channel merging to obtain feature information 2 output by the second grouping convolutional layer. The first feature information is output by the first convolutional unit based on feature information 2. The second convolutional layer performs operations such as convolution on the (i-1)th feature information to obtain second feature information output by the second convolutional unit. The first feature information and second feature information are merged, and then input to a third convolutional layer for convolutional operation to obtain output feature information of the i-th residual block.

[0152] For example, as shown in FIG. 17C , the first convolutional layer includes a first grouping sub-convolutional layer with a convolutional kernel size of 1×3 and a second grouping sub-convolutional layer with a convolutional kernel size of 3×1, which are obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of 3×3 using a tensor decomposition method and a group convolution method. The filtering device divides input feature information with K channels into G groups using the first grouping convolutional layer, performs a 3×1 convolution operation on the feature information of each group, and then merges the channels to obtain feature information 1 output by the first grouping convolutional layer. Assume that the number of channels of feature information 1 is H. Next, feature information 1 with H channels is input to the second grouping convolutional layer, which divides feature information 1 with H channels into P groups. A 1×3 convolution operation is performed on the feature information of each group, followed by channel merging to obtain feature information 2 output by the second grouping convolutional layer. Feature information 2 is processed using a first activation function to obtain first feature information output by the first convolutional unit. The second convolutional layer performs a convolution operation on the (i-1)th feature information, then processes it using a second activation function to obtain second feature information output by the second convolutional unit. The first feature information and second feature information are merged and input to the third convolutional layer for convolution and other operations to obtain output feature information for the i-th residual block.

[0153] 17D , the first convolutional layer includes a first grouping sub-convolutional layer and a second grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer having a convolutional kernel size of m×n using a tensor decomposition scheme and a group convolution scheme, the convolutional kernel size of the first grouping convolutional layer is 1×n, the convolutional kernel size of the second grouping convolutional layer is m×1, and at least one of the second convolutional layer and the third convolutional layer includes a grouping convolutional layer. The input information of the first convolutional layer and the second convolutional layer is defined as the (i-1)th feature information, and the number of channels of the (i-1)th feature information is defined as K. In this case, the filtering device inputs feature information with K channels to a first convolutional layer, and a first grouping convolutional layer in the first convolutional layer divides the input feature information with K channels into G groups, performs m×1 convolution operations on the feature information of each group, and then merges the number of channels to obtain feature information 1 output by the first grouping convolutional layer. Assume that feature information 1 has H channels. Next, feature information 1 with H channels is input to a second grouping convolutional layer, and the second grouping convolutional layer divides the feature information with H channels into P groups, performs 1×n convolution operations on the feature information of each group, and then merges the number of channels to obtain feature information 2 output by the second grouping convolutional layer. The first feature information output by the first convolutional unit is obtained based on feature information 2. The second convolutional layer performs operations such as convolution (or group convolution) on the (i-1)th feature information to obtain the second feature information output by the second convolutional unit, merges the first feature information and the second feature information, and then inputs the merged information into the third convolutional layer, where it performs operations such as convolution (or group convolution) to obtain the output feature information of the i-th residual block.

[0154] For example, as shown in FIG. 17E, the first convolutional layer includes a first grouping sub-convolutional layer with a convolutional kernel size of 1×3 and a second grouping sub-convolutional layer with a convolutional kernel size of 3×1, which are obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of 3×3 using a tensor decomposition method and a group convolution method. The second convolutional layer and the third convolutional layer include grouping convolutional layers. The first grouping convolutional layer divides input feature information with K channels into G groups, performs a 3×1 convolution operation on the feature information of each group, and then merges the channels to obtain feature information 1 output by the first grouping convolutional layer. Assume that the number of channels of feature information 1 is H. Next, feature information 1 with H channels is input to the second grouping convolutional layer, which divides feature information 1 with H channels into P groups. A 1x3 convolution operation is performed on the feature information of each group, followed by channel merging to obtain feature information 2 output by the second grouping convolutional layer. Feature information 2 is processed using a first activation function to obtain first feature information output by the first convolutional unit. The second convolutional layer performs a group convolution operation on the (i-1)th feature information, then processes it using a second activation function to obtain second feature information output by the second convolutional unit. The first feature information and second feature information are merged and input to the third convolutional layer, where group convolution and other operations are performed to obtain output feature information for the i-th residual block.

[0155] 17F , the first convolutional layer includes a grouping sub-convolutional layer with a convolutional kernel of size 1×n1 and a grouping sub-convolutional layer with a convolutional kernel of size m1×n1, obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel of size m1×n1 using a tensor decomposition scheme and a group convolutional scheme. The third convolutional layer includes a grouping sub-convolutional layer with a convolutional kernel of size 1×n2 and a grouping sub-convolutional layer with a convolutional kernel of size m2×n2, obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel of size m2×n2 using a tensor decomposition scheme and a group convolutional scheme. The input information of the first convolutional layer and the second convolutional layer is defined as the (i-1)th feature information, and the number of channels of the (i-1)th feature information is defined as K. In this case, the filtering device inputs feature information with K channels to a first convolutional layer, and a grouping sub-convolutional layer in the first convolutional layer, whose convolutional kernel size is m1×1, divides the input feature information with K channels into G groups, performs m1×1 convolution operations on the feature information of each group, and then merges the channels to obtain feature information 1 output by the grouping sub-convolutional layer, whose convolutional kernel size is m1×1. It is assumed that feature information 1 has H channels. Next, feature information 1 with H channels is input to a grouping sub-convolutional layer with a convolutional kernel size of 1×n1. The grouping sub-convolutional layer with a convolutional kernel size of 1×n1 divides feature information 1 with H channels into P groups, performs a 1×n1 convolution operation on the feature information for each group, and then performs channel merging to obtain feature information 2 output from the grouping sub-convolutional layer with a convolutional kernel size of 1×n1. Based on feature information 2, the first feature information is output from the first convolutional unit. The second convolutional layer performs an operation such as convolution (or group convolution) on the (i-1)th feature information to obtain second feature information output from the second convolutional unit, and merges the first feature information and the second feature information to obtain third feature information. The number of channels of the third feature information is Q, where Q is a positive integer greater than 1.Next, the third feature information is input to the third convolutional layer, and the grouping sub-convolutional layer in the third convolutional layer, whose convolutional kernel size is m2 × 1, divides the third feature information with Q channels into R groups, performs m2 × 1 convolution operations on the feature information of each group, and then merges the number of channels to obtain feature information 3 output by the grouping sub-convolutional layer with a convolutional kernel size of m2 × 1. Assume that the number of channels of feature information 3 is S. Next, the feature information 3 with S channels is input to a grouping sub-convolutional layer with a convolutional kernel size of 1×n2. The grouping sub-convolutional layer with a convolutional kernel size of 1×n2 divides the feature information 3 with S channels into T groups, performs a 1×n2 convolution operation on the feature information of each group, and then merges the channels to obtain feature information 4 output by the grouping sub-convolutional layer with a convolutional kernel size of 1×n2. The output feature information of the i-th residual block is obtained based on feature information 4.

[0156] For example, as shown in FIG. 17G , the first convolutional layer includes a grouping sub-convolutional layer with a convolutional kernel size of 1×3 and a grouping sub-convolutional layer with a convolutional kernel size of 3×1, obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of 3×3 using the tensor decomposition method and the group convolution method. The third convolutional layer includes a grouping sub-convolutional layer with a convolutional kernel size of 1×3 and a grouping sub-convolutional layer with a convolutional kernel size of 3×1, obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of 3×3 using the tensor decomposition method and the group convolution method. The second convolutional layer includes a grouping convolutional layer. The input information of the first convolutional layer and the second convolutional layer is defined as the (i-1)th feature information, and the number of channels of the (i-1)th feature information is defined as K. In this case, the filtering device inputs feature information with K channels to a first convolutional layer, and a grouping sub-convolutional layer in the first convolutional layer, with a convolutional kernel size of 3×1, divides the input feature information with K channels into G groups, performs a 3×1 convolution operation on each feature information group, and then merges the channels to obtain feature information 1 output by the grouping sub-convolutional layer with a convolutional kernel size of 3×1. It is assumed that the number of channels of feature information 1 is H. Next, feature information 1 with H channels is input to a grouping sub-convolutional layer with a convolutional kernel size of 1×3. The grouping sub-convolutional layer with a convolutional kernel size of 1×3 divides the feature information with H channels 1 into P groups, performs a 1×3 convolution operation on the feature information of each group, and then performs channel merging to obtain feature information 2 output by the grouping sub-convolutional layer with a convolutional kernel size of 1×3. The first feature information is output by the first convolutional unit based on this feature information 2. The second convolutional layer performs an operation such as convolution (or group convolution) on the (i-1)th feature information to obtain second feature information output by the second convolutional unit, and merges the first feature information and second feature information to obtain third feature information.Assume that the number of channels of the third feature information is Q, where Q is a positive integer greater than 1. Next, the third feature information is input to the third convolutional layer, and a grouping sub-convolutional layer in the third convolutional layer, with a convolutional kernel size of 3×1, divides the third feature information with Q channels into R groups, performs a 3×1 convolutional operation on the feature information of each group, and then merges the number of channels to obtain feature information 3 output by the grouping sub-convolutional layer with a convolutional kernel size of 3×1. Assume that the number of channels of feature information 3 is S. Next, feature information 3 with S channels is input to a grouping sub-convolutional layer with a convolutional kernel size of 1x3. The grouping sub-convolutional layer with a convolutional kernel size of 1x3 divides the feature information 3 with S channels into T groups, performs a 1x3 convolution operation on the feature information of each group, and then merges the channels to obtain feature information 4 output by the grouping sub-convolutional layer with a convolutional kernel size of 1x3. The output feature information of the i-th residual block is obtained based on feature information 4.

[0157] As can be seen from the above, in this embodiment, at least one of the first convolutional layer, the second convolutional layer, and the third convolutional layer in the i-th residual block includes a first grouped sub-convolutional layer and a second grouped sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer with a convolution kernel size of m×n using tensor decomposition and grouped convolution. Preferably, the remaining convolutional layers among the first convolutional layer, the second convolutional layer, and the third convolutional layer include grouped convolutional layers. In other words, in this embodiment, a multi-level receptive field scheme may be used, and then CP decomposition and grouped convolution may be simultaneously used to replace at least one m×n standard convolutional layer (m and n are positive integers greater than or equal to 3) in the existing residual block, and / or grouped convolution may be used to replace at least one of the remaining i×j standard convolutional layers (i and j are positive integers) in the existing residual block.

[0158] In some embodiments, based on the above embodiments, as shown in FIG. 18A , the third convolution unit according to the embodiments of the present invention further includes, in addition to the third convolution layer, a fourth convolution layer connected in series to the third convolution layer.

[0159] In some embodiments, as shown in FIG. 18B , the fourth convolutional layer includes a grouping convolutional layer. For example, if the feature information output by the third convolutional layer is denoted as fourth feature information and the number of channels of the fourth feature information is denoted as K2, the filtering device inputs the fourth feature information output by the third convolutional layer to the fourth convolutional layer. The fourth convolutional layer divides the fourth feature information, which has the K2 number of channels, into multiple groups and performs a standard convolution operation on each of the resulting multiple groups to obtain post-convolution feature information for each group. Next, the post-convolution feature information for each group is merged to obtain output feature information for the fourth convolutional layer. Finally, the output feature information of the fourth convolutional layer is added to the input feature information of the i-th residual block, e.g., the i-1-th feature information, to obtain output feature information for the i-th residual block.

[0160] In some embodiments, as shown in FIG. 18C , a convolutional layer with a convolutional kernel size of p×q is decomposed using tensor decomposition to obtain a third sub-convolutional layer and a fourth sub-convolutional layer, where p and q are both positive integers greater than 2.

[0161] In some embodiments, the size of the convolution kernels of the third and fourth subconvolution layers is not limited as long as it is smaller than p×q.

[0162] In one example, the size of the convolution kernel of the third sub-convolution layer is 1×q, and the size of the convolution kernel of the fourth sub-convolution layer is p×1.

[0163] In this embodiment, as shown in FIG. 18C , it is assumed that the feature information output by the third convolutional layer is referred to as the fourth feature information. The filtering device inputs the fourth feature information output by the third convolutional layer to the fourth convolutional layer. The third sub-convolutional layer in the fourth convolutional layer performs a simple 1×q convolution operation on the fourth feature information to output fifth feature information. Next, the fifth feature information is input to the fourth sub-convolutional layer, where a simple p×1 convolution operation is performed to output sixth feature information. Finally, the sixth feature information output by the fourth convolutional layer is added to the input feature information of the i-th residual block, e.g., the i-1-th feature information, to obtain the output feature information of the i-th residual block. In this way, the fourth convolutional layer decomposes a complex p×q convolution operation into two simple 1×q and p×1 convolution operations, thereby reducing the complexity of image filtering and improving the efficiency of image filtering.

[0164] In some embodiments, as shown in FIG. 18D , the fourth convolutional layer includes a third grouping sub-convolutional layer and a fourth grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer with a convolution kernel size of p×q using a tensor decomposition method and a group convolution method.

[0165] In this embodiment, as shown in FIG. 18D , the feature information output by the third convolutional layer is referred to as fourth feature information. It is assumed that the number of channels of the fourth feature information is K2, the size of the convolutional kernel of the third grouping sub-convolutional layer is 1×q, and the size of the convolutional kernel of the fourth grouping sub-convolutional layer is p×1. The filtering device inputs the fourth feature information output by the third convolutional layer to the fourth convolutional layer. The third grouping sub-convolutional layer in the fourth convolutional layer divides the fourth feature information with K2 channels into multiple groups, performs p×1 convolution operations on each group feature information, and then merges the number of channels to obtain feature information 4 output by the third grouping sub-convolutional layer. It is assumed that the number of channels of feature information 4 is K2. Next, feature information 4 with K2 channels is input to the fourth grouping sub-convolutional layer. The fourth grouping sub-convolutional layer divides the feature information 4 of the K2 channels into multiple groups, performs 1×q convolution operations on each group, and then merges the channels to obtain the feature information 5 output by the fourth grouping sub-convolutional layer. The fourth grouping sub-convolutional layer obtains the feature information output by the third convolutional unit based on the feature information 5, and adds the feature information output by the third convolutional unit to the input feature information of the i-th residual block, for example, the (i-1)-th feature information, to obtain the output feature information of the i-th residual block.

[0166] As can be seen from the above, a residual block according to an embodiment of the present invention includes a first convolutional unit, a second convolutional unit, and a third convolutional unit, where each of the first convolutional unit, the second convolutional unit, and the third convolutional unit includes at least one convolutional layer. At least one of the convolutional layers included in the residual block includes a grouped convolutional layer formed by a grouped convolutional scheme, and / or the at least one convolutional layer includes at least two grouped sub-convolutional layers obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of m×n by a tensor decomposition scheme and a grouped convolutional scheme.

[0167] In other words, the standard convolutional layer in the existing residual block is decomposed into multiple convolutional layers with low computational complexity using a multi-level receptive field method and a group convolution method, or the standard convolutional layer in the existing residual block is decomposed into multiple convolutional layers with low computational complexity using a multi-level receptive field method, a group convolution method, and a tensor decomposition method.

[0168] 19A , the i-th residual block includes a first convolutional unit, a second convolutional unit, and a third convolutional unit. Here, the first convolutional unit includes a grouping convolutional layer (first convolutional layer) with a convolutional kernel size of 3×3 and a first activation function (e.g., PReLU). The second convolutional unit includes a second convolutional layer with a convolutional kernel size of 1×1 and a second activation function (e.g., PReLU). The third convolutional unit includes a third convolutional layer with a convolutional kernel size of 1×1 and a grouping convolutional layer (i.e., fourth convolutional layer) with a convolutional kernel size of 3×3. The filtering device inputs the (i-1)th feature information to the ith residual block. A grouping convolutional layer in the ith residual block, with a convolutional kernel size of 3x3, divides the (i-1)th feature information into multiple groups, performs a 3x3 convolution operation on the feature information of each group, and then merges the number of channels to obtain feature information output by the first convolutional layer. Next, the feature information output by the first convolutional layer is processed using a first activation function to obtain first feature information output by the first convolutional layer. A second convolutional layer, with a convolutional kernel size of 1x1, performs a standard convolution operation on the (i-1)th feature information, and then processes the feature information output by the second convolutional layer using a second activation function to obtain second feature information output by the second convolutional layer. Next, the first feature information and the second feature information are concatenated to obtain third feature information, which is input to the third convolution unit. The third convolution layer in the third convolution unit performs a standard convolution operation on the third feature information to obtain fourth feature information, which is input to the fourth convolution layer. The fourth convolution layer includes a grouping convolution layer with a convolution kernel size of 3x3. The grouping convolution layer with a convolution kernel size of 3x3 divides the fourth feature information into multiple groups, performs a 3x3 convolution operation on the feature information of each group, and then merges the number of channels to obtain the feature information output by the third convolution unit.The feature information output by the third convolution unit and the (i-1)th feature information may be added to obtain the feature information output by the i-th residual block, which may be represented as the i-th feature information.

[0169] In one example, as shown in FIG. 19B , the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit. Here, the first convolution unit includes a first convolution layer with a convolution kernel size of 3×3 and a first activation function (e.g., PReLU). The second convolution unit includes a grouping convolution layer (i.e., the second convolution layer) with a convolution kernel size of 1×1 and a second activation function (e.g., PReLU). The third convolution unit includes a grouping convolution layer (i.e., the third convolution layer) with a convolution kernel size of 1×1 and a fourth convolution layer with a convolution kernel size of 3×3. The filtering device inputs the i-1-th feature information into the i-th residual block. The first convolutional layer in the i-th residual block, whose first convolutional unit has a convolutional kernel size of 3x3, performs a standard convolution operation on the i-1-th feature information, and then processes the feature information output from the first convolutional layer using a first activation function to obtain the first feature information output from the first convolutional unit. The second convolutional layer in the i-th residual block, whose second convolutional unit has a convolutional kernel size of 3x3, divides the i-1-th feature information into multiple groups, performs a 3x3 convolution operation on the feature information of each group, and then merges the number of channels to obtain the feature information output from the second convolutional layer. Next, the second activation function is used to process the feature information output from the second convolutional layer to obtain the second feature information output from the second convolutional unit. Next, the first feature information and the second feature information are concatenated to obtain the third feature information, which is input to the third convolutional unit. The third convolutional layer in the third convolutional unit includes a grouping convolutional layer with a convolutional kernel size of 1 × 1. The grouping convolutional layer with a convolutional kernel size of 1 × 1 divides the third feature information into multiple groups, performs a 3 × 3 convolution operation on the feature information of each group, and then merges the number of channels to obtain the fourth feature information output by the third convolutional layer.Next, the fourth feature information is input to the fourth convolution layer for standard convolution operation to obtain the feature information output by the third convolution unit. Finally, the feature information output by the third convolution unit is added to the (i-1)th feature information to obtain the feature information output by the i-th residual block, which may be denoted as the i-th feature information.

[0170] In one example, as shown in FIG. 19C , the i-th residual block includes a first convolutional unit, a second convolutional unit, and a third convolutional unit. Here, the first convolutional unit includes a first convolutional layer and a first activation function (e.g., PReLU), and the first convolutional layer includes a first grouping subconvolutional layer with a convolutional kernel size of 1×3 and a second grouping subconvolutional layer with a convolutional kernel size of 3×1. The second convolutional unit includes a second convolutional layer with a convolutional kernel size of 1×1 and a second activation function (e.g., PReLU). The third convolutional unit includes a third convolutional layer with a convolutional kernel size of 1×1 and a fourth convolutional layer. The fourth convolutional layer includes a third grouping subconvolutional layer with a convolutional kernel size of 1x3 and a fourth grouping subconvolutional layer with a convolutional kernel size of 3x1. The filtering device inputs the (i-1)th feature information to the ith residual block. The first grouping subconvolutional layer in the first convolutional unit of the ith residual block divides the (i-1)th feature information into multiple groups, performs a 1x3 convolution operation on the feature information of each group, and then performs channel merging to obtain output feature information of the first grouping subconvolutional layer. Next, the output feature information of the first grouping subconvolutional layer is input to the second grouping subconvolutional layer and performs a 3x1 convolution of the groups to obtain output feature information of the second grouping subconvolutional layer. The output feature information of the second grouping subconvolutional layer is processed using a first activation function to obtain first feature information output by the first convolutional unit. At the same time, the second convolutional layer in the second convolutional unit in the i-th residual block performs a 1x1 convolution operation on the (i-1)-th feature, and then processes the feature output from the second convolutional layer using the second activation function to obtain second feature output from the second convolutional unit. Next, the first feature and the second feature are concatenated to obtain third feature, which is input to the third convolutional unit.The third convolutional layer in the third convolutional unit performs a 1x1 standard convolution operation on the third feature information to obtain fourth feature information. The fourth feature information is then input to the fourth convolutional layer. The fourth convolutional layer includes a third grouping subconvolutional layer with a convolutional kernel size of 1x3 and a fourth grouping subconvolutional layer with a convolutional kernel size of 3x1. Here, the third grouping subconvolutional layer divides the fourth feature information into multiple groups, performs a 1x3 convolution operation on the feature information of each group, and then performs channel merging to obtain output feature information for the third grouping subconvolutional layer. The output feature information of the third grouping subconvolutional layer is then input to the fourth grouping subconvolutional layer for grouping 3x1 convolution to obtain output feature information for the fourth grouping subconvolutional layer. Finally, the feature information output by the third convolution unit and the (i-1)th feature information are added to obtain the feature information output by the i-th residual block, which may be denoted as the i-th feature information.

[0171] In one example, as shown in FIG. 19D , the i-th residual block includes a first convolutional unit, a second convolutional unit, and a third convolutional unit. Here, the first convolutional unit includes a first convolutional layer and a first activation function (e.g., PReLU), and the first convolutional layer includes a first grouping subconvolutional layer with a convolutional kernel size of 1×3 and a second grouping subconvolutional layer with a convolutional kernel size of 3×1. The second convolutional unit includes a grouping convolutional layer with a convolutional kernel size of 1×1 and a second activation function (e.g., PReLU). The third convolutional unit includes a grouping convolutional layer with a convolutional kernel size of 1×1 and a fourth convolutional layer. The fourth convolutional layer includes a third grouping subconvolutional layer with a convolutional kernel size of 1x3 and a fourth grouping subconvolutional layer with a convolutional kernel size of 3x1. The filtering device inputs the (i-1)th feature information to the ith residual block. The first grouping subconvolutional layer in the first convolutional unit of the ith residual block divides the (i-1)th feature information into multiple groups, performs a 1x3 convolution operation on the feature information of each group, and then performs channel merging to obtain output feature information of the first grouping subconvolutional layer. The output feature information of the first grouping subconvolutional layer is then input to the second grouping subconvolutional layer for a 3x1 grouping convolution to obtain output feature information of the second grouping subconvolutional layer. The output feature information of the second grouping subconvolutional layer is processed using a first activation function to obtain first feature information output by the first convolutional unit. At the same time, the second convolutional layer in the second convolutional unit of the ith residual block performs a 1x1 convolution operation to group the (i-1)th feature information, then processes the feature information output by the second convolutional layer using the second activation function to obtain the second feature information output by the second convolutional unit. Next, the first feature information and the second feature information are concatenated to obtain the third feature information, which is input to the third convolutional unit. The third convolutional layer in the third convolutional unit performs a 1x1 convolution operation to group the third feature information, and then obtains the fourth feature information.Next, the fourth feature information is input to a fourth convolutional layer. The fourth convolutional layer includes a third grouping subconvolutional layer with a convolutional kernel size of 1×3 and a fourth grouping subconvolutional layer with a convolutional kernel size of 3×1. Here, the third grouping subconvolutional layer divides the fourth feature information into multiple groups, performs a 1×3 convolution operation on the feature information of each group, and then performs channel merging to obtain output feature information of the third grouping subconvolutional layer. Next, the output feature information of the third grouping subconvolutional layer is input to a fourth grouping subconvolutional layer and performs a grouping 3×1 convolution to obtain output feature information of the fourth grouping subconvolutional layer. Finally, the feature information output by the third convolutional unit is added to the (i-1)th feature information to obtain the feature information output by the i-th residual block, which may be referred to as the i-th feature information.

[0172] In the above embodiment, in Scenario 1, the filtering device processes the standard convolution layer based on the multi-level receptive field method and the group convolution method, and based on the multi-level receptive field method, the group convolution method, and the tensor decomposition method to obtain the i-th residual block.

[0173] Scenario 2: The filtering device processes the standard convolution layer based on the group convolution method and the tensor decomposition method to obtain the i-th residual block.

[0174] Specifically, as shown in Figure 20, the i-th residual block in this embodiment of the present invention includes N convolutional layers connected in series. The k-th convolutional layer among the N convolutional layers includes a fifth grouping sub-convolutional layer and a sixth grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of r x s using a tensor decomposition method and a group convolution method. r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer less than or equal to N.

[0175] In an embodiment of the present invention, the i-th residual block includes N serially connected convolution layers, some or all of which include at least two sub-convolution layers with low computational complexity obtained by decomposition using a tensor decomposition method and a group convolution method. In this way, the computational complexity of the residual block can be reduced without degrading the image processing performance of the residual block, thereby improving the image filtering performance of the neural network filter.

[0176] In one example, an embodiment of the present invention may perform decomposition and channel grouping on the first 3×3 convolutional layer in the residual block shown in FIG. 7A using a tensor decomposition method and a group convolution method, to obtain a grouped sub-convolutional layer with a convolutional kernel size of 1×3 and a grouped sub-convolutional layer with a convolutional kernel size of 3×1.

[0177] Preferably, as shown in FIG. 21A, the filtering device may not process the second 3×3 convolutional layer in the residual block shown in FIG. 7A.

[0178] Preferably, as shown in FIG. 21B, the filtering device may decompose the second 3×3 convolutional layer in the residual block shown in FIG. 7A into a 1×3 sub-convolutional layer and a 3×1 sub-convolutional layer using a tensor decomposition method.

[0179] Preferably, as shown in FIG. 21C, the filtering device processes the second 3×3 convolution layer in the residual block shown in FIG. 7A into a 3×3 grouped convolution layer by group convolution.

[0180] Preferably, as shown in FIG. 21D , the filtering device may perform decomposition and channelization on the second 3×3 convolutional layer in the residual block shown in FIG. 7 using a tensor decomposition method and a group convolution method, to obtain a grouped sub-convolutional layer with a convolutional kernel size of 1×3 and a grouped sub-convolutional layer with a convolutional kernel size of 3×1.

[0181] In one example, an embodiment of the present invention may perform decomposition and channelization on the third 3×3 convolutional layer in the residual block shown in FIG. 7B using a tensor decomposition method and a group convolution method to obtain a grouped sub-convolutional layer with a convolutional kernel size of 1×3 and a grouped sub-convolutional layer with a convolutional kernel size of 3×1.

[0182] Preferably, the filtering device does not process the first and second 1x1 convolutional layers in the residual block shown in Figure 7B.

[0183] Preferably, the filtering device decomposes at least one of the first and second 1×1 convolutional layers in the residual block shown in FIG. 7B into grouped convolutional layers by group convolution. For example, the filtering device decomposes the first 1×1 convolutional layer and the second 1×1 convolutional layer in FIG. 7B into grouped convolutional layers by group convolution.

[0184] In the above embodiment, in Scenario 2, the filtering device processes the standard convolution layer based on the group convolution method and the tensor decomposition method to obtain the i-th residual block.

[0185] In some embodiments, as shown in Figure 22A, the neural network filter of the embodiments of the present invention further includes a shallow feature extraction unit including at least one convolutional layer, which extracts shallow feature information from the input information and inputs the shallow feature information to the residual unit.

[0186] Specifically, as shown in Figure 22A, the neural network filter of the embodiment of the present invention includes a shallow feature extraction unit and a residual unit. During actual filtering, the filtering device inputs input information corresponding to the image to be filtered to the shallow feature extraction unit, which extracts shallow feature information from the input information. The shallow feature information is then input to the residual unit, which re-extracts the shallow feature information to obtain image feature information of the image to be filtered.

[0187] The shallow feature extraction unit of the embodiment of the present invention corresponds to the head part of the neural network filter shown in FIG.

[0188] In some embodiments, some convolutional layers in the shallow feature extraction unit of the embodiments of the present invention are obtained by processing standard convolutional layers using at least two methods, including a multi-level receptive field method, a tensor decomposition method, and a group convolution method. For example, the top 3x3 convolutional layer of the neural network filter shown in FIG. 6 described above is decomposed using at least two methods, including a multi-level receptive field method, a tensor decomposition method, and a group convolution method, to obtain multiple sub-convolutional layers with low computational complexity. For example, the top 3x3 convolutional layer of the neural network filter shown in FIG. 6 described above is decomposed and grouped using a tensor decomposition method and a group convolution method to obtain two grouped sub-convolutional layers. For specific decomposition processes, please refer to the relevant descriptions in the above embodiments, and further description will be omitted here.

[0189] In some embodiments, as shown in Figure 22B, the neural network filter of the embodiments of the present invention further includes a feature mapping unit including at least one convolution layer, which performs a mapping process on the image feature information output by the residual unit to obtain a filtered image.

[0190] Specifically, as shown in Figure 22B, the neural network filter of the embodiment of the present invention includes a residual unit and a feature mapping unit. During actual filtering, the filtering device inputs input information corresponding to the image to be filtered to the residual unit, which extracts feature information from the input information to obtain image feature information of the image to be filtered. Next, the image feature information is input to the feature mapping unit, which performs a mapping process on the image feature information to obtain a filtered image of the image to be filtered.

[0191] In some embodiments, as shown in FIG. 22C , a neural network filter according to an embodiment of the present invention includes a shallow feature extraction unit, a residual unit, and a feature mapping unit. During actual filtering, the filtering device inputs input information corresponding to the image to be filtered to the shallow feature extraction unit, which extracts shallow feature information from the input information. Next, the filtering device inputs the shallow feature information to the residual unit, which performs feature re-extraction on the shallow feature information to obtain image feature information for the image to be filtered. Finally, the filtering device inputs the image feature information to the feature mapping unit, which performs mapping to obtain a filtered image.

[0192] The feature mapping unit in the embodiment of the present invention corresponds to the tail end of the neural network filter shown in FIG.

[0193] In some embodiments, some convolutional layers in the feature mapping unit of the embodiments of the present invention are obtained by processing standard convolutional layers using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method. For example, the filtering device decomposes the 3x3 convolutional layer at the end of the neural network filter shown in Figure 6 described above using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method to obtain multiple sub-convolutional layers with low computational complexity. For example, the 3x3 convolutional layer at the end of the neural network filter shown in Figure 6 is decomposed and grouped using the tensor decomposition method and the group convolution method to obtain two grouped sub-convolutional layers. For the specific decomposition process, please refer to the relevant descriptions in the above embodiments, and the description thereof will be omitted here.

[0194] In some embodiments, the neural network filter of an embodiment of the present invention may be a single-branch convolutional neural network that performs filtering operations on the luma and chroma components of the image to be filtered.

[0195] In some embodiments, the neural network filter of the present invention may be a double-branch convolutional neural network that performs separation processing on the luma component and the chroma component. In this case, as shown in FIG. 23A , the residual unit of the present invention includes a luma component residual unit and a chroma component residual unit. Here, the luma component residual unit performs feature extraction processing on the luma component of the filtering target image to obtain image feature information of the luma component of the filtering target image. The chroma component residual unit performs feature extraction processing on the chroma component of the filtering target image to obtain image feature information of the chroma component of the filtering target image.

[0196] In some embodiments, when the neural network filter includes a feature mapping unit, the feature mapping unit includes a luma component feature mapping unit connected to the luma component residual unit and a chroma component feature mapping unit connected to the chroma component residual unit, as shown in FIG. 23B.

[0197] Here, the luminance component feature mapping unit performs a mapping process on the image luminance feature information output by the luminance component residual unit, and the chrominance component feature mapping unit performs a mapping process on the image chrominance feature information output by the chrominance component residual unit.

[0198] In some embodiments, as shown in FIG. 23C , the neural network filter further includes a shallow feature extraction unit. As shown in FIG. 23C , during actual filtering, the filtering device inputs input information corresponding to the image to be filtered to the shallow feature extraction unit, which performs shallow feature extraction on the input information to obtain shallow feature information for the input information. Next, the filtering device inputs luminance component shallow feature information from the shallow feature information to the luminance component residual unit, which performs feature extraction on the luminance component shallow feature information to obtain image feature information for the luminance component of the image to be filtered. Next, the filtering device inputs the luminance component image feature information of the image to be filtered to the luminance component feature mapping unit to perform mapping, thereby obtaining filtered image information for the luminance component of the image to be filtered. Similarly, the filtering device inputs chrominance component shallow feature information from the shallow feature information to the chrominance component residual unit, which performs feature extraction on the chrominance component shallow feature information to obtain image feature information for the chrominance component of the image to be filtered. Next, the image feature information of the chromaticity components of the image to be filtered is input to a chromaticity component feature mapping unit for mapping, thereby obtaining filtered image information of the chromaticity components of the image to be filtered. Finally, the filtering device adds the filtered image information of the luminance component and the filtered image information of the chromaticity components of the image to be filtered and the input information, thereby obtaining a filtered image of the image to be filtered.

[0199] In some embodiments, when the above-mentioned target image to be filtered is a reconstructed image, the filtering method of the embodiments of the present invention can be applied to video post-processing and video encoding / decoding.

[0200] For example, when applied to video encoding / decoding, the filtering device may input the filtered image of the reconstructed image directly to a display device for display, or may store the filtered image of the reconstructed image in a decoded image buffer and use it as a reference image for other subsequent images to predict other images.

[0201] For example, when applied to video post-processing, the filtered image of the reconstructed image may be input to a display device and displayed directly, without being used as a reference image for other images.

[0202] A neural network-based image filtering method according to an embodiment of the present invention provides a new neural network filter, which includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method. For example, in an embodiment of the present invention, a standard convolution layer in an existing neural network filter, such as a standard convolution layer (e.g., a 3x3 convolution layer) in the head, body, or tail of a neural network filter shown in Figure 6 or Figure 8, is processed using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method, to convert it into a convolution operation with low computational complexity. This reduces the complexity of the filtering operation of the neural network filter without degrading the filtering performance of the neural network filter, and improves the image filtering effect.

[0203] The above describes an image filtering method and a network configuration of a neural network filter according to an embodiment of the present invention. The neural network filter described in the above embodiment can be applied to video decoding.

[0204] 24 is a flowchart of an image decoding method according to an embodiment of the present invention. The method according to an embodiment of the present invention is applied to a decoding device or decoder. As shown in FIG. 24, the method includes the following steps:

[0205] Step S201: Decode the bitstream, determine a reconstructed image of the current image, and determine input information of the neural network filter according to the reconstructed image.

[0206] Here, the input information includes a reconstructed image.

[0207] The method according to the embodiment of the present invention may be applied to video decoding schemes or to video post-processing.

[0208] In some embodiments, when applied to a video decoding scheme, the encoding side first divides the current image into coding blocks and encodes the coding blocks as coding units block by block when encoding a current image. For example, for a current block to be encoded in the current image, first obtain a predicted value of the current block by inter-frame prediction and / or intra-frame prediction. Then, obtain a residual value of the current block based on the predicted value of the current block and the current block. The encoding side transforms the residual value of the current block to obtain transform coefficients. In one example, the encoding side directly codes the transform coefficients of the current block without quantizing them to obtain a bitstream. In another example, the encoding side quantizes the transform coefficients of the current block, obtains quantized coefficients, encodes the quantized coefficients, and obtains a bitstream.

[0209] In the encoding process, as shown in Figure 2, the transform coefficients are inversely transformed to obtain residual values, and the residual values ​​are added to the predicted values ​​to obtain reconstructed values ​​for the current block. Through the above steps, reconstructed values ​​for each coding block of the current image can be obtained, and these reconstructed values ​​constitute a reconstructed image for the current image. Then, to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image for the current image. In one example, the decoded image may be stored in a decoded image buffer for prediction of subsequent images.

[0210] As shown in FIG. 3, for each block to be decoded in the current image, e.g., the current block, the decoding side obtains a bitstream, decodes the bitstream, and obtains the transform coefficients of the current block. In one example, after the encoding side quantizes and re-encodes the transform coefficients, the decoding side obtains the decoded bitstream, obtains the quantized coefficients of the current block, and dequantizes the quantized coefficients to obtain the transform coefficients of the current image. Next, the decoding side inversely transforms the transform coefficients of the current block to obtain the residual values ​​of the current block. At the same time, the decoding side may predict the predicted values ​​of the current block using inter-frame and / or intra-frame prediction methods. Next, the predicted values ​​and the residual values ​​of the current block are added to obtain the reconstructed values ​​of the current block. The decoding side decodes and determines the reconstructed values ​​of each block to be decoded in the current image based on the above steps, and these reconstructed values ​​constitute a reconstructed image of the current image. Next, the decoding side filters the reconstructed image to obtain a decoded image of the current image in order to further improve the quality of the reconstructed image. In one example, the decoding side may store the decoded image in a decoded image buffer for prediction of subsequent images. In one example, the decoding side may output the decoded image to a display device for display.

[0211] In some embodiments, the image filtering method according to the embodiments of the present invention may be used to filter at least one frame image in a video, i.e., the current image is an image in the video.

[0212] In some embodiments, the image filtering method according to the embodiments of the present invention may be used to decode a single image, i.e., the current image is a single image, for example, an image generated by an electronic device.

[0213] In an embodiment of the present invention, the decoding side filters the reconstructed image using a neural network filter.

[0214] In an embodiment of the present invention, when filtering a reconstructed image, the entire reconstructed image may be directly used as a filtering unit to perform filtering, or the reconstructed image may be divided into multiple blocks and each block may be used as a filtering unit to perform filtering. For convenience of explanation, the embodiment of the present invention uses the reconstructed image as one filtering unit, and the reconstructed image may be one reconstructed image (i.e., one frame) or a partial region of the reconstructed image.

[0215] In one preferred embodiment, the reconstructed image includes at least one CTU.

[0216] In another preferred embodiment, the reconstructed image includes a predetermined image region of one frame of the reconstructed image.

[0217] The embodiment of the present invention is not limited to a specific method in which the decoding side determines input information to the neural network filter based on the reconstructed image.

[0218] In one preferred embodiment, the input information includes only the reconstructed image.

[0219] In one preferred embodiment, the input information includes, in addition to the reconstructed image, at least one of a predicted image corresponding to the reconstructed image, a boundary intensity image, a frame type (slice type) of the reconstructed image, and quantization parameter information.

[0220] Step S202: A filtering process is performed on the input information using a neural network filter to obtain a filtered image of the reconstructed image.

[0221] Here, the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0222] Here, the specific network configuration of the neural network filter may be referred to in the description of the above embodiment, and the description thereof will be omitted here.

[0223] In some embodiments, the image filtering method according to the present invention may be applied to a video decoder, for example, to loop filtering, and store the filtered reconstructed image in a decoded image buffer for subsequent image decoding, where the filter output influences video decoding. The present invention may improve the filtering quality of the reconstructed image, thereby improving the video decoding effect when subsequent decoding is performed based on the reconstructed image with better quality.

[0224] In some embodiments, the image filtering method according to the embodiments of the present invention may be applied to the product of video post-processing, for example, on the display side, i.e., the reconstructed image after filtering may be displayed directly without being stored in a decoded image buffer, in which case the output of the filter does not affect the video decoding.

[0225] In some embodiments, the image decoding method according to the embodiments of the present invention may be applied to any module using a neural network in neural network-based video coding, such as a neural network supervisor, a neural network intra-frame prediction module, etc., and only needs to replace the reconstructed image block with the corresponding image block.

[0226] In an image filtering method according to an embodiment of the present invention, a decoding side decodes a bitstream, determines a reconstructed image of a current image, and determines input information for a neural network filter based on the reconstructed image. The input information includes the reconstructed image. The neural network filter performs a filtering process on the input information to obtain a filtered image of the reconstructed image. The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method. That is, the embodiment of the present invention decomposes existing standard convolution operations into convolution operations with lower computational complexity using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method, thereby reducing the complexity of the filtering operation of the neural network filter without degrading the filtering performance of the neural network filter and improving the image filtering effect.

[0227] The image filtering method according to the embodiment of the present invention has been described above by taking the decoding side as an example. Hereinafter, the image filtering method according to the embodiment of the present invention will be described by taking the encoding side as an example.

[0228] 25 is a flowchart of an image filtering method according to an embodiment of the present invention. The method according to the embodiment of the present invention is applied to the encoder shown in FIG. 1 or 2 or other electronic devices. For convenience of explanation, the encoder will be taken as an example for description. As shown in FIG. 25, the method according to the embodiment of the present invention includes the following steps:

[0229] Step S301: Encode the current image, determine a reconstructed image of the current image, and determine input information for the neural network filter based on the reconstructed image.

[0230] Here, the input information includes a reconstructed image.

[0231] Methods according to embodiments of the present invention may be applied to video coding schemes or to video post-processing.

[0232] In some embodiments, when applied to a video coding scheme, when encoding a current image, the encoding side first divides the current image into coding blocks and encodes the coding blocks as coding units block by block. For example, for a current block to be coded in the current image, first obtain a predicted value of the current block by inter-frame prediction and / or intra-frame prediction. Then, obtain a residual value of the current block based on the predicted value of the current block and the current block. The encoding side transforms the residual value of the current block to obtain transform coefficients. In one example, the encoding side directly codes the transform coefficients of the current block without quantizing them to obtain a bitstream. In another example, the encoding side quantizes the transform coefficients of the current block, obtains quantized coefficients, codes the quantized coefficients, and obtains a bitstream.

[0233] In the encoding process, as shown in Figure 2, the encoding side inversely transforms the transform coefficients to obtain residual values, and then adds the residual values ​​to the predicted values ​​to obtain the reconstructed values ​​of the current block. Through the above steps, the reconstructed values ​​of each coding block of the current image can be obtained, and these reconstructed values ​​constitute a reconstructed image of the current image. Then, to further improve the quality of the reconstructed image, the reconstructed image is filtered to obtain a decoded image of the current image. In one example, the decoded image may be stored in a decoded image buffer for prediction of subsequent images.

[0234] In some embodiments, the image filtering method according to the embodiments of the present invention may be used to filter at least one frame image in a video, i.e., the current image is an image in the video.

[0235] In some embodiments, the image filtering method according to the embodiments of the present invention may be used to decode a single image, i.e., the current image is a single image, for example, an image generated by an electronic device.

[0236] In an embodiment of the present invention, the encoding side filters the reconstructed image using a neural network filter.

[0237] In an embodiment of the present invention, when filtering a reconstructed image, the entire reconstructed image may be directly used as a filtering unit to perform filtering, or the reconstructed image may be divided into multiple blocks and each block may be used as a filtering unit to perform filtering. For convenience of explanation, the embodiment of the present invention uses the reconstructed image as one filtering unit, and the reconstructed image may be one reconstructed image (i.e., one frame) or a partial region of the reconstructed image.

[0238] In one preferred embodiment, the reconstructed image includes at least one CTU.

[0239] In another preferred embodiment, the reconstructed image includes a predetermined image region of one frame of the reconstructed image.

[0240] The embodiments of the present invention are not limited to a specific method in which the encoding side determines input information to the neural network filter based on the reconstructed image.

[0241] In one preferred embodiment, the input information includes only the reconstructed image.

[0242] In one preferred embodiment, the input information includes, in addition to the reconstructed image, at least one of a predicted image corresponding to the reconstructed image, a boundary intensity image, a frame type (slice type) of the reconstructed image, and quantization parameter information.

[0243] Step S302: The input information is filtered by the neural network filter to obtain a filtered image of the reconstructed image.

[0244] Here, the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of the following methods: a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0245] Here, the specific network configuration of the neural network filter may be referred to in the description of the above embodiment, and the description thereof will be omitted here.

[0246] In some embodiments, the image filtering method according to the present invention may be applied to a video encoder, for example, to loop filtering, and store the filtered reconstructed image in an encoded image buffer for encoding subsequent images, where the output of the filter influences video decoding. The present invention may improve the filtering quality of the reconstructed image, thereby improving the video encoding effect when subsequent encoding is performed based on the reconstructed image with better quality.

[0247] In some embodiments, the image filtering method according to the embodiments of the present invention may be applied to the production of video post-processing, for example, on the display side, i.e., the reconstructed image after filtering may be displayed directly without being stored in a coded image buffer, in which case the output of the filter does not affect the video coding.

[0248] In some embodiments, the image coding method according to the embodiments of the present invention may be applied to any module using a neural network in neural network-based video coding, such as a neural network supervisor, a neural network intra-frame prediction module, etc., and only needs to replace the reconstructed image block with the corresponding image block.

[0249] In an image filtering method according to an embodiment of the present invention, an encoding side encodes a current image, determines a reconstructed image of the current image, and determines input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image, and the neural network filter performs a filtering process on the input information to obtain a filtered image of the reconstructed image, where the neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method. That is, the embodiment of the present invention decomposes the existing standard convolution operation into a convolution operation with lower computational complexity using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method, thereby reducing the complexity of the filtering operation of the neural network filter without degrading the filtering performance of the neural network filter and improving the image filtering effect.

[0250] Although the preferred embodiments of the present invention have been described in detail above with reference to the drawings, the present invention is not limited to the specific details of the above embodiments. Various simple modifications can be made to the technical concept of the present invention within the scope of the technical concept of the present invention, and all such simple modifications fall within the scope of protection of the present invention. For example, the specific technical features described in the above specific embodiments can be combined in any appropriate manner as long as they are not contradictory. To avoid unnecessary repetition, redundant descriptions of various possible combinations of the present invention will be omitted. Furthermore, for example, various different embodiments of the present invention may be arbitrarily combined, and as long as they do not contradict the concept of the present invention, they should be considered as being disclosed in the present invention.

[0251] In addition, in various embodiments of the method of the present invention, the magnitude of the sequence numbers of the above processes does not mean the order of execution, and the execution order of the processes does not constitute any limitation on the implementation process of the embodiments of the present invention, but should be determined in its functional and inherent logic.

[0252] An embodiment of the method of the present invention has been described above in detail with reference to Figures 9 to 25. An embodiment of the apparatus of the present invention will now be described in detail with reference to Figures 26 to 28.

[0253] 26 is a schematic block diagram of a neural network based image filtering device according to an embodiment of the present invention, which is a filtering device or part of a filtering device.

[0254] As shown in FIG. 26, the image filtering device includes the following components:

[0255] The determining unit 11 determines input information of the neural network filter based on the image to be filtered, where the input information includes the image to be filtered.

[0256] The filtering unit 12 performs filtering on the input information using a neural network filter to obtain a filtered image of the target image, where the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of the following methods: a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

[0257] In some embodiments, the neural network filter includes a residual unit including a plurality of sequentially connected residual blocks, wherein an i-th residual block among the plurality of residual blocks includes all or a portion of the plurality of convolutional layers, where i is a positive integer.

[0258] In some embodiments, the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, the first convolution unit and the second convolution unit are connected in parallel and then connected in series with the third convolution unit, the first convolution unit includes a first convolution layer, the second convolution unit includes a second convolution layer, the third convolution unit includes a third convolution layer, and the first convolution layer and the second convolution layer are connected in series with each other. The convolutional layers have different sizes of convolutional kernels, and at least one of the first, second and third convolutional layers includes a grouping convolutional layer, which divides the input feature information of the grouping convolutional layer into multiple groups based on the number of channels using a grouping convolution method, performs a convolution operation on each group feature information, and then performs channel number merging to obtain the output information of the grouping convolutional layer.

[0259] In some embodiments, the j-th convolutional layer among the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a first grouping sub-convolutional layer and a second grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer whose convolution kernel size is m×n using a tensor decomposition method and a group convolution method, where m and n are both positive integers greater than 2, and j is a positive integer.

[0260] In some embodiments, the size of the convolution kernels of the first grouping sub-convolution layer is 1×n, and the size of the convolution kernels of the second grouping sub-convolution layer is m×1.

[0261] In some embodiments, the third convolution unit further includes a fourth convolution layer connected in series with the third convolution layer.

[0262] In some embodiments, the fourth convolutional layer includes a grouped convolutional layer; or the fourth convolutional layer includes a third sub-convolutional layer and a fourth sub-convolutional layer obtained by performing decomposition on a convolutional layer having a convolutional kernel of size p×q using a tensor decomposition scheme, where p and q are both positive integers greater than 2; or the fourth convolutional layer includes a third grouped sub-convolutional layer and a fourth grouped sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer having a convolutional kernel of size p×q using a tensor decomposition scheme and a grouped convolution scheme.

[0263] In some embodiments, the i-th residual block includes N convolutional layers connected in series, and the k-th convolutional layer among the N convolutional layers includes a fifth grouping sub-convolutional layer and a sixth grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer with a convolutional kernel size of r×s using a tensor decomposition method and a group convolution method, where r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer less than or equal to N.

[0264] In some embodiments, the neural network filter further includes a shallow feature extraction unit including at least one convolutional layer, wherein the shallow feature extraction unit extracts shallow feature information of the input information and inputs the shallow feature information to the residual unit.

[0265] In some embodiments, the neural network filter further includes a feature mapping unit including at least one convolution layer, and the feature mapping unit performs a mapping process on the image feature information output by the residual unit to obtain a filtered image.

[0266] In some embodiments, the residual unit includes a luminance component residual unit and a chrominance component residual unit, wherein the luminance component residual unit performs a feature extraction process on the luminance component of the image to be filtered to obtain image feature information of the luminance component of the image to be filtered, and the chrominance component residual unit performs a feature extraction process on the chrominance component of the image to be filtered to obtain image feature information of the chrominance component of the image to be filtered.

[0267] In some embodiments, the neural network filter further includes a luminance component feature mapping unit connected to the luminance component residual unit and a chrominance component feature mapping unit connected to the chrominance component residual unit, where the luminance component feature mapping unit performs a mapping operation on the image luminance feature information output by the luminance component residual unit, and the chrominance component feature mapping unit performs a mapping operation on the image chrominance feature information output by the chrominance component residual unit.

[0268] In some embodiments, the image to be filtered is a reconstructed image, and the filtering unit 12 inputs the filtered image of the reconstructed image to a display device for display, stores the filtered image of the reconstructed image in a decoded image buffer as a reference image for other images, or inputs the filtered image of the reconstructed image to a display device for display without using it as a reference image.

[0269] It should be noted that the embodiments of the apparatus may correspond to the embodiments of the method, and similar descriptions may refer to the embodiments of the method. To avoid redundant description, the description thereof will be omitted here. Specifically, the apparatus shown in FIG. 26 can execute the embodiments of the filtering processing method described above, and the above and other operations and / or functions of various modules in the apparatus are for realizing the embodiments of the method corresponding to the filtering processing device, and the description thereof will be omitted here for brevity.

[0270] 27 is a schematic block diagram of an image decoding apparatus according to an embodiment of the present invention, which is a decoding device or a part of a decoding device.

[0271] As shown in FIG. 27, the image decoding device includes the following components.

[0272] The decoding unit 21 decodes the bitstream, determines a reconstructed image of the current image, and determines input information for the neural network filter based on the reconstructed image, where the input information includes the reconstructed image.

[0273] The filtering unit 22 performs filtering on the input information using a neural network filter to obtain a filtered image of the reconstructed image, where the neural network filter includes multiple convolution layers for extracting image feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0274] It should be noted that the device embodiments may correspond to the method embodiments, and similar descriptions may refer to the method embodiments. In order to avoid repetitive descriptions, the description thereof will be omitted here. Specifically, the device shown in FIG. 27 can execute the above-mentioned filtering processing method embodiment, and the above and other operations and / or functions of various modules in the device are for realizing the method embodiments corresponding to the decoder, and the description thereof will be omitted here for brevity.

[0275] 28 is a schematic block diagram of an image coding apparatus according to an embodiment of the present invention, which is an encoding device or part of an encoding device.

[0276] As shown in FIG. 28, the image encoding device 30 includes the following components.

[0277] The encoding unit 31 encodes the current image, determines a reconstructed image of the current image, and determines input information for the neural network filter based on the reconstructed image. The input information includes the reconstructed image.

[0278] The filtering unit 32 filters the input information using a neural network filter to obtain a filtered image of the reconstructed image, where the neural network filter includes multiple convolution layers for extracting feature information, and the multiple convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

[0279] It should be noted that the device embodiments may correspond to the method embodiments, and similar descriptions may refer to the method embodiments. To avoid repetitive descriptions, the description thereof will be omitted here. Specifically, the device shown in FIG. 28 can execute the above filtering processing method embodiment, and the above and other operations and / or functions of various modules in the device are for realizing the method embodiments corresponding to the encoder, respectively, and the description thereof will be omitted here for brevity.

[0280] The above describes the apparatus and system according to the embodiments of the present invention in terms of functional modules, with reference to the accompanying drawings. The functional modules may be implemented by instructions in hardware, software, or a combination of hardware and software modules. Specifically, steps of the method aspects according to the embodiments of the present invention may be performed by instructions in the form of integrated logic circuits and / or software in a hardware processor. The steps of the methods disclosed in connection with the embodiments of the present invention may be performed directly by a hardware decoding processor or by a combination of hardware and software modules in the decoding processor. Preferably, the software modules may be located in a storage medium established in the art, such as a random memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically rewritable programmable memory, or a register. Preferably, the storage medium is in a memory, and the processor reads information in the memory and executes the steps of the above method embodiments in combination with the hardware.

[0281] 29 is a schematic block diagram of an electronic device according to an embodiment of the present invention, which may be the decoder, encoder, or filtering device described above.

[0282] As shown in FIG. 29, the electronic device 30 may include the following components.

[0283] The memory 31 stores a computer program 33 and transmits the code of the computer program 33 to the processor 32. In other words, the processor 32 can call up and execute the computer program from the memory 31 to implement the method according to the embodiment of the present invention.

[0284] For example, the processor 32 may follow instructions in the computer program 33 to perform the steps in the above methods.

[0285] In some embodiments of the present invention, processor 32 may include, but is not limited to, a general-purpose processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, etc.

[0286] In some embodiments of the present invention, memory 31 may include, but is not limited to, volatile and / or nonvolatile memory. Nonvolatile memory may be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory may be random access memory (RAM), acting as external cache memory. Various forms of RAM may be used, including, by way of example and not limitation, static random access memory (Static RAM (SRAM)), dynamic random access memory (DRAM)), synchronous dynamic random access memory (Synchronous DRAM (SDRAM)), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus random access memory (Direct Rambus RAM (DR RAM)).

[0287] In some embodiments of the present invention, the computer program 33 may be divided into one or more modules that are stored in the memory 31 and executed by the processor 32 to perform the method of the present invention. The one or more modules may be a series of computer program instruction segments that describe the execution of the computer program in an electronic device and may perform a particular function.

[0288] 29, the electronic device 30 may further include a transceiver 34 connectable to the processor 32 or the memory 31. Here, the processor 32 may control the transceiver 34 to communicate with other devices, specifically, to transmit information or data to the other devices or to receive information or data transmitted by the other devices. The transceiver 34 may include a transmitter and a receiver. The transceiver 34 may further include an antenna, and the number of antennas may be one or more.

[0289] The components of the electronic device 30 are connected by a bus system, which includes a power bus, a control bus, and a status signal bus in addition to a data bus.

[0290] In one aspect of the present invention, there is provided a computer storage medium having stored thereon a computer program which, when executed by a computer, causes the computer to perform the method of the method embodiments described above.

[0291] In one aspect of the present invention, there is further provided a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of the method embodiment described above.

[0292] Another aspect of the present invention provides a computer program product or computer program comprising computer instructions stored on a computer readable storage medium, wherein a processor of the computing device reads the computer instructions from the computer readable storage medium and executes the computer instructions to cause the computing device to perform the method of the method embodiment described above.

[0293] In other words, when implemented using software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded into a computer and executed, the flow or function according to the embodiment of the present invention is generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored on a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from a website site, computer, server, or data center to another website site, computer, server, or data center via wire (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (infrared, radio, microwave, etc.). The computer-readable storage medium can be any available medium accessible by a computer, or a data storage device such as a server, data center, or the like that integrates one or more available media. The computer readable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a digital video disc (DVD)), a semiconductor medium (e.g., a solid state disk (SSD)), or the like.

[0294] Those skilled in the art will understand that the exemplary modules and algorithm steps described in connection with the embodiments disclosed herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether the functions disclosed herein are performed in hardware or software depends on the specific application and design constraints of the technical solution. The functions described herein may be implemented in different ways by those skilled in the art for each specific application, but such implementations should not be considered outside the scope of this specification.

[0295] The above are only specific embodiments of the present invention, but the scope of the present invention is not limited thereto, and any modifications or replacements that can be easily thought of by those skilled in the art within the technical scope disclosed in the present invention are within the scope of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. 1. A neural network-based image filtering method implemented by a filtering device, comprising: determining input information for a neural network filter based on the image to be filtered, the input information including the image to be filtered; and performing a filtering process on the input information by the neural network filter to obtain a filtered image of the filtering target image, The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

2. 2. The method of claim 1, wherein the neural network filter includes a residual unit including a plurality of sequentially connected residual blocks, and an i-th residual block among the plurality of residual blocks includes all or a portion of the plurality of convolutional layers, where i is a positive integer.

3. the i-th residual block includes a first convolution unit, a second convolution unit, and a third convolution unit, the first convolution unit and the second convolution unit being connected in parallel, and then connected in series with the third convolution unit; the first convolution unit includes a first convolution layer; the second convolution unit includes a second convolution layer; the third convolution unit includes a third convolution layer; the first convolutional layer and the second convolutional layer have different convolution kernel sizes; at least one of the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a grouping convolutional layer; 3. The method of claim 2, wherein the grouping convolutional layer divides input feature information of the grouping convolutional layer into a plurality of groups based on the number of channels in the group convolutional manner, performs a convolution operation on each group feature information, and then performs channel number merging to obtain output information of the grouping convolutional layer.

4. 4. The method of claim 3, wherein a j-th convolutional layer among the first convolutional layer, the second convolutional layer, and the third convolutional layer includes a first grouped sub-convolutional layer and a second grouped sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer whose convolution kernel size is m×n using the tensor decomposition scheme and the group convolution scheme, wherein m and n are both positive integers greater than 2, and j is a positive integer.

5. The size of the convolution kernel of the first grouping sub-convolution layer is 1×n; The method of claim 4 , wherein the size of the convolution kernels of the second grouping sub-convolutional layer is m×1.

6. The method of claim 3 , wherein the third convolution unit further includes a fourth convolution layer connected in series with the third convolution layer.

7. the fourth convolutional layer includes a grouping convolutional layer, or The fourth convolutional layer includes a third sub-convolutional layer and a fourth sub-convolutional layer obtained by performing decomposition on a convolutional layer having a convolution kernel size of p×q using the tensor decomposition method, where p and q are both positive integers greater than 2, or 7. The method of claim 6, wherein the fourth convolutional layer includes a third grouping sub-convolutional layer and a fourth grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer having a convolution kernel size of p × q using the tensor decomposition scheme and the group convolution scheme.

8. The i-th residual block includes N convolution layers connected in series, 3. The method of claim 2, wherein a k-th convolutional layer among the N convolutional layers includes a fifth grouping sub-convolutional layer and a sixth grouping sub-convolutional layer obtained by performing decomposition and channel grouping on a convolutional layer having a convolutional kernel size of r×s using the tensor decomposition scheme and the group convolution scheme, wherein r and s are both positive integers greater than 2, N is a positive integer, and k is a positive integer equal to or less than N.

9. the neural network filter further includes a shallow feature extraction unit including at least one convolutional layer; The method of claim 2 , wherein the shallow feature extraction unit extracts shallow feature information from the input information and inputs the shallow feature information to the residual unit.

10. the neural network filter further includes a feature mapping unit including at least one convolutional layer; The method of claim 2 , wherein the feature mapping unit performs a mapping process on the image feature information output by the residual unit to obtain the filtered image.

11. the residual units include a luma component residual unit and a chroma component residual unit; the luminance component residual unit performs a feature extraction process on the luminance component of the filtering target image to obtain image feature information of the luminance component of the filtering target image; The method according to claim 2 , wherein the chrominance component residual unit performs a feature extraction process on the chrominance components of the filtering target image to obtain image feature information of the chrominance components of the filtering target image.

12. the neural network filter further includes a luminance component feature mapping unit connected to the luminance component residual unit, and a chrominance component feature mapping unit connected to the chrominance component residual unit; the luminance component feature mapping unit performs a mapping process on the image luminance feature information output by the luminance component residual unit; The method of claim 11 , wherein the chrominance component feature mapping unit performs a mapping process on the image chrominance feature information output by the chrominance component residual unit.

13. the image to be filtered is a reconstructed image, The method comprises:

2. The method of claim 1, further comprising the steps of inputting the filtered image of the reconstructed image to a display device for display, storing the filtered image of the reconstructed image in a decoded image buffer as a reference image for other images, or inputting the filtered image of the reconstructed image to a display device for display without using it as a reference image.

14. An image decoding method executed by a decoding device, comprising: decoding the bitstream to determine a reconstructed image of the current image, and determining input information for a neural network filter based on the reconstructed image, wherein the input information includes the reconstructed image; and performing a filtering process on the input information by the neural network filter to obtain a filtered image of the reconstructed image; The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

15. 1. An image encoding method performed by an encoding device, comprising: encoding a current image, determining a reconstructed image of the current image, and determining input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image; filtering the input information using a neural network filter to obtain a filtered image of the reconstructed image; The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multi-level receptive field method, a tensor decomposition method, and a group convolution method.

16. 1. A neural network-based image filtering device applied to a filtering device, comprising: a determining unit for determining input information of a neural network filter based on an image to be filtered, the input information including the image to be filtered; a filtering unit that performs a filtering process on the input information using the neural network filter to obtain a filtered image of the filtering target image; The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

17. An image decoding device applied to a decoding device, a decoding unit for decoding the bitstream, determining a reconstructed image of a current image, and determining input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image; a filtering unit that performs a filtering process on the input information using the neural network filter to obtain a filtered image of the reconstructed image; The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

18. An image encoding device applied to an encoding device, comprising: an encoding unit that encodes a current image, determines a reconstructed image of the current image, and determines input information for a neural network filter based on the reconstructed image, the input information including the reconstructed image; a filtering unit that performs a filtering process on the input information using a neural network filter to obtain a filtered image of the reconstructed image; The neural network filter includes a plurality of convolution layers for extracting image feature information, and the plurality of convolution layers are obtained by processing standard convolution layers using at least two of a multilevel receptive field method, a tensor decomposition method, and a group convolution method.

19. 16. An electronic device comprising a processor and a memory having a computer program stored therein, the processor executing the computer program to implement a method according to any one of claims 1 to 13, 14 and 15.

20. A computer program causing a computer to carry out the method according to any one of claims 1 to 13, 14 and 15.

Images